Extending DocIntel with Modules

In DocIntel, a module is a set of files that enhances the platform’s functionalities. These modules are dynamically loaded during startup.

Each module requires a configuration file named “module.json” placed at the root of its corresponding folder. For instance, a module designed to import all Malpedia reports would have the following folder structure:

/modules
- malpedia
  - module.json
  - ...  

All the dependencies required for the module, except those provided by DocIntel.Core, should be included in the subfolder.

Configuration with module.json

The module.json file serves as a descriptor for the modules to be loaded in DocIntel. It contains several key-value pairs:

• Name: Identifies the name of the module.
• Assembly: Specifies the assembly to be loaded.
• Exporters: A dictionary that associates exporter endpoints with their respective classes (currently undocumented).
• Metadata: A dictionary that maps known DocIntel types to additional metadata. For example, additional geographical metadata for tags.
• Profiles: A list of AutoMapper profiles to load.
• Collectors: A dictionary that maps collector names to their associated classes. This allows adding additional collectors to DocIntel.

For instance, consider a module that provides additional geographical metadata to tags, which would have a configuration file similar to the example below:

{
  "name": "geo",
  "assembly": "Cyantlabs.DocIntel.Geo.dll",
  "metadata": {
    "DocIntel.Core.Models.Tag": {
      "name": "geo",
      "title": "Geography Mapping",
      "classname": "Cyantlabs.DocIntel.Geo.GeoTagMetadata"
    }
  },
  "profiles": [ "Cyantlabs.DocIntel.Geo.GeoProfile" ]
}

Similarly, a module providing additional collectors might have a configuration file like this:

{
  "name": "malpedia",
  "assembly": "Cyantlabs.DocIntel.Malpedia.dll",
  "collectors": {
    "threat-intel-reports": {
      "class": "Cyantlabs.DocIntel.Malpedia.ReportCollector",
      "settings": "Cyantlabs.DocIntel.Malpedia.ReportCollectorSettings"
    }
  },
  "metadata": {
  },
  "profiles": [ 
    "Cyantlabs.DocIntel.Malpedia.MalpediaProfile"
  ]
}

Metadata

A metadata class defines the fields to be presented to the user when editing the corresponding type. For instance, consider the following example:

public class GeoTagMetadata
{
    [Title("Country Code")]
    public string CountryCode { get; set; }
}

In this case, the GeoTagMetadata class contains a field named CountryCode, which will be displayed to the user as “Country Code” when editing the corresponding type.

You can view an example of such metadata in the documentation of the ThreatQ module.

Collectors

A collector is a class that implements the IDocumentCollector interface. The method Collect is invoked by the DocIntel.Service.Cron service according to the specified configuration. The collector returns objects that are then processed and updated based on its configuration.

public interface IDocumentCollector
{
    IAsyncEnumerable<DocumentImport> Collect(Collector lastCollection, object settings = null);
}

The type of the settings object is defined in the settings key of the corresponding object in the configuration file. For example, in the provided interface, the settings object type would be Cyantlabs.DocIntel.Malpedia.ReportCollectorSettings.

The returned objects are instances of DocumentImport, which describe the documents to be imported. You can find further details about the DocumentImport class in the DocIntel.Core.Collectors namespace.

To import files, you add FileImport objects, containing a StringContent object or a SteamContent object, to the DocumentImport object via the AddFile method.

mapped.AddFile(new FileImport()
{
    Title = "JSON Report",
    Filename = $"{reportEntry.ReportId}.json",
    MimeType = "application/json",
    Content = new StringContent(await _client.Report.Json.FindByIdAsync(reportEntry.ReportId)),
    Preview = false,
    Visible = false
});

mapped.AddFile(new FileImport()
{
    Title = "PDF Report",
    Filename = $"{reportEntry.ReportId}.pdf",
    MimeType = "application/pdf",
    Content = new StreamContent(await _client.Report.Download(reportEntry.ReportId)),
    Preview = true,
    Visible = true
});

Profiles

DocIntel automatically loads profiles, which are standard AutoMapper profiles. To learn how to write these profiles, you can refer to the official documentation of AutoMapper.

Here is a skeleton of a profiles class as an example:

using AutoMapper;
...

namespace Cyantlabs.DocIntel.Geo
{
    public class GeoProfile : Profile
    {
        public GeoProfile(ApplicationSettings settings)
        {
            ...
        }
    }
}

In this example, the GeoProfile class extends Profile from AutoMapper, and it can be used to configure mapping between different types. You can include specific mappings within the constructor of the profile class, utilizing the ApplicationSettings to further customize the mappings as needed.