Truxton .NET Core

Truxton .NET Core builds upon the Truxton.ETL Managed API and introduces dotnet core features like dependency injection, configuration and idiomatic message consumer patterns.

It is recommended to use this SDK for most C# ETLs.

You should also familiarize yourself with How_Truxton_Works, its Message_Bus and the different ETL_Stages.

Overview

Truxton .NET Core runs against .Net Core 3.1 on windows 10 on 64-bit platforms (TargetFramework: netcoreapp3.1, RuntimeIdentifier: win10-x64).

Getting Started

Truxton .NET Core ETLs are simply .Net Core 3.1 console exes that include references to the the Truxton .NET Core libraries.

Prerequisites

You will first need to install the .NET Core 3.1 SDK. IMPORTANT You must choose the Windows x64 SDK. It is recommended to install using the windows installer. Here is a link to the latest 3.1 SDK - 3.1.404 (newer versions might be available when you read this).

Once you have the SDK installed, install Visual Studio 2019 Community.

Project Setup

Now lets choose the appropriate project style to begin building our ETL.

• First, Launch Visual Studio 2019 Community and select Create a New Project
• Select Language: C#, Platform: Windows, Project Type: Console. Choose the Console App (.NET Core) template and click Next
• Choose a suitable name and location for your ETL and click Create

Truxton DLL References

All needed dlls should be in Truxton's install directory SDK folder e.g. C:\Program Files\Truxton\SDK

At a minimum you will need to add a reference to the following Truxton .NET Core DLLs to implement your .NET Core ETL

• Truxton.Messaging.dll - The main library for Truxton .NET Core ETLs
• Truxton.Messaging.Postgres.dll - Implementation library for Postgres Messaging flavor of Truxton (shipped by default)

You might also need to reference the following:

• Truxton.DataAccess.dll - Main library for accessing metadata and content in Truxton
• Truxton.DataAccess.Postgres.dll - Implementation library of Postgres DataAccess flavor of Truxton (shipped by default)
• TruxtonAssembly.dll - Contains most POCO models and other core utilities for Truxton domain specifics.

Consuming Messages

HEADS UP: All of the interfaces listed in this section are found in the Truxton.Messaging.dll .Net Core library.

We now need to configure our ETL to accept certain kinds of information found when media is being processed and exploited. This is achieved by implementing one or more IETLMessageConsumer interfaces.

All consumers will have to implement the following properties:

    /// <summary>
    /// A Message consumer which consumes different kind of truxton messages
    /// </summary>
    public interface IETLMessageConsumer
    {
        /// <summary>
        /// Which stage this consumer will take part in. Value should be between 1 and 255 and should adhere to Truxton.Messaging.Stages enum
        /// </summary>
        int Stage { get; }

        /// <summary>
        /// The name of this ETL
        /// </summary>
        string ApplicationName { get; }

        /// <summary>
        /// The message queue this ETL will identify as. Do not use spaces.
        /// </summary>
        string MessageQueueName { get; }

        /// <summary>
        /// The purpose of this ETL and what its trying to achieve
        /// </summary>
        string Description { get; }

        /// <summary>
        /// How this ETL should be further configured
        /// </summary>
        ETLOptions Options { get; }
    }

    /// <summary>
    /// How to configure this consumer
    /// </summary>
    public class ETLOptions
    {
        /// <summary>
        /// If the consumer should have verbose message logging
        /// </summary>
        public bool VerboseMessageLogging { get; set; }

        /// <summary>
        /// If the consumer should not load hashsets (this can speed up the ETL in certain circumstances)
        /// </summary>
        public bool DontEliminateByHash { get; set;  }
    }

All of the higher level Consumers described below inherit from the following interface:

    /// <summary>
    /// This is a MessageConsumer of a specific type
    /// </summary>
    /// <typeparam name="TMessageType"></typeparam>
    public interface IETLMessageConsumer<TMessageType> : IETLMessageConsumer
    {
        /// <summary>
        /// Handles a message asynchronously
        /// </summary>
        /// <param name="message">The message to handle</param>
        /// <param name="ct">A cancellation token to signal operation cancellation</param>
        /// <returns></returns>
        Task HandleMessageAsync(TMessageType message, CancellationToken ct);
    }

This means your consumer must implement this method for the Specific MessageType and this will be the main logic handler for receiving messages from Truxton. As an example, the IFileConsumer would implement it like so:

    public class MyETL : IFileConsumer
    {
        public async Task HandleMessageAsync(FileMessage message,
            CancellationToken ct)
        {
             //now do something with the FileMessage
        }
    }

The most common kind of IETLMessageConsumer is an IFileConsumer which we will describe more in the next section.

IFileConsumer

If your ETL implements the IFileConsumer interface it will receive messages when files are discovered when media is loading. See File_Types_Supported for the different kinds of Files that can be found or generated from media.

Here is the IFileConsumer interface and FileMessage it consumes:

    /// <summary>
    /// This ETL is a consumer of file messages
    /// </summary>
    public interface IFileConsumer : IETLMessageConsumer<FileMessage>
    {
        /// <summary>
        /// Returns which file types IDs this file consumer cares about
        /// </summary>
        /// <param name="ct">A cancellation token to signal operation cancellation</param>
        /// <returns>An IEnumerable of file type ids which this Consumer is interested in</returns>
        Task<IEnumerable<ushort>> ConfigureFileTypesAsync(CancellationToken ct);
    }

    /// <summary>
    /// A file message
    /// Represents a file of interest routed through Truxton
    /// </summary>
    public class FileMessage
    {
        /// <summary>
        /// The ID of this file
        /// </summary>
        public Guid ID { get; }

        /// <summary>
        /// The name of this file
        /// </summary>
        public string Name { get; }

        /// <summary>
        /// The type of this file
        /// </summary>
        public ushort FileType { get; private set; }

        /// <summary>
        /// The 32 character md5 hash
        /// </summary>
        public string MD5 { get; }

        /// <summary>
        /// The 40 character sha1 hash
        /// </summary>
        public string SHA1 { get; }

        /// <summary>
        /// The first 4 bytes of the file represented as a unsigned integer
        /// </summary>
        public uint Signature { get; }
        
        /// <summary>
        /// The size in bytes of this file
        /// </summary>
        public ulong Length { get; }

        /// <summary>
        /// The offset where this file is stored in the depot
        /// </summary>
        public ulong Offset { get; }

        /// <summary>
        /// The file id of the parent file that contained this file. 
        /// Empty guid if this file did not have a parent
        /// </summary>
        public Guid ParentID { get; }

        /// <summary>
        /// The media id of the media where this file came from
        /// </summary>
        public Guid MediaID { get; }

        /// <summary>
        /// The depot id of where this files content is stored
        /// </summary>
        public Guid DepotID { get; }

        /// <summary>
        /// The name of the depot where this file is found.
        /// </summary>
        public string DepotFilename { get; }

        /// <summary>
        /// If this File has contents available. If false this file is most likely eliminated via hash.
        /// </summary>
        public bool ContentsAvailable { get; }

        /// <summary>
        /// The original message this message was derived from
        /// </summary>
        public TruxtonMessage OriginalMessage { get; }
    }

In order to implement the IFileConsumer, it needs to implement ConfigureFileTypesAsync which tells Truxton which filetypes this consumer is interested in.

Here is an example which subscribes to different kinds of documents that might need additional processing.

    ...

    public Task<IEnumerable<ushort>> ConfigureFileTypesAsync(CancellationToken ct)
    {
        return Task.FromResult<IEnumerable<ushort>>(
             new List<ushort>
             {
                 (ushort)FileTypes.Type_Adobe_PDF,
                 (ushort)FileTypes.Type_Excel_Spreadsheet,
                 (ushort)FileTypes.Type_Excel_2007,
                 (ushort)FileTypes.Type_PowerPoint_2007,
                 (ushort)FileTypes.Type_Word_Document,
                 (ushort)FileTypes.Type_Word_2007,
                 (ushort)FileTypes.Type_RichText,
             });
    }

    ...

IArtifactConsumer

If your ETL implements the IArtifactConsumer interface it will receive messages pertaining to artifacts (also known as Entities) when media is being processed. See Entity_Types for the different kinds of Artifacts that could be exploited from media.

ICameraInfoConsumer

If your ETL implements the ICameraInfoConsumer interface it will receive messages pertaining to metadata like EXIFs information when media is being processed. Camera info could be to metadata generated from cameras - either static or video. See the EXIF_Table entry for information that can be exploited.

ICommunicationConsumer

If your ETL implements the ICommunicationConsumer interface it will receive messages pertaining to communications exploited from files like SMS, Chat Programs or Email. See Message_Types for the different kinds of Communications which can be exploited from media.

HEADS UP This consumer is still a work in progress.

IEventConsumer

If your ETL implements the IEventConsumer interface it will receive messages pertaining to notable points in time found in the media being processed. This is slated more towards notable events and not include 'noisy' timestamps like file created times. See Event_Types for the different kinds of Events which can be exploited from the media.

ILocationConsumer

If your ETL implements the ILocationConsumer interface it will receive messages pertaining to geographic coordinates found in the media being processed. See Location_Types for the different kinds of Locations which can be exploited from the media.

IRelationConsumer

If your ETL implements the IRelationConsumer interface it will receive messages pertaining to relations associated together in the media being processed.

ITagConsumer

If your ETL implements the ITagConsumer interface it will receive messages pertaining to tags being associated in the media being processed.

IWebsiteConsumer

If your ETL implements the IWebsiteConsumer interface it will receive messages pertaining to websites which were visited in the media being processed.

IMediaConsumer

If your ETL implements the IMediaConsumer interface it will receive messages pertaining to media that is progressing thru the load and exploited phases. See ETL_Stages for information about how media interacts with different stages and their states. This kind of consumer is useful when you want it 'run one' per loaded media.

IETL

The last consumer is a generic catch-all for all messages. You still have to configure which File_Types_Supported you care about and is generally only useful for compatibility with legacy etls.

The ETL Host

coming soon

Dependency Injection

coming soon

Common Truxton Interfaces Which Can Be Injected

coming soon

Injecting Your Own Dependencies

coming soon

Outside Configuration

coming soon

Common ETL Operations

This section will list a common set of scenarios you may encounter when writing an ETL and details how to implement a solution.

Getting Content

Inevitable when dealing with Files found from Truxton and passed to your ETL, you will need to access the content or the bytes which comprise said file.

The interface of interest to achieve this is the Truxton.Messaging.IETLContentService interface. Here is the method

    /// <summary>
    /// A content service with extra methods tailored for ETLs
    /// </summary>
    public interface IETLContentService : IContentService
    {
        /// <summary>
        /// Gets a stream of a file from a FileMessage asynchronously
        /// </summary>
        /// <param name="message">A FileMessage to get a stream to</param>
        /// <param name="ct">A cancellation token to signal operation cancellation</param>
        /// <returns>A Stream of the content for this FileMessage</returns>
        Task<Stream> GetStreamAsync(FileMessage message, CancellationToken ct);

        ...
    }

This interface is available as one of the default available services provided by the PostgresETLHostBuilder mentioned above (this means you do not need to register it yourself). Inject it into your consumer into your consumer and use it in your ETL consumer.

    public class MyFileETL: IFileConsumer
    {
         readonly IETLContentService _contentService;

         public MyFileETL(IETLContentService contentService)
         {
             _contentService = contentService;
         }

         ...
    }

When you need to obtain a stream to a FileMessage you can do the following:

    public class MyFileETL: IFileConsumer
    {
        ...
        public async Task HandleMessageAsync(FileMessage message,
            CancellationToken ct)
        {
            using (var stream = await _contentService.GetStreamAsync(message, ct))
            {
                //you now have stream access to the bytes of the file!
            }
        }
        ... 
    }

The stream you are given is a stream to the bytes of the file associated to that file message. Do whatever you like with it! Just remember best practices in your method body like always async/awaiting where possible and disposing of your IDisposables.

Adding Content without an associated file

In some cases you may want to permanent persist content to Truxton without it being associated to a file entry. You can achieve this using the same interface mentioned above, except a different method:

    /// <summary>
    /// A content service with extra methods tailored for ETLs
    /// </summary>
    public interface IETLContentService : IContentService
    {
        ...
        <summary>
        /// Saves a stream of data into truxton. It is up to you to persist the response location somewhere.
        /// </summary>
        /// <param name="stream">The stream of data to save</param>
        /// <returns></returns>
        SavedContentResponse SaveStream(Stream stream);

        ...
    }

And here is the SavedContentRespone class

    /// <summary>
    /// The response of a SaveStream operation from IETLContentService
    /// </summary>
    public class SavedContentResponse
    {
        /// <summary>
        /// If this operation succeeded
        /// </summary>
        public bool Success { get; }

        /// <summary>
        /// The DepotID where the stream was saved
        /// </summary>
        public Guid DepotID { get; }

        /// <summary>
        /// The offset into the depot where the stream was saved
        /// </summary>
        public long Offset { get; }

        /// <summary>
        /// The length that was saved to the depot
        /// </summary>
        public long Length { get; }
    }

So to save a block of content to truxton you can call it as such

    byte[] byteArray = Encoding.ASCII.GetBytes("My super important content");
    MemoryStream stream = new MemoryStream(byteArray);
    SavedContentResponse response = await _contentService.SaveStream(stream);

If the operation was a success you can now persist the SavedContentResponse to somewhere else so you can reference it and recall your data at a later time.

Fetching Related Information

Most related information can be retrieved using the ITruxtonDatabase interface. Which is also available as a default injectable service:

    public class MyFileETL: IFileConsumer
    {
         readonly ITruxtonDatabase _database;

         public MyFileETL(ITruxtonDatabase database)
         {
             _database = database;
         }

         ...
    }

The ITruxtonDatase interface has too many methods to list here but we'll list a method of interest here to fetch the parent file information for a FileMessage:

    FileModel fileModel = await _database.GetFileAsync(message.ParentID, ct);

or if you needed to write a custom query to access the Truxton Database:

    ...
    using (var connection = await _database.ConnectAsync(ct))
    using (var command = connection.CreateCommand())
    {
        command.CommandText = "SQL QUERY"
        using (var data_reader = await command.ExecuteReaderAsync(ct))
        {
            //enumerate the response of the data_reader
        }
    }

Interfacing With External Programs Or WebAPIs

coming soon

Broadcasting To Others

coming soon