Truxton .NET Core

Truxton .NET Core builds upon the Truxton.ETL Managed API and introduces dotnet core features like dependency injection, configuration and idiomatic message consumer patterns.

It is recommended to use this SDK for most C# ETLs.

You should also familiarize yourself with How Truxton Works, its Message Bus and the different ETL Stages.

Overview

Truxton .NET Core runs against .Net Core 3.1 on windows 10 on 64-bit platforms (TargetFramework: netcoreapp3.1, RuntimeIdentifier: win10-x64).

Getting Started

Truxton .NET Core ETLs are .Net Core 3.1 console executables that include references to the the Truxton .NET Core libraries.

Prerequisites

You will first need to install the .NET Core 3.1 SDK. IMPORTANT You must choose the Windows x64 SDK. It is recommended to install using the windows installer. Here is a link to the latest 3.1 SDK - 3.1.404 (newer versions might be available when you read this).

Once you have the SDK installed, install Visual Studio 2019 Community.

Project Setup

Now lets choose the appropriate project style to begin building our ETL.

• First, Launch Visual Studio 2019 Community and select Create a New Project
• Select Language: C#, Platform: Windows, Project Type: Console. Choose the Console App (.NET Core) template and click Next
• Choose a suitable name and location for your ETL and click Create

Project Architecture

Truxton and its supporting libraries are strictly 64bit. We have no plans on producing or supporting 32bit. When you start a new project and soluation in Visual Studio, it typically defaults To "Any CPU". We will have to change this using the following steps:

• From The Top Menu select Build->Configuration Manager
• In the "Active solution platform" dropdown select "New"
• In The "New Solution Platform" modal ensure 'x64' is in the new platform field. Choose 'OK'
• Clean up remaining references to Any CPU
  • In the "Active solution platform" dropdown select "Edit"
  • In the "Edit Solution Platform" modal select "Any CPU", Choose "Remove" and confirm and close
  • In your specific projects Platform select "Edit"
  • In the "Edit Project Platforms" modal select "Any CPU", Choose "Remove" and confirm and close

We are now ready to build a x64 console application

Configure Logging

Truxton .NET Core uses NLog for logging. You will have to either add the configuration yourself or use the one provided by the SDK. See here for how to configure NLog. The config file we use is in the SDK folder: C:\Program Files\Truxton\SDK\NLog.config

Here is its contents for your reference:

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <!-- 
  See https://github.com/nlog/nlog/wiki/Configuration-file 
  for information on customizing logging rules and outputs.
   -->
  <targets>
    <!-- add your targets here -->

    <target xsi:type="Console" 
            name="ConsoleLog" 
            detectConsoleAvailable="true"
            layout="${longdate} ${uppercase:${level}} ${message} ${exception:format=tostring}" />
    
    <target xsi:type="Debugger"
            name="DebuggerLog"
            layout="${longdate} ${uppercase:${level}} ${message} ${exception:format=tostring}" />
    
    <!-- This target needs a variable called 'appName'. 
         It will automatically be set for you if you don't do it manually. 
         You can specify it by calling the Init function in TLog.cs -->
    <target xsi:type="File" 
            name="FileLog" 
            fileName="${specialfolder:folder=LocalApplicationData}/Probity/Truxton/${gdc:item=appName}.log"
            archiveFileName="${specialfolder:folder=LocalApplicationData}/Probity/Truxton/${gdc:item=appName}.{#}.log"
            archiveEvery="Day"
            archiveNumbering="Rolling"
            maxArchiveFiles="7"
            keepFileOpen="true"
            concurrentWrites="false"
            openFileCacheTimeout="30"
            layout="${longdate} ${uppercase:${level}} ${message} ${exception:format=tostring}" />

    <target xsi:type="MethodCall" 
            name="MethodLog"
            className="Truxton.TLog, TruxtonAssembly"
            methodName="OnMessageRecieved">
      <parameter layout="${level}" />
      <parameter layout="${message}" />
    </target>
  </targets>

  <rules>
    <!-- add your logging rules here -->
    <logger name="*" minlevel="Trace" writeTo="DebuggerLog" />
    <logger name="*" minlevel="Trace" writeTo="ConsoleLog" />
    <logger name="*" minlevel="Trace" writeTo="FileLog" />
    <logger name="*" minlevel="Trace" writeTo="MethodLog" />
  </rules>
</nlog>

This sample configuration adds several targets include console out and local file logging which is handy to see when debugging ETLs.

Once you have a logging configured, it is recommended you set the following properties on the file to ensure it ends up in your output directory:

• Build Action to None
• Copy To Output Directory to Copy always

Truxton References

Nuget Packages

The SDK provides several convenient NuGet packages to reference Truxton libraries. This is the recommended way to add Truxton references to your ETL. They are located in the SDK folder: C:\Program Files\Truxton\SDK\NugetPackages.

Here's how to add them your ETL.

Add a local NuGet feed to Visual studio

In order to consumer local NuGet packages we will need to add them as a NuGet feed. Here's how to add the Truxton SDK local feed:

From Visual Studio:

• Choose Tools from the top menu, then NuGet Package Manager, then Package Manager Settings
• Select the Package Sources option on the left side.
• Click the green plus button in the top right. It will create a new entry for you
• Under the Name field, enter: Truxton SDK
• Under the Source field, enter: C:\Program Files\Truxton\SDK\NuGetPackages
• Click OK to save this new NuGet source

You should only need to configure this one time as it is a Global Config for Visual studio.

Add the desired package reference

Now that we have the Truxton SDK NuGet package source added, we can add the right PackageDependecies to your ETL. From Visual Studio, do the following

• Right click on your ETL Project and select Manage NuGet Packages
• In the top right under Package Source choose Truxton SDK
• You should see a feed of the following packages:
  • SamBlackburn.Geodesy
  • Truxton.DataAccess
  • Truxton.DataAccess.Postgres
  • Truxton.Messaging
  • Truxton.Messaging.Postgres
  • Truxton.Notifications.Core
  • Truxton.Plugins.Core
  • Truxton.Reports.Core
  • Truxton.Transliteration
  • TruxtonAssembly
• Choose Truxton.Messaging.Postgres and select Install to install it as a package reference to your ETL project.

The only package you should need to add for our purposes is Truxton.Messaging.Postgres. This particular package should contain all the sub-dependencies you need to make an ETL and reference core helper classes.

DLL CheatSheet

The following dlls are located SDK folder e.g. C:\Program Files\Truxton\SDK in an unpackaged format.

• Truxton.Messaging.dll - The main library for Truxton .NET Core ETLs
• Truxton.Messaging.Postgres.dll - Implementation library for Postgres Messaging flavor of Truxton (shipped by default)
• Truxton.DataAccess.dll - Main library for accessing metadata and content in Truxton
• Truxton.DataAccess.Postgres.dll - Implementation library of Postgres DataAccess flavor of Truxton (shipped by default)
• TruxtonAssembly.dll - Contains most POCO models and other core utilities for Truxton domain specifics.
• Truxton.ETL.dll - C++/CLI library that contains core classes and methods for ETLs.
• MBCPG.dll - Native C++ library that is a message bus client for postgres. Brokers messages to/from the message bus.

Consuming Messages

HEADS UP: All of the interfaces listed in this section are found in the Truxton.Messaging.dll .NET Core library.

We now need to configure our ETL to accept certain kinds of information found when media is being processed and exploited. This is achieved by implementing one or more IETLMessageConsumer interfaces.

All consumers will have to implement the following properties:

/// <summary>
/// A Message consumer which consumes different kind of truxton messages
/// </summary>
public interface IETLMessageConsumer
{
    /// <summary>
    /// Which stage this consumer will take part in. Value should be between 1 and 255 and should adhere to Truxton.Messaging.Stages enum
    /// </summary>
    int Stage { get; }

    /// <summary>
    /// The name of this ETL
    /// </summary>
    string ApplicationName { get; }

    /// <summary>
    /// The message queue this ETL will identify as. Do not use spaces.
    /// </summary>
    string MessageQueueName { get; }

    /// <summary>
    /// The purpose of this ETL and what its trying to achieve
    /// </summary>
    string Description { get; }

    /// <summary>
    /// How this ETL should be further configured
    /// </summary>
    ETLOptions Options { get; }
}

/// <summary>
/// How to configure this consumer
/// </summary>
public class ETLOptions
{
    /// <summary>
    /// If the consumer should have verbose message logging
    /// </summary>
    public bool VerboseMessageLogging { get; set; }

    /// <summary>
    /// If the consumer should not load hashsets (this can speed up the ETL in certain circumstances)
    /// </summary>
    public bool DontEliminateByHash { get; set;  }
}

All of the higher level Consumers described below inherit from the following interface:

/// <summary>
/// This is a MessageConsumer of a specific type
/// </summary>
/// <typeparam name="TMessageType"></typeparam>
public interface IETLMessageConsumer<TMessageType> : IETLMessageConsumer
{
    /// <summary>
    /// Handles a message asynchronously
    /// </summary>
    /// <param name="message">The message to handle</param>
    /// <param name="ct">A cancellation token to signal operation cancellation</param>
    /// <returns></returns>
    Task HandleMessageAsync(TMessageType message, CancellationToken ct);
}

This means your consumer must implement this method for the specific TMessageType and this will be the main logic handler for receiving messages from Truxton. As an example, the IFileConsumer would implement it like so:

public class MyETL : IFileConsumer
{
    public async Task HandleMessageAsync(FileMessage message, CancellationToken ct)
    {
         //now do something with the FileMessage
    }
}

The most common kind of IETLMessageConsumer is an IFileConsumer which we will describe more in the next section.

IFileConsumer

If your ETL implements the IFileConsumer interface it will receive messages when files are discovered when media is loading. See File_Types_Supported for the different kinds of Files that can be found or generated from media.

Here is the IFileConsumer interface and FileMessage it consumes:

/// <summary>
/// This ETL is a consumer of file messages
/// </summary>
public interface IFileConsumer : IETLMessageConsumer<FileMessage>
{
    /// <summary>
    /// Returns which file types IDs this file consumer cares about
    /// </summary>
    /// <param name="ct">A cancellation token to signal operation cancellation</param>
    /// <returns>An IEnumerable of file type ids which this Consumer is interested in</returns>
    Task<IEnumerable<ushort>> ConfigureFileTypesAsync(CancellationToken ct);
}

/// <summary>
/// A file message
/// Represents a file of interest routed through Truxton
/// </summary>
public class FileMessage
{
    /// <summary>
    /// The ID of this file
    /// </summary>
    public Guid ID { get; }

    /// <summary>
    /// The name of this file
    /// </summary>
    public string Name { get; }

    /// <summary>
    /// The type of this file
    /// </summary>
    public ushort FileType { get; private set; }

    /// <summary>
    /// The 32 character md5 hash
    /// </summary>
    public string MD5 { get; }

    /// <summary>
    /// The 40 character sha1 hash
    /// </summary>
    public string SHA1 { get; }

    /// <summary>
    /// The first 4 bytes of the file represented as a unsigned integer
    /// </summary>
    public uint Signature { get; }
        
    /// <summary>
    /// The size in bytes of this file
    /// </summary>
    public ulong Length { get; }

    /// <summary>
    /// The offset where this file is stored in the depot
    /// </summary>
    public ulong Offset { get; }

    /// <summary>
    /// The file id of the parent file that contained this file. 
    /// Empty guid if this file did not have a parent
    /// </summary>
    public Guid ParentID { get; }

    /// <summary>
    /// The media id of the media where this file came from
    /// </summary>
    public Guid MediaID { get; }

    /// <summary>
    /// The depot id of where this files content is stored
    /// </summary>
    public Guid DepotID { get; }

    /// <summary>
    /// The name of the depot where this file is found.
    /// </summary>
    public string DepotFilename { get; }

    /// <summary>
    /// If this File has contents available. If false this file is most likely eliminated via hash.
    /// </summary>
    public bool ContentsAvailable { get; }

    /// <summary>
    /// The original message this message was derived from
    /// </summary>
    public TruxtonMessage OriginalMessage { get; }
}

In order to implement the IFileConsumer, it needs to implement the ConfigureFileTypesAsync method which tells Truxton which filetypes this consumer is interested in.

Here is an example which subscribes to different kinds of documents that might need additional processing.

...

public Task<IEnumerable<ushort>> ConfigureFileTypesAsync(CancellationToken ct)
{
    return Task.FromResult<IEnumerable<ushort>>(
         new List<ushort>
         {
             (ushort)FileTypes.Type_Adobe_PDF,
             (ushort)FileTypes.Type_Excel_Spreadsheet,
             (ushort)FileTypes.Type_Excel_2007,
             (ushort)FileTypes.Type_PowerPoint_2007,
             (ushort)FileTypes.Type_Word_Document,
             (ushort)FileTypes.Type_Word_2007,
             (ushort)FileTypes.Type_RichText,
         });
}
...

IArtifactConsumer

If your ETL implements the IArtifactConsumer interface it will receive messages pertaining to artifacts (also known as Entities) when media is being processed. See Entity_Types for the different kinds of Artifacts that could be exploited from media.

Here is the IArtifactConsumer interface and ArtifactMessage it consumes:

/// <summary>
/// This ETL is a consumer of Artifact messages
/// </summary>
public interface IArtifactConsumer : IETLMessageConsumer<ArtifactMessage>
{
}

/// <summary>
/// This represents an Artifact in Truxton. It corresponds to the Entity table
/// </summary>
public class ArtifactMessage
{
    /// <summary>
    /// The ID of the artifact
    /// </summary>
    public Guid ID { get; }

    /// <summary>
    /// The type id of the artifact
    /// </summary>
    public uint TypeID { get; }

    /// <summary>
    /// The parent file ID where the artifact came from
    /// </summary>
    public Guid FileID { get; }

    /// <summary>
    /// The media id of the media where this file/artifact came from
    /// </summary>
    public Guid MediaID { get; }

    /// <summary>
    /// The object id
    /// </summary>
    public Guid ObjectID { get; }

    /// <summary>
    /// The object type
    /// </summary>
    public uint ObjectType { get; }

    /// <summary>
    /// The data type
    /// </summary>
    public uint DataType { get; }

    /// <summary>
    /// The offset into the file where the artifact came from
    /// -1 if this offset cannot be derived
    /// </summary>
    public ulong Offset { get; }

    /// <summary>
    /// The length in bytes of this artifact
    /// </summary>
    public ulong Length { get; }

    /// <summary>
    /// The value of the artifact
    /// </summary>
    public string Value { get; }

    /// <summary>
    /// The original message for this artifact
    /// </summary>
    public TruxtonMessage OriginalMessage { get; }
}

ICameraInfoConsumer

If your ETL implements the ICameraInfoConsumer interface it will receive messages pertaining to metadata like EXIFs information when media is being processed. Camera info could be to metadata generated from cameras - either static or video. See the EXIF_Table entry for information that can be exploited.

Here is the ICameraInfoConsumer interface and CameraInfoMessage it consumes:

/// <summary>
/// This ETL is a consumer of Camera Info messages
/// </summary>
public interface ICameraInfoConsumer : IETLMessageConsumer<CameraInfoMessage>
{
}

/// <summary>
/// Represents EXIF or video metadata extracted in Truxton.
/// </summary>
public class CameraInfoMessage
{
    /// <summary>
    /// The ID this CameraInfo
    /// </summary>
    public Guid ID { get; }

    /// <summary>
    /// The id for this file where this CameraInfo was found
    /// </summary>
    public Guid FileID { get; }

    /// <summary>
    /// The id for this media for this File/ camera info
    /// </summary>
    public Guid MediaID { get; }

    /// <summary>
    /// The 32 characters md5 hash of this file where this camera info came from
    /// </summary>
    public string MD5 { get; }

    /// <summary>
    /// The WGS84 latitude of where the image was taken.
    /// If empty will be 0.0
    /// </summary>
    public double Latitude { get; }

    /// <summary>
    /// The WGS84 longitude of where the image was taken.
    /// If empty will be 0.0
    /// </summary>
    public double Longitude { get; }

    /// <summary>
    /// The altitude in meters of where the image was taken.
    /// If empty will be 0.0
    /// </summary>
    public double Altitude { get; }

    /// <summary>
    /// The compass heading of the device that produced the image.
    /// </summary>
    public double Heading { get; }

    /// <summary>
    /// The 35mm focal length of the device that produced the image.
    /// </summary>
    public double FocalLength { get; }

    /// <summary>
    /// The number of times the camera has taken a picture.
    /// </summary>
    public uint ShutterCount { get; }

    /// <summary>
    /// The date and time taken from the GPS on the device that produced the image.
    /// </summary>
    public DateTime GPSTime { get; }

    /// <summary>
    /// The date and time from the internal clock on the device that produced the image.
    /// </summary>
    public DateTime DeviceTime { get; }

    /// <summary>
    /// The make of the device that produced the image.
    /// </summary>
    public string Make { get; }

    /// <summary>
    /// The model of the device that produced the image.
    /// </summary>
    public string Model { get; }

    /// <summary>
    /// The serial number of the body of the device that produced the image.
    /// </summary>
    public string BodySerialNumber { get; }

    /// <summary>
    /// The serial number of the lens of the device that produced the image.
    /// </summary>
    public string LensSerialNumber { get; }

    /// <summary>
    /// The offset into the file where the EXIF data block began.
    /// </summary>
    public ulong Offset { get; }

    /// <summary>
    /// The original message for this Camera Info
    /// </summary>
    public TruxtonMessage OriginalMessage { get; }
}

ICommunicationConsumer

If your ETL implements the ICommunicationConsumer interface it will receive messages pertaining to communications exploited from files like SMS, Chat Programs or Email. See Message_Types for the different kinds of Communications which can be exploited from media.

HEADS UP This consumer is still a work in progress.

Here is the ICommunicationConsumer interface and CommunicationMessage it consumes:

/// <summary>
/// This ETL is a consumer of Communication messages
/// </summary>
public interface ICommunicationConsumer : IETLMessageConsumer<CommunicationMessage>
{
}

/// <summary>
/// Represents a communication that occured. Like sms or email
/// </summary>
public class CommunicationMessage
{
    /// <summary>
    /// The ID of this communication
    /// </summary>
    public Guid ID { get; }

    /// <summary>
    /// The id of the file where this communication was found
    /// </summary>
    public Guid FileID { get; }

    /// <summary>
    /// The id of the media where this file/communication was found
    /// </summary>
    public Guid MediaID { get; }

    /// <summary>
    /// The type id of this communication
    /// </summary>
    public short TypeID { get; }
        
    /// <summary>
    /// When this communication was sent
    /// </summary>
    public DateTime Sent { get; }

    /// <summary>
    /// When this communication was received
    /// </summary>
    public DateTime Received { get; }

    /// <summary>
    /// The original message for this Communication
    /// </summary>
    public TruxtonMessage OriginalMessage { get; }
}

IEventConsumer

If your ETL implements the IEventConsumerinterface it will receive messages pertaining to notable points in time found in the media being processed. This is slated more towards notable events and not include 'noisy' timestamps like file created times. See Event_Types for the different kinds of Events which can be exploited from the media.

Here is the IEventConsumer interface and EventMessage it consumes:

/// <summary>
/// This ETL is a consumer of Event messages
/// </summary>
public interface IEventConsumer : IETLMessageConsumer<EventMessage>
{
}

/// <summary>
/// This represents an Event found in Truxton.
/// </summary>
public class EventMessage
{
    /// <summary>
    /// The ID of the event
    /// </summary>
    public Guid ID { get; }

    /// <summary>
    /// The id of the file where this event came from
    /// </summary>
    public Guid FileID { get; }

    /// <summary>
    /// The id of the media where this file/event came from
    /// </summary>
    public Guid MediaID { get; }

    /// <summary>
    /// The type id of this event
    /// </summary>
    public uint TypeID { get; }

    /// <summary>
    /// The start time of this event
    /// </summary>
    public DateTime Start { get; }

    /// <summary>
    /// The end time of this event
    /// </summary>
    public DateTime End { get; }

    /// <summary>
    /// The title of this event
    /// </summary>
    public string Title { get; }

    /// <summary>
    /// The description of this event
    /// </summary>
    public string Description { get; }

    /// <summary>
    /// The original message for this Event
    /// </summary>
    public TruxtonMessage OriginalMessage { get; }
}

ILocationConsumer

If your ETL implements the ILocationConsumer interface it will receive messages pertaining to geographic coordinates found in the media being processed. See Location_Types for the different kinds of Locations which can be exploited from the media.

Here is the ILocationConsumer interface and LocationMessage it consumes:

    /// <summary>
    /// This ETL is a consumer of Location messages
    /// </summary>
    public interface ILocationConsumer : IETLMessageConsumer<LocationMessage>
    {
    }

    /// <summary>
    /// Represents a geographic location in Truxton
    /// </summary>
    public class LocationMessage
    {
        /// <summary>
        /// The id of the location
        /// </summary>
        public Guid ID { get; }

        /// <summary>
        /// The id of the file where this location was found
        /// </summary>
        public Guid FileID { get; }

        /// <summary>
        /// The id of the media where the file/location was found
        /// </summary>
        public Guid MediaID { get; }
        
        /// <summary>
        /// The type id of the location
        /// </summary>
        public uint TypeID { get; }

        /// <summary>
        /// The WGS84 latitude of the location
        /// </summary>
        public double Latitude { get; }

        /// <summary>
        /// The WGS84 longitude of the location
        /// </summary>
        public double Longitude { get; }

        /// <summary>
        /// The altitude in meters of the location
        /// </summary>
        public double Altitude { get; }

        /// <summary>
        /// The date of the location.
        /// </summary>
        public DateTime When { get; }

        /// <summary>
        /// The label of the location
        /// </summary>
        public string Label { get; }

        /// <summary>
        /// The original message for this location
        /// </summary>
        public TruxtonMessage OriginalMessage { get; }
    }

IRelationConsumer

If your ETL implements the IRelationConsumerinterface it will receive messages pertaining to relations associated together in the media being processed.

Here is the IRelationConsumer interface and RelationMessage it consumes:

    /// <summary>
    /// This ETL is a consumer of Relation messages
    /// </summary>
    public interface IRelationConsumer : IETLMessageConsumer<RelationMessage>
    {
    }

    /// <summary>
    /// This represents a relation in Truxton. It corresponds to the Relation table.
    /// </summary>
    public class RelationMessage
    {
        /// <summary>
        /// This corresponds to the [A_ID] column of the [Relation] table.
        /// </summary>
        public Guid A_ID { get; }

        /// <summary>
        /// This corresponds to the [B_ID] column of the [Relation] table.
        /// </summary>
        public Guid B_ID { get; }

        /// <summary>
        /// This corresponds to the [A_ObjectTypeID] column of the [Relation] table.
        /// </summary>
        public ushort A_ObjectTypeID { get; }

        /// <summary>
        /// This corresponds to the [B_ObjectTypeID] column of the [Relation] table.
        /// </summary>
        public ushort B_ObjectTypeID { get; }

        /// <summary>
        /// This corresponds to the [RelationTypeID] column of the [Relation] table.
        /// </summary>
        public uint RelationTypeID { get; }

        /// <summary>
        /// This corresponds to the [SourceID] column of the [Relation] table.
        /// </summary>
        public Guid SourceID { get; }

        /// <summary>
        /// This corresponds to the [Source_ObjectTypeID] column of the [Relation] table.
        /// </summary>
        public ushort Source_ObjectTypeID { get; }

        /// <summary>
        /// The original message for this relation
        /// </summary>
        public TruxtonMessage OriginalMessage { get; }
    }

ITagConsumer

If your ETL implements the ITagConsumer interface it will receive messages pertaining to tags being associated in the media being processed.

Here is the ITagConsumer interface and TagMessage it consumes:

    /// <summary>
    /// This ETL is a consumer of Tag messages
    /// </summary>
    public interface ITagConsumer : IETLMessageConsumer<TagMessage>
    {
    }

    /// <summary>
    /// Represents a tagged item in Truxton. It corresponds to the Tagged table.
    /// </summary>
    public class TagMessage
    {
        /// <summary>
        /// The globally unique identifier of the item that was tagged.
        /// This corresponds to the [ItemID] column of the [Tagged] table.
        /// </summary>
        public Guid ItemID { get; }

        /// <summary>
        /// This corresponds to the [ObjectTypeID] column of the [Tagged] table.
        /// </summary>
        public uint ItemObjectType { get; }

        /// <summary>
        /// This corresponds to the [TagID] column of the [Tagged] table.
        /// </summary>
        public Guid TagID { get; }

        /// <summary>
        /// This corresponds to the [MediaID] column of the [Tagged] table.
        /// </summary>
        public Guid MediaID { get; }

        /// <summary>
        /// This corresponds to the [Reason] column of the [Tagged] table.
        /// </summary>
        public string Why { get; }

        /// <summary>
        /// This corresponds to the [Source] column of the [Tagged] table.
        /// </summary>
        public uint Source { get; }

        /// <summary>
        /// The original message for this tag
        /// </summary>
        public TruxtonMessage OriginalMessage { get; }
    }

IWebsiteConsumer

If your ETL implements the IWebsiteConsumerinterface it will receive messages pertaining to websites which were visited in the media being processed.

Here is the IWebsiteConsumer interface and WebsiteMessage it consumes:

/// <summary>
/// This ETL is a consumer of Website messages
/// </summary>
public interface IWebsiteConsumer : IETLMessageConsumer<WebsiteMessage>
{
}

/// <summary>
/// Represents a visit to a website in truxton
/// </summary>
public class WebsiteMessage
{
    /// <summary>
    /// The id for this website visit
    /// </summary>
    public Guid ID { get; }

    /// <summary>
    /// The id of the file where this website visit was found
    /// </summary>
    public Guid FileID { get; }

    /// <summary>
    /// The id of the media where this file/website visit was found
    /// </summary>
    public Guid MediaID { get; }

    /// <summary>
    /// The type id of this website
    /// </summary>
    public uint Type { get; }

    /// <summary>
    /// When this website visit occured
    /// </summary>
    public DateTime When { get; }

    /// <summary>
    /// The account used to access this website visit
    /// </summary>
    public string Account { get; }

    /// <summary>
    /// The url of this website visit
    /// </summary>
    public string URL { get; }

    /// <summary>
    /// The method type used to visit this website
    /// </summary>
    public int MethodID { get; }

    /// <summary>
    /// The offset of the account used where it was found
    /// </summary>
    public ulong AccountOffset { get; }

    /// <summary>
    /// The offset into the file where this URL record was found.
    /// </summary>
    public ulong URLOffset { get; }

    /// <summary>
    /// The local file name where this URL is stored on the media.
    /// </summary
    public string LocalFilename { get; }

    /// <summary>
    /// The original message for this website
    /// </summary>
    public TruxtonMessage OriginalMessage { get; }
}

IMediaConsumer

If your ETL implements the IMediaConsumerinterface it will receive messages pertaining to media that is progressing thru the load and exploited phases. See ETL_Stages for information about how media interacts with different stages and their states. This kind of consumer is useful when you want it 'run one' per loaded media.

Here is the IMediaConsumer interface and MediaMessage it consumes:

/// <summary>
/// This ETL is a consumer of Media messages
/// </summary>
public interface IMediaConsumer : IETLMessageConsumer<MediaMessage>
{
}
    
/// <summary>
/// Represents a media being processed in truxton
/// </summary>
public class MediaMessage
{
    /// <summary>
    /// The id of the media 
    /// </summary>
    public Guid MediaID { get; }

    /// <summary>
    /// The original message for this item
    /// </summary>
    public TruxtonMessage OriginalMessage { get; }
}

IETL

The last consumer is a generic catch-all for all messages. You still have to configure which File_Types_Supported you care about and is generally only useful for compatibility with legacy etls.

Here is the IETL interface

/// <summary>
/// A catch all consumer which will just recieve TruxtonMessages
/// </summary>
public interface IETL : IETLMessageConsumer<TruxtonMessage>
{
    /// <summary>
    /// Returns which filetypes this ETL cares about
    /// </summary>
    /// <param name="ct"></param>
    /// <returns></returns>
    Task<IEnumerable<ushort>> ConfigureFileTypesAsync(CancellationToken ct);
}

The TruxtonMessage class will have a mapping identical to C++ TruxtonMessage

The ETL Host

Borrowing from .NET Core concepts, Truxton .NET Core ETLs are 'hosted' inside an IETLHost. This provides some normalization when it comes to things like configuration and injecting dependencies.

Here is an excerpt from the IETLHost interface:

/// <summary>
/// A host that hosts our ETL
/// </summary>
/// <typeparam name="TETLMessageConsumer"></typeparam>
public interface IETLHost<TETLMessageConsumer> 
        where TETLMessageConsumer : class, IETLMessageConsumer
{
    /// <summary>
    /// The consumer implemented
    /// </summary>
    TETLMessageConsumer ETL { get; }

    /// <summary>
    /// Adds a configuration handler which will be invoked when .Run() is called
    /// </summary>
    /// <param name="configure">A configuration handler with our consumer</param>
    /// <returns>self</returns>
    IETLHost<TETLMessageConsumer> Configure(Action<TETLMessageConsumer> configure);

    /// <summary>
    /// Adds a configuration handler which will be invoked when .Run() is called
    /// </summary>
    /// <param name="configure">A configuration handler with our consumer and service collection</param>
    /// <returns>self</returns>
    IETLHost<TETLMessageConsumer> Configure(Action<TETLMessageConsumer, IServiceProvider> configure);

    /// <summary>
    /// Runs the host and the consumer, which recieves messages from Truxton. 
    /// THIS METHOD WILL BLOCK
    /// </summary>
    void Run();

    /// <summary>
    /// Initializes the host, invoke it before run
    /// </summary>
    /// <param name="ct"></param>
    /// <returns></returns>
    Task InitializeAsync(CancellationToken ct);

    ...
}

To make constructing the IETLHost easier, Truxton provides a fluent api class named IETLHostBuilder:

    /// <summary>
    /// Fluent api class to build an IETLHost
    /// </summary>
    /// <typeparam name="TETLMessageConsumer">The type of Consumer to host</typeparam>
    /// <typeparam name="TETLHostBuilderContext">The Host builder context</typeparam>
    public interface IETLHostBuilder<TETLMessageConsumer, TETLHostBuilderContext>
        where TETLMessageConsumer : class, IETLMessageConsumer
        where TETLHostBuilderContext : class, IETLHostBuilderContext
    {
        /// <summary>
        /// Allows you to configure your own dependencies to be injected via handler
        /// </summary>
        /// <param name="configureServices">A handler which configures dependencies to be injected</param>
        /// <returns>self</returns>
        IETLHostBuilder<TETLMessageConsumer, TETLHostBuilderContext> ConfigureServices(Action<IServiceCollection> configureServices);

        /// <summary>
        /// Allows you to configure your own dependencies to be injected via handler
        /// </summary>
        /// <param name="configureServices">A handler which configures dependencies to be injected</param>
        /// <returns>self </returns>
        IETLHostBuilder<TETLMessageConsumer, TETLHostBuilderContext> ConfigureServices(Action<TETLHostBuilderContext, IServiceCollection> configureServices);

        /// <summary>
        /// Builds the host asynchronously
        /// </summary>
        /// <param name="ct">A cancellation token to nofity async cancellation</param>
        /// <returns>A constructed IETLHost</returns>
        Task<IETLHost<TETLMessageConsumer>> BuildAsync(CancellationToken ct = default);

        /// <summary>
        /// Builds the host
        /// </summary>
        /// <returns>A constructed IETLHost</returns>
        IETLHost<TETLMessageConsumer> Build();

        /// <summary>
        /// Builds the IETLHost and runs it. 
        /// THIS METHOD WILL BLOCK
        /// </summary>
        void Run();
    }

The concrete implementation of IETLHostBuilder recommended to use is the PostgresETLHostBuilder. This class is located in the Truxton.Messaging.Postgres.dll library. It injects a slew of helpful services and default configurations for most scenarios when interacting with Truxton.

Here is an example:

Suppose you had a IFileConsumer ETL that looked like such:

public class MyETL : IFileConsumer
{
    //implementation
}

To run your ETL, all you have to do is the following:

using Microsoft.Extensions.DependencyInjection;
using Truxton.Messaging;

namespace MyNamespace
{
    class Program
    {
        static void Main(string[] args)
        {
            new PostgresETLHostBuilder<MyETL>()
                .Run();
        }
    }
}

This will build, host and run the ETL and begin receiving and consuming relevant messages from Truxton while media is being processed. The next few sections will describe more advanced scenarios in working with the ETLHost.

Dependency Injection

This section will describe how you can inject dependencies into your ETL and assumes you have some rudimentary familiarity with .NET Core Dependency Injection using the ServiceCollection class.

In general it is highly recommended to encapsulate stateless application logic into a service and inject it into your ETL rather then handle all the business logic within your ETL. This helps enforce decoupling, single responsibility, code re-usability and testing.

Common Truxton Interfaces Which Can Be Injected

The PostgresETLHostBuilder registers and makes available several useful services you can then inject into your ETL. Here is a brief list and explanation of them:

• ITruxtonEnvironment - Found in TruxtonAssembly.dll. Provides useful information about the environment the ETL is being run in
• IConfiguration - A .NET Core configuration interface found in Microsoft.Extensions.Configuration.Abstractions.dll. Helpful to setup configuration from outside sources
• ILogger - A .NET Core logging interface found in Microsoft.Extensions.Logging.Abstractions.dll. Preconfigured to log for you. Read more about logging in .NET Core
• IStatusUpdater<TEnum> - Found in Truxton.Messaging.dll. Handy little class to notify others about discrete stages.
• ITruxtonDatabase - Found in Truxton.DataAccess.dll. Provides access to read and write metadata.
• ITruxtonMessageBus - Found in Truxton.DataAccess.dll. Provides access to read information from the MessageBus (usually unnecessary).
• IETLContentService - Found in Truxton.Messaging.dll. Provides access to read and write content.
• IBroadcastService - Found in Truxton.Messaging.dll. Provides ability to broadcast new items to other ETLs.
• IETLApplicationLifetime - Found in Truxton.Messaging.dll. Encapsulates the lifetime of our ETL and access to the root CancellationToken to assist in exiting gracefully out of asynchronous operations.

So to use any of these already registered services you would add any number of them to the constructor of your ETL. Here is an example:

public class MyETL : IFileConsumer
{
    readonly IBroadcastService _broadcastService;
    readonly IETLContentService _contentService;
    readonly ITruxtonDatabase _database;
    readonly ILogger _logger; 

    public MyETL(IBroadcastService broadcastService,
        IETLContentService contentService,
        ITruxtonDatabase database,
        ILogger<TextExtractETL> logger)
    {
        _broadcastService = broadcastService;
        _contentService = contentService;
        _database = database;
        _logger = logger;
    }

    ...
}

If you have other dependencies either from your own code or external sources please read the next section.

Injecting Your Own Dependencies

Inevitably you will need to add some custom injection to your ETL either be from outside libraries or your own domain logic. The ETLHostBuilder provides an easy way to inject and configure your own dependencies prior to the ETL being run.

Suppose you have a service that looks like this:

public class IFileExploiter { }
public class MyCoolFileExploiter : IFileExploiter { }

And we want to use that in our IFileConsumer. We add it during the builder phase:

class Program
{
    static void Main(string[] args)
    {
        new PostgresETLHostBuilder<MyETL>()
            .ConfigureServices(services =>
            {
                services.AddSingleton<IFileExploiter, MyCoolFileExploiter >();
            })
            .Run();
    }
}

Now this service is available to be injected anywhere throughout the lifetime of the ETL. This could be in the ETL itself, or perhaps a child dependency. Assign it to a field in your ETL constructor:

public class MyETL : IFileConsumer
{
    readonly IFileExploiter _fileExploiter;
        
    public MyETL(IFileExploiter fileExploiter)
    {
        _fileExploiter = fileExploiter;
    }
    ...
}

and you are ready to use the methods available to your IFileExploiter.

Outside Configuration

Configuration can come from multiple places. Luckily the .NET Core IConfiguration interface allows us to easily enumerate these sources and find the desired key without much trouble. This sections assumes some familiarity with .NET Core configuration.

Configuration Priority

The following configuration sources take priority from highest to lowest. That is, if a configuration entry exists in the highest source listed, it will take priority over one from a lower source.

• Command Line Arguments (highest!) - See command line arguments for format
• Environment Variables prefixed with TRUXTON_ - E.G. TRUXTON_tempdir, but not the env-var FOO_tempdir or tempdir.
• Local json file appSettings.Development.json - Located in the current working directory if the application is built in DEBUG mode
• Local xml file EXENAME.config - Where EXENAME is the name of the binary. E.G. MyETL.config Located in the current working directory
• Local xml file TruxtonSettings.xml - Located in the current working directory
• Local json file appSettings.json - Located in the current working directory
• Global xml file TruxtonSettings.xml (lowest!) Located in C:\Program Data\Truxton\Settings\TruxtonSettings.xml

Getting configuration values

To get a configuration value from a known key, inject IConfiguration and inspect it. There are helpful extension methods Truxton provides by using Truxton.Configuration from TruxtonAssembly.dll

appSettings.json:

{
    "My_Key":"foo"
}

public class MyETL : IFileConsumer
{
    public MyETL(IConfiguration configuration)
    {
        string myValue = configuration.GetString("My_Key")
        //myValue will be "foo"
    }
}

Section Based Configuration

You might encounter a scenario where its nice to have several domain specific configurations for your ETL. In this scenario we can use section based configuration to get all the values at once.

appSettings.json:

{
    "MyETLConfig": { 
        "Foo" : "Bar",
        "Fizz" : 1234,
        "Buzz" : true
    }
}

Injecting the section:

    new PostgresETLHostBuilder<MyETl>()
        .ConfigureServices((ctx, services) =>
        {
            var myETLConfigSection = ctx.Configuration.GetSection("MyETLConfig");
            services.Configure<MyETLConfig>(myETLConfigSection );
        })
        .Run();

Now its available in our hosted application lifetime:

public MyETLConfig 
{
    public string Foo {get;set;}
    public int Fizz {get;set;}
    public bool Buzz {get;set;} 
}

public class MyETL : IFileConsumer
{
    public MyETL(MyETLConfig myETLConfig)
    {
        //myETLConfig.Foo will be "Bar" etc
    }
}

Common ETL Operations

This section will list a common set of scenarios you may encounter when writing an ETL and details how to implement a solution.

Getting Content

Inevitable when dealing with Files found from Truxton and passed to your ETL, you will need to access the content or the bytes which comprise said file.

The interface of interest to achieve this is the Truxton.Messaging.IETLContentService interface. Here is the method

/// <summary>
/// A content service with extra methods tailored for ETLs
/// </summary>
public interface IETLContentService : IContentService
{
    /// <summary>
    /// Gets a stream of a file from a FileMessage asynchronously
    /// </summary>
    /// <param name="message">A FileMessage to get a stream to</param>
    /// <param name="ct">A cancellation token to signal operation cancellation</param>
    /// <returns>A Stream of the content for this FileMessage</returns>
    Task<Stream> GetStreamAsync(FileMessage message, CancellationToken ct);

    ...
}

This interface is available as one of the default available services provided by the PostgresETLHostBuilder mentioned above (this means you do not need to register it yourself). Inject it into your consumer into your consumer and use it in your ETL consumer.

public class MyFileETL: IFileConsumer
{
     readonly IETLContentService _contentService;

     public MyFileETL(IETLContentService contentService)
     {
         _contentService = contentService;
     }

     ...
}

When you need to obtain a stream to a FileMessage you can do the following:

public class MyFileETL: IFileConsumer
{
    ...
    public async Task HandleMessageAsync(FileMessage message, CancellationToken ct)
    {
        using (var stream = await _contentService.GetStreamAsync(message, ct))
        {
            //you now have stream access to the bytes of the file!
        }
    }
    ... 
}

The stream you are given is a stream to the bytes of the file associated to that file message. Do whatever you like with it! Just remember best practices in your method body like always async/awaiting where possible and disposing of your IDisposables.

Adding Content without an associated file

In some cases you may want to permanent persist content to Truxton without it being associated to a file entry. You can achieve this using the same interface mentioned above: the IETLContentService except using a different method:

/// <summary>
/// A content service with extra methods tailored for ETLs
/// </summary>
public interface IETLContentService : IContentService
{
    ...
    /// <summary>
    /// Saves a stream of data into truxton. It is up to you to persist the response location somewhere.
    /// </summary>
    /// <param name="stream">The stream of data to save</param>
    /// <returns></returns>
    SavedContentResponse SaveStream(Stream stream);

    ...
}

And here is the SavedContentRespone class

/// <summary>
/// The response of a SaveStream operation from IETLContentService
/// </summary>
public class SavedContentResponse
{
    /// <summary>
    /// If this operation succeeded
    /// </summary>
    public bool Success { get; }

    /// <summary>
    /// The DepotID where the stream was saved
    /// </summary>
    public Guid DepotID { get; }

    /// <summary>
    /// The offset into the depot where the stream was saved
    /// </summary>
    public long Offset { get; }

    /// <summary>
    /// The length that was saved to the depot
    /// </summary>
    public long Length { get; }
}

So to save a block of content to truxton you can call it as such

byte[] byteArray = Encoding.ASCII.GetBytes("My super important content");
MemoryStream stream = new MemoryStream(byteArray);
SavedContentResponse response = await _contentService.SaveStream(stream);

If the operation was a success you can now persist the SavedContentResponse to somewhere else so you can reference it and recall your data at a later time.

Fetching Related Information

Most related information can be retrieved using the ITruxtonDatabase interface. Which is also available as a default injectable service:

public class MyFileETL: IFileConsumer
{
     readonly ITruxtonDatabase _database;

     public MyFileETL(ITruxtonDatabase database)
     {
         _database = database;
     }

     ...
}

The ITruxtonDatase interface has too many methods to list here but we'll list a method of interest here to fetch the parent file information for a FileMessage:

FileModel fileModel = await _database.GetFileAsync(message.ParentID, ct);

or if you needed to write a custom query to access the Truxton Database:

...
using (var connection = await _database.ConnectAsync(ct))
using (var command = connection.CreateCommand())
{
    command.CommandText = "SQL QUERY"
    using (var data_reader = await command.ExecuteReaderAsync(ct))
    {
        //enumerate the response of the data_reader
    }
}

Interfacing With External Programs Or WebAPIs

Many times there are applications out there that add the forensic functionality we care about, but aren't written in C#. Or perhaps theres an external webapi we can available from a SaaS we want to get the output from. Not a problem. We can call these outside resources, capture their output, then either add their result to truxton, or parse it and do something else with it.

External Executables

The primary way we can interface with external executables is thru the System.Diagnostics.Process class from .NET Core. Here is a sample using a made up external process that takes commandline arguments.

public class MyETL : IFileConsumer
{
    readonly IETLContentService _contentService;
    
    public MyETL(IETLContentService contentService)
    {
        _contentService = contentService;
    }

    public async Task HandleMessageAsync(FileMessage message,
            CancellationToken ct)
    {
        //write the content of the file to a temp file on disk
        string temporary_file_name;

        using (var stream = await _contentService.GetStreamAsync(message, ct))
        using (var file_contents = new StreamContent(stream, false))
        {
            temporary_file_name = Utilities.SaveToTempFile(Path.GetTempPath(), "MyETL_", file_contents);
        }

        string results = runExternalExe(temporary_file_name);
 
        //now do something with the results from the external exe
    }

    string runExternalExe(string filePath)
    {
        string exePath = "External.exe";
        var sb = new StringBuilder();

        using (var process = new Process())
        {
            process.StartInfo.FileName = exePath;
            process.StartInfo.Arguments = $"-f {filePath}";                
            process.StartInfo.WorkingDirectory = Path.GetDirectoryName(exePath);
            process.StartInfo.UseShellExecute = false;
            process.StartInfo.CreateNoWindow = true;
            process.StartInfo.WindowStyle = ProcessWindowStyle.Hidden;
            process.StartInfo.WindowStyle = ProcessWindowStyle.Normal;
            process.StartInfo.RedirectStandardOutput = true;
            process.EnableRaisingEvents = true;

            process.OutputDataReceived += new DataReceivedEventHandler
            (
                delegate (object sender, DataReceivedEventArgs e)
                {
                  sb.appendLine(e.Data);
                }
            );

            if (process.Start())
            {
                // Now tell .Net we want the callback (OutputDataReceived)
                // to be called when a line of text is written to stdout.
                process.BeginOutputReadLine();
                    
                //Wait until the processs has exited to continue
                process.WaitForExit();

                // Now tell .Net we will no longer read from stdout
                // This "closes" the BeginOutputReadLine() call
                process.CancelOutputRead();
            }

            process.Close();

            return sb.ToString();
        }
    }
}

Sometimes the external executable can take another file path as output which you could then read as a stream or into a byte[] buffer and do with it what you will.

WebApis

The primary way we can interface with webapis is thru the System.Net.Http.HttpClient class from .NET Core. Here is a sample using a made up webapi that takes a stream and returns a string body

public class MyETL : IFileConsumer
{
    readonly IETLContentService _contentService;
    readonly HttpClient _httpClient;
    
    public MyETL(IETLContentService contentService)
    {
        _contentService = contentService;
        _httpClient = new HttpClient();
    }

    public async Task HandleMessageAsync(FileMessage message,
            CancellationToken ct)
    {
        using (var stream = await _contentService.GetStreamAsync(message, ct))
        {
            string responseBody = await getResponseFromWebApi(stream, ct);
            //now do something with the results from the webapi. Perhaps deserialize the output?
        }
    }
        
    async Task<string> getResponseFromWebApi(Stream stream, CancellationToken ct)
    {
        string uri = "https://localhost:8888/api/fileExploiter";

        using (var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, uri))
        using (var content = new StreamContent(stream))
        {
            httpRequestMessage.Content = content;
            content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
 
            using (var response = await _httpClient.SendAsync(httpRequestMessage, ct))
            {
                return await response.Content.ReadAsStringAsync();
            }
        }
    } 
}

There are a plethora of SaaS webapis out there that can do amazing things like image recognition or OCR. As you can see it is very simple to integrate these offers and add extra value into Truxton.

Broadcasting To Others

OK! You've done the actually work of doing something interesting to a subscribed file. What if your business logic produced another file? Or wants to tag the file? This is where the IBroadcastService comes into play. Its purpose is to provide a set of APIs to notify others of work you did so they can react accordingly to it. The IBroadcastService is available in Truxton.Messaging.dll and is provided for you in the PostgresHostBuilder.

Lets provide a simple example of adding a new file a string in memory:

Here is the method we'll use:

public interface IBroadcastService
{
    ...

    /// <summary>
    /// Adds a new file child and notifies interested parties. The content will be assumed to be utf8 encoding.
    /// </summary>
    /// <param name="media_id">The media id to associate this file to</param>
    /// <param name="parent_file_id">The parent file id to associate this file to</param>
    /// <param name="content">The content of the file. This will be encoded using utf8.</param>
    /// <param name="name">The name of the file</param>
    /// <param name="file_type">The file type id of the file</param>
    /// <param name="origin">The origin of this file</param>
    /// <param name="created_filetime_ticks">When this file was created</param>
    /// <param name="accessed_filetime_ticks">When this file was last accessed</param>
    /// <param name="modified_filetime_ticks">When this file was last modified</param>
    /// <returns>The new file id created</returns>
    Guid AddFile(Guid media_id,
            Guid parent_file_id,
            string content,
            string name,
            ushort file_type,
            Origin origin,
            ulong created_filetime_ticks,
            ulong accessed_filetime_ticks,
            ulong modified_filetime_ticks);

    ...
}

And our application logic:

public class MyETL : IFileConsumer
{
    ...
    readonly IBroadcastService _broadcastService;
    public MyETL(IBroadcastService broadcastService)
    {
        _broadcastService = broadcastService
    }

    public async Task HandleMessageAsync(FileMessage message,
            CancellationToken ct)
    {
        //we did some cool things with the message, now lets add a file back into truxton
        string fileContent = "The results of my cool stuff";
        ulong now = (ulong)DateTime.Now.ToFileTimeUtc();

        // Save the results as a child file.
        _broadcastService.AddFile(message.MediaID,
                message.ID,
                fileContent,
                "MyFilename.txt",
                (ushort)FileTypes.Type_UTF8_Encoded_Text,
                Origin.Expanded,// origin is expanded because we want other ETLs to be able to subscribe to this file
                now,
                now,
                now);
    }
}

Now, any other ETL in our pipeline who cares about UTF8 txt files will be notified with a FileMessage and they can continue further exploitation. You can see how this can get very powerful for well known filetypes (zip archives) or files with rich exploitative metadata (sqlite files).

Sample ETL code

Here is a snippet of sample code that puts all the concepts discussed together. It subscribes to ascii text files and produces a unicode child file associated to it with the string "foobar" appended to it. This sample assumes you have the correct package references and logging configured as mentioned earlier in this article

using Microsoft.Extensions.Logging;
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Truxton;
using Truxton.DataAccess;
using Truxton.Messaging;

namespace MyETL
{
    class Program
    {
        static void Main(string[] args)
        {
            //create a postgres flavored host and run our ETL in it
            new PostgresETLHostBuilder<MyETL>()
                .Run();
        }
    }

    /// <summary>
    /// This sample ETL takes an ascii text file, appends "foobar" to the content found, then add that as a child 
    /// </summary>
    public class MyETL : IFileConsumer
    {
        /// <summary>
        /// We want files during the LIEC loop. 
        /// That is: Load, Identify, Extract, Carve
        /// </summary>
        public int Stage => (int)Stages.LIECLoop_Stage_Minimum;

        /// <summary>
        /// Our unique name for our ETL
        /// </summary>
        public string ApplicationName => "MyETL";

        /// <summary>
        /// The unique message queue name, remember, no spaces!
        /// </summary>
        public string MessageQueueName => "myetl";

        /// <summary>
        /// Our ETLs purpose
        /// </summary>
        public string Description => "MyETL cares about txt files";

        /// <summary>
        /// Options for this ETL, no need to customize this for this sample
        /// </summary>
        public ETLOptions Options { get; } = new ETLOptions();

        readonly ITruxtonDatabase _truxtonDatabase;
        readonly IETLContentService _etlContentService;
        readonly IBroadcastService _broadcastService;
        readonly ILogger _logger;

        public MyETL(ITruxtonDatabase truxtonDatabase,
            IBroadcastService broadcastService,
            IETLContentService etlContentService, 
            ILogger<MyETL> logger)
        {
            _truxtonDatabase = truxtonDatabase;
            _broadcastService = broadcastService;
            _etlContentService = etlContentService;
            _logger = logger;
        }

        /// <summary>
        /// Method which configures which file types we care to recieve
        /// </summary>
        /// <param name="ct"></param>
        /// <returns></returns>
        public Task<IEnumerable<ushort>> ConfigureFileTypesAsync(CancellationToken ct)
        {
            // we only care about ASCII files
            var fileTypes = new List<ushort>()
            {
                (ushort)FileTypes.Type_ASCII_Text,
            };

            return Task.FromResult<IEnumerable<ushort>>(fileTypes);
        }

        /// <summary>
        /// Main method which will handle incoming messages
        /// </summary>
        /// <param name="message"></param>
        /// <param name="ct"></param>
        /// <returns></returns>
        public async Task HandleMessageAsync(FileMessage message, CancellationToken ct)
        {
            _logger.LogDebug($@"Received File {message.Name}. Type: {message.FileType}");

            //use the content service to get a stream and read the contents to a string
            using (var stream = await _etlContentService.GetStreamAsync(message, ct))
            using (var reader = new StreamReader(stream, Encoding.ASCII))
            {
                string stringContent = await reader.ReadToEndAsync();
                _logger.LogDebug($"Content received: {stringContent}");

                string foobaredContent = stringContent + " foobar.";

                //our string has been foobared, lets add it back to Truxton via _broadcastService
                byte[] unicodeBuffer = Encoding.Unicode.GetBytes(foobaredContent);
                ulong now = (ulong)DateTime.Now.ToFileTimeUtc();

                // Save the results as a child file.
                _broadcastService.AddFile(message.MediaID,
                    message.ID,
                    unicodeBuffer,
                    $"{message.Name} - Foobared.txt",
                    (ushort)FileTypes.Type_Unicode_Text,
                    Origin.Expanded,// origin is expanded because we want other ETLs to be able to subscribe to this file
                    now,
                    now,
                    now);
            }
        }
    }
}