Difference between revisions of "Why is Truxton so Extensible"

From truxwiki.com
Jump to navigation Jump to search
 
(11 intermediate revisions by the same user not shown)
Line 14: Line 14:
  
 
There will always be new types of files springing into existence.
 
There will always be new types of files springing into existence.
Suppose you have expended many hours creating your own file identification system that uses [https://mark0.net/soft-tridscan-e.html XML files] to describe file signatures.
+
Suppose you have spent many hours creating your own file identification system that uses [https://mark0.net/soft-tridscan-e.html XML files] to describe file signatures.
 
Here's how to use your custom file identifier in Truxton:
 
Here's how to use your custom file identifier in Truxton:
  
Line 30: Line 30:
 
# [[TruxtonService.xml#Adding_Your_Own_ETL_Process|Register your ETL]] with the [[Truxton Service]] so it will run
 
# [[TruxtonService.xml#Adding_Your_Own_ETL_Process|Register your ETL]] with the [[Truxton Service]] so it will run
  
==I have a better way to exploit this file type==
+
==I have a better way to exploit a file type==
 
You can replace Truxton's existing file processing by updating the <code><nowiki>[</nowiki>[[ETLRoute Table|ETLRoute]]<nowiki>]</nowiki></code> table.
 
You can replace Truxton's existing file processing by updating the <code><nowiki>[</nowiki>[[ETLRoute Table|ETLRoute]]<nowiki>]</nowiki></code> table.
 
Find the records matching the <code>FileTypeID</code> column and modify the <code>ETLQueueName</code> column to hold the name of the queue associated with your exploiter.
 
Find the records matching the <code>FileTypeID</code> column and modify the <code>ETLQueueName</code> column to hold the name of the queue associated with your exploiter.
  
If you created a better [[Type_CSV|CSV]] exploitation process, you can tell Truxton to use that instead of the default exploiter by using this SQL statement:
+
If you created a better [[Type_CSV|CSV]] exploitation process, you can tell Truxton to use that instead of the default exploiter by executing this SQL statement:
 
<source lang="sql">
 
<source lang="sql">
 
UPDATE "ETLRoute" SET "ETLQueueName" = 'bettercsv' WHERE "FileTypeID" = 241
 
UPDATE "ETLRoute" SET "ETLQueueName" = 'bettercsv' WHERE "FileTypeID" = 241
Line 46: Line 46:
 
==I want to use my own file carver==
 
==I want to use my own file carver==
 
While Truxton's [[Carve|file carver]] is good, it focuses on speed and accuracy, not extensibility.
 
While Truxton's [[Carve|file carver]] is good, it focuses on speed and accuracy, not extensibility.
Rather than plug into the existing carver, you would write your own carving ETL and set it to run as a [[ETL_Stages#Semi-Chaotic|Semi-Chaotic]] stage.
+
Rather than plug into the existing carver, you would write your own carving ETL and set it to run at a [[ETL_Stages#Semi-Chaotic|Semi-Chaotic]] stage.
 
When notified to run, your ETL would:
 
When notified to run, your ETL would:
# Query the <code>[Free]</code> table to gather the regions of free space
+
# Query the <code><nowiki>[</nowiki>[[Free Table|Free]]<nowiki>]</nowiki></code> table to gather the regions of free space
 
# Read the bytes in those regions and look for file signatures
 
# Read the bytes in those regions and look for file signatures
 
# Add any files found to Truxton (with an [[Origin|origin]] of <code>ORIGIN_CARVED</code>)
 
# Add any files found to Truxton (with an [[Origin|origin]] of <code>ORIGIN_CARVED</code>)
Line 54: Line 54:
 
If you want to replace Truxton's default file carver with your own, you would:  
 
If you want to replace Truxton's default file carver with your own, you would:  
 
# Create an ETL that listens to the "carve" message queue
 
# Create an ETL that listens to the "carve" message queue
# Replace the [[TruxtonService.xml#Sample_Configuration_File|"carve"]] ETL in TruxtonService.xml to use your executable
+
# Replace the [[TruxtonService.xml#Sample_Configuration_File|"carve"]] ETL in [[TruxtonService.xml]] to use your executable
# Handle the [[Type_Truxton_Control_Message_Carve|carve message]] in your code
+
# Handle the [[Message_Bus_Messages#Carve|carve message]] in your code
 +
 
 +
==How do I know my code is working?==
 +
Writing forensic tools is hard.
 +
When a vendor fixes one part of their software, they may have silently broken something else.
 +
It sucks.
 +
Truxton is not immune to this behavior but we have a tool to help us detect these problems.
 +
[[Load_Validation|ValidateLoad]] is Truxton's tool that queries the database to detect missing or incorrect data.
 +
Here's how to use it.
 +
# Select a piece of media that you are familiar with ([https://www.khyrenz.com/resources Khyrenz] has a particularly devious test image)
 +
# Create a [[Load_Validation#Sample|validator data file]]
 +
# Load the media into Truxton
 +
# Execute [[Load_Validation|ValidateLoad]]
 +
 
 +
If everything is working, you should get no errors reported.
 +
If not, the validator will tell you exactly what is missing.
  
 
==I want to use my own User Interface==
 
==I want to use my own User Interface==
 
Truxton’s Analyst Desktop is a graphical user interface that queries a [[Database_Tables|database]] and puts the results onto the screen.
 
Truxton’s Analyst Desktop is a graphical user interface that queries a [[Database_Tables|database]] and puts the results onto the screen.
 
There’s nothing stopping you from creating your own interface or integrating Truxton into an existing UI.
 
There’s nothing stopping you from creating your own interface or integrating Truxton into an existing UI.
 +
Truxton's GUI is completely independent from the forensic processing and contains no exploitation capability.
  
 
The database schema is designed to be easily understood by humans.
 
The database schema is designed to be easily understood by humans.
Line 65: Line 81:
 
There are also some [[Useful_SQL_Queries|useful queries]] to get you started.
 
There are also some [[Useful_SQL_Queries|useful queries]] to get you started.
  
==I don't want a User Interface==
+
==I want results not a User Interface==
 
For minimally trained users that say:
 
For minimally trained users that say:
 
* I just want to push a button and see the reports.
 
* I just want to push a button and see the reports.
Line 72: Line 88:
  
 
The [[Easy Button Load#Sample_Scenario|Easy Button]] is the one-stop shop for Truxton forensics.
 
The [[Easy Button Load#Sample_Scenario|Easy Button]] is the one-stop shop for Truxton forensics.
You right button on a folder, then wait for a browser to pop up.
+
You right button on a top level folder, select Easy Button Load, then wait for a browser to pop up.
 
It will [[Easy Button Load#Determining_What_to_Load|find media]] to load in that folder.
 
It will [[Easy Button Load#Determining_What_to_Load|find media]] to load in that folder.
 
When the last piece of media completes, reports are generated and a web browser pops up with links to all of the reports.
 
When the last piece of media completes, reports are generated and a web browser pops up with links to all of the reports.
 +
 +
==I'm a small office, I don't need a big system==
 +
Truxton is a highly modular system.
 +
It can scale up by running more of the modules on one or more machines on your network.
 +
It can scale down by running fewer modules on a laptop.
 +
If you need to share an investigation with another office, you can send them a [[TPIF]] instead of raw media.

Latest revision as of 07:36, 23 October 2025

Well, to meet your needs, that’s why.

Introduction

The field of digital forensics is far too large to be mastered by any single vender, lab, organization, agency, or government. No single tool can meet all of a customer’s needs. Truxton’s extensible processing architecture allows you to integrate your existing tools and know-how into a scalable work flow. In short, Truxton is extensible so it can become part of your investigative workflow.

Sample Scenarios

Let’s demonstrate some of this extensibility with a few sample scenarios.

Truxton doesn’t identify file types I need

There will always be new types of files springing into existence. Suppose you have spent many hours creating your own file identification system that uses XML files to describe file signatures. Here's how to use your custom file identifier in Truxton:

  1. Reserve a region of file type values (will be recorded on this wiki)
  2. Create an algorithm to convert your file type identifier to a value that falls within your file type id region.
  3. Enumerate through your file types and add them to the [FileType] table. Here’s sample code in Python and C.
  4. Create an ETL process to receive unknown file types. Here’s sample code in Python or C.
  5. Register your ETL with Truxton. Add your ETL to the Truxton Service for it to automatically start with the other ETL processes.

Congratulations! You have now extended Truxton’s file identification capability to match your own.

Truxton doesn’t exploit my file types

If Truxton doesn’t know how to exploit a particular file type:

  1. Create an ETL process to exploit the file type. Here’s sample code in Python or C.
  2. Register your ETL with the Truxton Service so it will run

I have a better way to exploit a file type

You can replace Truxton's existing file processing by updating the [ETLRoute] table. Find the records matching the FileTypeID column and modify the ETLQueueName column to hold the name of the queue associated with your exploiter.

If you created a better CSV exploitation process, you can tell Truxton to use that instead of the default exploiter by executing this SQL statement:

UPDATE "ETLRoute" SET "ETLQueueName" = 'bettercsv' WHERE "FileTypeID" = 241

I want to export Truxton data in a custom JSON format

Truxton was designed to make it easy to get data out of it. You can use SQL for simple formats. For more complex formats, like JSON, use Python or C to enumerate through Truxton objects and do whatever you'd like with them.

I want to use my own file carver

While Truxton's file carver is good, it focuses on speed and accuracy, not extensibility. Rather than plug into the existing carver, you would write your own carving ETL and set it to run at a Semi-Chaotic stage. When notified to run, your ETL would:

  1. Query the [Free] table to gather the regions of free space
  2. Read the bytes in those regions and look for file signatures
  3. Add any files found to Truxton (with an origin of ORIGIN_CARVED)

If you want to replace Truxton's default file carver with your own, you would:

  1. Create an ETL that listens to the "carve" message queue
  2. Replace the "carve" ETL in TruxtonService.xml to use your executable
  3. Handle the carve message in your code

How do I know my code is working?

Writing forensic tools is hard. When a vendor fixes one part of their software, they may have silently broken something else. It sucks. Truxton is not immune to this behavior but we have a tool to help us detect these problems. ValidateLoad is Truxton's tool that queries the database to detect missing or incorrect data. Here's how to use it.

  1. Select a piece of media that you are familiar with (Khyrenz has a particularly devious test image)
  2. Create a validator data file
  3. Load the media into Truxton
  4. Execute ValidateLoad

If everything is working, you should get no errors reported. If not, the validator will tell you exactly what is missing.

I want to use my own User Interface

Truxton’s Analyst Desktop is a graphical user interface that queries a database and puts the results onto the screen. There’s nothing stopping you from creating your own interface or integrating Truxton into an existing UI. Truxton's GUI is completely independent from the forensic processing and contains no exploitation capability.

The database schema is designed to be easily understood by humans. All tables and columns are documented. There are also some useful queries to get you started.

I want results not a User Interface

For minimally trained users that say:

  • I just want to push a button and see the reports.
  • Don't confuse me with options and another GUI to figure out.
  • Please only show me the results.

The Easy Button is the one-stop shop for Truxton forensics. You right button on a top level folder, select Easy Button Load, then wait for a browser to pop up. It will find media to load in that folder. When the last piece of media completes, reports are generated and a web browser pops up with links to all of the reports.

I'm a small office, I don't need a big system

Truxton is a highly modular system. It can scale up by running more of the modules on one or more machines on your network. It can scale down by running fewer modules on a laptop. If you need to share an investigation with another office, you can send them a TPIF instead of raw media.