Truxton Forensic Rack

The Truxton Forensic Rack is a 7 foot tall rack populated with hardware dedicated to forensic processing. It has a Yellowbrick database appliance, seven dedicated loader/exploitation machines and 1PB of usable depot storage. With everything separated, it gives us many knobs to turn for loading.

Hardware

Here's how the hardware is stacked:

U42	Network Switch
U27	Yellowbrick
U26	Yellowbrick Manager 0
U25	Yellowbrick Manager 1
U23	KVM Server
U22	Keyboard and Monitor
U13	Loader Node 7
U11	Loader Node 6
U09	Loader Node 5
U07	Loader Node 4
U05	Loader Node 3
U03	Loader Node 2
U01	Loader Node 1

Installing Hardware

Normally you won't have to do this but should you want to move the hardware out of the existing rack an into a different one, populate the new rack from the bottom up.

1. Locate Server Node 1
   1. Install the rails for the server into the rack position
   2. Slide the server onto the rails
2. Repeat for the remaining servers working your way up the rack
3. Install rails for KVM
4. Install KVM
5. Install rails for Yellowbrick Management Node 1
6. Install Yellowbrick Management Node 1
7. Install rails for Yellowbrick Management Node 2
8. Install Yellowbrick Management Node 2
9. Install rails for Yellowbrick Server
10. Install Yellowbrick Server
11. Populate Yellowbrick Server with 9 blades
12. Fill the empty spaces in Yellowbrick Server with empty blade enclosures
13. Install rails for network switches
14. Install network switches
15. Add the hundreds of cables

Power Up Sequence

Generally speaking, you start the rack from the top down.

1. Turn on the network switches
2. Turn on the KVM switch
3. Connect the Yellowbrick blades to power then wait for fans to slow/quiet (4-5 mins to boot if blades off for 3+ days).
4. Connect a KVM Crash Cart to Yellowbrick Manager node 0.
5. Power up Yellowbrick Manager node 0 wait 30 seconds.
6. Power up Yellowbrick Manager node 1 wait 3 minutes.
7. Log into Yellowbrick as ybdadmin user using KVM monitor. Run the following command to verify everything is running:

   ybcli system status
   

8. Login to Yellowbrick SMC console from a browser as yellowbrick at your virtual IP YB URL.In the Truxton DB this URL is configured to be http://10.10.182.103. You should see at status OK on the dashboard.
9. Turn on the Servers (7 through 1)

It will take several more minutes for everything to boot.

Software

Server Configuration

Before installing software on the server, you should make the following changes:

• Stop virus scanning - When Truxton is processing media, it will encounter malware. Having a virus scanner running will cause Truxton to fail in mysterious ways. We have seen:
  • Processes freeze
  • Executables disappear from the installation folder
  • Services stop

Installing Software

Prior to software installation, several decisions must be made about where to store things on each node in the rack.

Where are temporary files going to go?

ETL processes may have to write files to a temporary location during forensic processing. This folder should ideally be on an SSD local to the node for speed. However, in your testing, having the temporary folder on the SAN may be fast enough.

Where are the hashsets going to go?

This should definitely be on an SSD local to the server. The hashset files are searched every time a file (a record in the [File] table) is added to Truxon. This will result in several IO operations being performed on the hashset file. Multiply this number of operations by the number of files being added and you can see why these operations must have as little latency as possible. Read/write speeds of the disk are far less important than seek time.

Where are the SOLR indexes going to go?

SOLR indexes are written when content is indexed in Truxton. If you are running the rack as a production server, SOLR should have its own storage volume on the SAN. If you are testing different configurations with the equipment in the rack, have SOLR store its contents on the Scratch drive.

Where are the software installation packages?

There will always be software that you want to install on the server nodes. It is best to put them on the SAN where you can maintain one location accessible by all nodes.

Where are the PostgreSQL databases going to go?

As with SOLR indexes, you must pick a location for the data files. You want to host them on an SSD because databases are notoriously IO intensive.

Defaults as Shipped

The Truxton Forensic Rack is delivered with the following defaults:

What	Where
Temporary Files	D:\Temp
Hash Sets	D:\Hashsets
SOLR	C:\Clusterstorage\Scratch\UXX\SOLR
Software Installers	C:\Clusterstorage\SampleData\Rack Management\Software
Databases	D:\Truxton Data\data

Each server node is pre-installed with Notepad++ and dBeaver tools.

Truxton was installed to write data to the D:\Truxton Data folder.

Populate Folders

Now that you have decided where to put files, take the time to populate the folders with your files. Copy whatever software you will want on the server nodes to the Software Installer folder. Copy hash sets to the Hash Sets folder, etc.

Helper Batch Files

The following are some batch files that ease the task of configuring Truxton. These batch files will be in the D:\ folder by default.

WARNING! These batch files presume you are trying different configurations and will wipe out data on the servers.

install.bat

This batch will start the installer process.

cleanup.bat

This batch file will prepare a server to have Truxton installed on it. It was designed to be run after Truxton was uninstalled from a machine. It will remove any lingering settings, temporary files and SOLR data.

ClearLocalDatabase.bat

This batch file will clear the [Truxton] database running on the local PostgreSQL server.

ClearLocalMessageBus.bat

This batch file will clear the [TruxtonMessageBus] database running on the local PostgreSQL server.

Initial Software Installation

Your first install of Truxton on a virgin rack server won't be like on a laptop. You should"

1. Start the installer (install.bat)
2. Do not use default options when installing Truxton
3. When the installer shows the options window, set Data Depot Directory to D:\Truxton Data
4. Finish the installer

Initial Postgres Installation

For the rack version of Truxton, we need to modify the default installation to expand the number of database connections. In the D:\Truxton Data\data folder, edit the postgresql.conf and change the follwing line to read:

max_connections = 1000			# (change requires restart)

After saving your changes, restart the Postgres service.

net stop postgresql-x64-12
net start postgresql-x64-12

Configuring Software

With this rack, you can test different configurations to meet your needs. There is no need to uninstall/reinstall Truxton in order to reconfigure it. The process of re-configuring a server is basically:

1. Stop Trutxon
2. Replace configuration files in C:\ProgramData\Truxton\Settings
3. Start Truxton

Default Truxton Configurations

The Truxton Forensic Rack comes with several different configurations on it. These are stored in folders in the C:\ClusterStorage\SampleData\Rack Management\Configurations folder. More details about configuring the rack can be found on the Loader Configuration page.

2x5 Configuration

The 2x5 configuration uses two machines to handle database and message bus, five servers for forensic exploitation.

iDRAC

The servers in the rack are Dell Poweredge machines that have Integraged Dell Remote Access Controller (iDRAC) support. This allows you to remotely manage the server as if you were physically there.

USB Connection

To access the iDRAC interface over the USB port:

• Connect to the server through the USB port under the power button on the right of the front panel.
• Turn off any wireless networks and disconnect from any other hard wired network.
• Ensure that the USB port is enabled. For more information, see Configuring USB management port settings.
• Wait for the laptop to acquire IP address 169.254.0.4. It may take several seconds for the IP addresses to be acquired. iDRAC acquires the IP address 169.254.0.3.
• Start using iDRAC network interfaces such as the web interface, RACADM, Redfish or WSMan.

To access the iDRAC web interface, open a supported browser, and type the address 169.254.0.3 and press enter. When iDRAC is using the USB port, the LED blinks indicating activity. The blink frequency is four per second. After completing the desired actions, disconnect the USB cable from the system. The LED turns off.

Storage

The storage in the rack uses Microsoft Storage Spaces Direct to present SAN storage to each of the server nodes as a folder on their local file system.

Top Level Mount Point

The mount point for SAN storage is C:\ClusterStorage All allocated drives on the SAN will appear here.

Allocated Drives

When you create a new drive on the SAN, it will appear as a sub-folder to C:\ClusterStorage For example, a drive was allocated on the SAN and named Depot To access this drive, you simply navigate to the C:\ClusterStorage\Depot folder.

Shares

The share exposed to allow clients access to depots is a Storage Spaces Direct resource. This means that the share can be serviced by any server node in the rack. The default share for Truxton is \\sofs-01.truxton.lab\Depot This share has been created for the C:\ClusterStorage\Depot\Depot (yes, two Depot) folder. You should not attempt to connect to the share using the IP address instead of the server name. Storage Spaces Direct will cycle through the server addresses to resolve sofs-01.truxton.lab

Gathering Performance Metrics

Tuning Truxton requires the ability to perform identical loads. The basic steps to do this are:

1. Perform a load
2. Gather metrics
3. Adjust parameters
4. Reset nodes
5. Go to step 1

Perform a Load

Loading a single piece of media will not provide good enough data to test the system. Using a load list will allow you to load many pieces of media. You can keep adding entries to the list until you reach the amount of data you want to test. On one of the server nodes, load the load list.

Server Performance Data

Live Optics is used to gather performance metrics from the operating system. Run the client data gatherer on the node that will be used to load the data. Start a new session. Once data is being gathered, run [Load] to process the load list.

Truxton Performance Data

Truxton records quite a bit of data about a load in the [Statistics] table in the database. By using the information recorded here, you can see how performance changes over time or you can find anomalous loader machines.

In the Rack Management/Database Scripts/Load Statistics folder, there is a Powershell script named GatherLoadStatistics.ps1 which will perform a series of queries to gather data and output it in a CSV and plain English format. It will report on the total resources used to process all media in the test load.

Putting It All Together

To test Truxton and get results:

1. Install Truxton - On each machine, run the Truxton_Installer.exe
2. Configure Truxton - 2x6config.bat
3. Turn on Monitoring - Live Optics
4. Load Data - Load your Load List
5. Gather load metrics - GatherLoadStatistics.ps1
6. Gather monitoring metrics - Live Optics

Checklist

1. Run Truxton_Installer.exe
   1. Do not use defaults
   2. Data Depot Directory: D:\Truxton Data
2. Stop Truxton
   1. net stop truxton
   2. net stop les
   3. Close the Tray App
3. Configure Truxton
4. Start Truxton
   1. net start truxton
   2. net start les
5. Load Something
   1. Load.exe LoadListName.txt
6. Wait until all media in list have loaded
7. GatherLoadStatistics.ps1

Single Node (Laptop Configuration)

The following steps are an example of turning the server at position U11 in the rack into an independent node.

1. Stop and uninstall Truxton
2. Run D:\Cleanup.bat
3. Run D:\ClearLocalDatabase.bat
4. Run D:\ClearLocalMessageBus.bat
5. Run D:\Install.cmd
   1. Do not use defaults
   2. Data Depot Directory: D:\Truxton Data
6. Stop Truxton
   1. net stop truxton
   2. net stop les
   3. Close the Tray App
7. Configure Truxton
   1. Copy D:\Truxton Data\Search to C:\ClusterStorage\Scratch\U11 (To your scratch directory)
   2. Delete the files in C:\ProgramData\Truxton\Settings
   3. Copy the U11_* files from C:\ClusterStorage\SampleData\Rack Management\Configurations\Laptop to C:\ProgramData\Truxton\Settings
   4. Rename the files in C:\ProgramData\Truxton\Settings to get rid of the preambles (U11_137_)
8. Ensure Virus Scanning won't interfere with Truxton
9. Ensure PostgreSQL can handle the number of connections
10. Start Truxton
    1. net start truxton
    2. net start les
11. Load Something
    1. Load.exe LoadListName.txt
12. Wait until all media in list have loaded
13. GatherLoadStatistics.ps1

2x5 Publish Configuration

The following steps detail how to reconfigure the rack so that all servers process data and finished loads publish up to the Yellowbrick.

1. On each server
   1. Stop and uninstall Truxton
   2. Run D:\Cleanup.bat
   3. Run D:\ClearLocalDatabase.bat
   4. Run D:\ClearLocalMessageBus.bat
   5. Run D:\Install.cmd
      1. Do not use defaults
      2. Data Depot Directory: D:\Truxton Data
   6. Stop Truxton
      1. net stop truxton
      2. net stop les
      3. Close the Tray App
2. Configure Truxton
   1. Copy D:\Truxton Data\Search to C:\ClusterStorage\Scratch\U11 (To your scratch directory)
   2. Delete the files in C:\ProgramData\Truxton\Settings
   3. Copy the U11_* files from C:\ClusterStorage\SampleData\Rack Management\Configurations\Laptop to C:\ProgramData\Truxton\Settings
   4. Rename the files in C:\ProgramData\Truxton\Settings to get rid of the preambles (U11_137_)
3. Ensure Virus Scanning won't interfere with Truxton
4. Ensure PostgreSQL can handle the number of connections
5. Start Truxton
   1. net start truxton
   2. net start les
6. Load Something
   1. Load.exe LoadListName.txt
7. Wait until all media in list have loaded
8. GatherLoadStatistics.ps1