Truxton Forensic Rack
The Truxton Forensic Rack is a 7 foot tall rack populated with hardware dedicated to forensic processing. It has a Yellowbrick database appliance, seven dedicated loader/exploitation machines and 1PB of usable depot storage. With everything separated, it gives us many knobs to turn for loading.
Contents
Hardware
Here's how the hardware is stacked:
| U42 | Network Switch |
| U27 | Yellowbrick |
| U26 | Yellowbrick Manager 0 |
| U25 | Yellowbrick Manager 1 |
| U23 | KVM Server |
| U22 | Keyboard and Monitor |
| U13 | Loader Node 7 |
| U11 | Loader Node 6 |
| U09 | Loader Node 5 |
| U07 | Loader Node 4 |
| U05 | Loader Node 3 |
| U03 | Loader Node 2 |
| U01 | Loader Node 1 |
Installing Hardware
Normally you won't have to do this but should you want to move the hardware out of the existing rack an into a different one, populate the new rack from the bottom up.
- Locate Server Node 1
- Install the rails for the server into the rack position
- Slide the server onto the rails
- Repeat for the remaining servers working your way up the rack
- Install rails for KVM
- Install KVM
- Install rails for Yellowbrick Management Node 1
- Install Yellowbrick Management Node 1
- Install rails for Yellowbrick Management Node 2
- Install Yellowbrick Management Node 2
- Install rails for Yellowbrick Server
- Install Yellowbrick Server
- Populate Yellowbrick Server with 9 blades
- Fill the empty spaces in Yellowbrick Server with empty blade enclosures
- Install rails for network switches
- Install network switches
- Add the hundreds of cables
Power Up Sequence
Generally speaking, you start the rack from the top down.
- Turn on the network switches
- Turn on the KVM switch
- Connect the Yellowbrick blades to power then wait for fans to slow/quiet (4-5 mins to boot if blades off for 3+ days).
- Connect a KVM Crash Cart to Yellowbrick Manager node 0.
- Power up Yellowbrick Manager node 0 wait 30 seconds.
- Power up Yellowbrick Manager node 1 wait 3 minutes.
- Log into Yellowbrick as
ybdadminuser using KVM monitor. Run the following command to verify everything is running:ybcli system status
- Login to Yellowbrick SMC console from a browser as
yellowbrickat your virtual IP YB URL.In the Truxton DB this URL is configured to be http://10.10.182.103. You should see at status OK on the dashboard. - Turn on the Servers (7 through 1)
It will take several more minutes for everything to boot.
Software
Installing Software
Prior to software installation, several decisions must be made about where to store things on each node in the rack.
Where are temporary files going to go?
ETL processes may have to write files to a temporary location during forensic processing. This folder should ideally be on an SSD local to the node for speed. However, in your testing, having the temporary folder on the SAN may be fast enough.
Where are the hashsets going to go?
This should definitely be on an SSD local to the server.
The hashset files are searched every time a file (a record in the [File] table) is added to Truxon.
This will result in several IO operations being performed on the hashset file.
Multiply this number of operations by the number of files being added and you can see why these operations must have as little latency as possible.
Read/write speeds of the disk are far less important than seek time.
Where are the SOLR indexes going to go?
SOLR indexes are written when content is indexed in Truxton. If you are running the rack as a production server, SOLR should have its own storage volume on the SAN. If you are testing different configurations with the equipment in the rack, have SOLR store its contents on the Scratch drive.
Where are the software installation packages?
There will always be software that you want to install on the server nodes. It is best to put them on the SAN where you can maintain one location accessible by all nodes.
Where are the PostgreSQL databases going to go?
As with SOLR indexes, you must pick a location for the data files. You want to host them on an SSD because databases are notoriously IO intensive.
Defaults as Shipped
The Truxton Forensic Rack is delivered with the following defaults:
| What | Where |
|---|---|
| Temporary Files | D:\Temp
|
| Hash Sets | D:\Hashsets
|
| SOLR | C:\Clusterstorage\Scratch\UXX\SOLR
|
| Software Installers | C:\Clusterstorage\SampleData\Rack Management\Software
|
| Databases | D:\Truxton Data\data
|
Each server node is pre-installed with Notepad++ and dBeaver tools.
Truxton was installed to write data to the D:\Truxton Data folder.
Populate Folders
Now that you have decided where to put files, take the time to populate the folders with your files. Copy whatever software you will want on the server nodes to the Software Installer folder. Copy hash sets to the Hash Sets folder, etc.
Helper Batch Files
The following are some batch files that ease the task of configuring Truxton.
These batch files will be in the D:\ folder by default.
WARNING! These batch files presume you are trying different configurations and will wipe out data on the servers.
install.bat
This batch will start the installer process.
cleanup.bat
This batch file will prepare a server to have Truxton installed on it. It was designed to be run after Truxton was uninstalled from a machine. It will remove any lingering settings, temporary files and SOLR data.
ClearLocalDatabase.bat
This batch file will clear the [Truxton] database running on the local PostgreSQL server.
ClearLocalMessageBus.bat
This batch file will clear the [TruxtonMessageBus] database running on the local PostgreSQL server.
Initial Software Installation
Your first install of Truxton on a virgin rack server won't be like on a laptop. You should"
- Start the installer (
install.bat) - Do not use default options when installing Truxton
- When the installer shows the options window, set Data Depot Directory to
D:\Truxton Data - Finish the instaler
Configuring Software
With this rack, you can test different configurations to meet your needs. There is no need to uninstall/reinstall Truxton in order to reconfigure it. The process of re-configuring a server is basically:
- Stop Trutxon
- Replace configuration files in
C:\ProgramData\Truxton\Settings - Start Truxton
Default Truxton Configurations
The Truxton Forensic Rack comes with several different configurations on it.
These are stored in folders in the C:\ClusterStorage\SampleData\Rack Management\Configurations folder.
More details about configuring the rack can be found on the Loader Configuration page.
2x5 Configuration
The 2x5 configuration uses two machines to handle database and message bus, five servers for forensic exploitation.
Gathering Performance Metrics
Tuning Truxton requires the ability to perform identical loads. The basic steps to do this are:
- Perform a load
- Gather metrics
- Adjust parameters
- Reset nodes
- Go to step 1
Perform a Load
Loading a single piece of media will not provide good enough data to test the system. Using a load list will allow you to load many pieces of media. You can keep adding entries to the list until you reach the amount of data you want to test. On one of the server nodes, load the load list.
Server Performance Data
Live Optics is used to gather performance metrics from the operating system. Run the client data gatherer on the node that will be used to load the data. Start a new session. Once data is being gathered, run [Load] to process the load list.
Truxton Performance Data
Truxton records quite a bit of data about a load in the [Statistics] table in the database.
By using the information recorded here, you can see how performance changes over time or you can find anomalous loader machines.
In the Rack Management/Database Scripts/Load Statistics folder, there is a Powershell script named GatherLoadStatistics.ps1 which will perform a series of queries to gather data and output it in a CSV and plain English format.
Putting It All Together
To test Truxton and get results:
- Install Truxton
- Configure Truxton
- Turn on Monitoring
- Load Data
- Gather load metrics
- Gather monitoring metrics