Difference between revisions of "Truxton Forensic Rack"

From truxwiki.com
Jump to navigation Jump to search
 
(21 intermediate revisions by 3 users not shown)
Line 74: Line 74:
 
* [[Rack Reset|Reset Truxton]] - Clear out Truxton data
 
* [[Rack Reset|Reset Truxton]] - Clear out Truxton data
 
* [[Rack Install Truxton|Install Truxton]] - Installing a new release of Truxton
 
* [[Rack Install Truxton|Install Truxton]] - Installing a new release of Truxton
 +
* [[Rack First Load|First Load]] - Starting Truxton for loading data
  
 
=Software=
 
=Software=
Line 91: Line 92:
 
However, in your testing, having the temporary folder on the SAN may be fast enough.
 
However, in your testing, having the temporary folder on the SAN may be fast enough.
  
−
===Where are the [[Hash Set|hashsets]] going to go?===
+
===Where are the [[Hash Set|hash sets]] going to go?===
 
This should definitely be on an [https://en.wikipedia.org/wiki/Solid-state_drive SSD] local to the server.
 
This should definitely be on an [https://en.wikipedia.org/wiki/Solid-state_drive SSD] local to the server.
−
The hashset files are searched every time a file (a record in the <code><nowiki>[</nowiki>[[File Table|File]]<nowiki>]</nowiki></code> table) is added to Truxon.
+
The [[Hash Set|hash set]] files are searched every time a file (a record in the <code><nowiki>[</nowiki>[[File Table|File]]<nowiki>]</nowiki></code> table) is added to Truxton.
−
This will result in several [https://en.wikipedia.org/wiki/Input/output#Higher-level_implementation IO operations] being performed on the hashset file.
+
This will result in several [https://en.wikipedia.org/wiki/Input/output#Higher-level_implementation IO operations] being performed on the hash set file.
 
Multiply this number of operations by the number of files being added and you can see why these operations must have as little [https://en.wikipedia.org/wiki/Network_delay latency] as possible.
 
Multiply this number of operations by the number of files being added and you can see why these operations must have as little [https://en.wikipedia.org/wiki/Network_delay latency] as possible.
 
Read/write speeds of the disk are far less important than [https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics#Seek_time seek time.]
 
Read/write speeds of the disk are far less important than [https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics#Seek_time seek time.]
Line 143: Line 144:
  
 
==Helper Batch Files==
 
==Helper Batch Files==
−
The following are some batch files that ease the task of configuring Truxton.
+
There are two sets of batch files on the rack.
 +
The ones contained in <code>C:\ClusterStorage\SampleData\Rack Management</code> are designed to operate on the rack as a whole from any node.
 +
The ones in the <code>D:\</code> folder are meant to operated on only the machine they are executed from.
 +
 
 +
The following are some batch files that ease the task of configuring Truxton on a single node.
 
These batch files will be in the <code>D:\</code> folder by default.
 
These batch files will be in the <code>D:\</code> folder by default.
  
Line 172: Line 177:
 
==Initial Postgres Installation==
 
==Initial Postgres Installation==
 
For the rack version of Truxton, we need to modify the default installation to expand the number of database connections.
 
For the rack version of Truxton, we need to modify the default installation to expand the number of database connections.
−
In the <code>D:\Truxton Data\data</code> folder, edit the <code>postgresql.conf</code> and change the follwing line to read:
+
In the <code>D:\Truxton Data\data</code> folder, edit the <code>postgresql.conf</code> and change the following line to read:
 
<pre>max_connections = 1000 # (change requires restart)</pre>
 
<pre>max_connections = 1000 # (change requires restart)</pre>
 
After saving your changes, restart the Postgres service.
 
After saving your changes, restart the Postgres service.
Line 184: Line 189:
 
There is no need to uninstall/reinstall Truxton in order to reconfigure it.
 
There is no need to uninstall/reinstall Truxton in order to reconfigure it.
 
The process of re-configuring a server is basically:
 
The process of re-configuring a server is basically:
−
# Stop Trutxon
+
# Stop Truxton
 
# Replace configuration files in <code>C:\ProgramData\Truxton\Settings</code>
 
# Replace configuration files in <code>C:\ProgramData\Truxton\Settings</code>
 
# Start Truxton
 
# Start Truxton
Line 193: Line 198:
 
More details about configuring the rack can be found on the [[Loader Configuration#Truxton Forensic Rack|Loader Configuration]] page.
 
More details about configuring the rack can be found on the [[Loader Configuration#Truxton Forensic Rack|Loader Configuration]] page.
  
−
===2x5 Configuration===
+
Sample configurations are stored in the <code>C:\ClusterStorage\SampleData\Rack Management\Configurations</code> folder.
−
The 2x5 configuration uses two machines to handle database and message bus, five servers for forensic exploitation.
+
 
 +
===Laptop===
 +
The [[Loader Configuration#Laptop|laptop configuration]] is where all parts of Truxton are installed on a single machine.
 +
That would make for 7 "laptops" in the rack.
 +
 
 +
The files for this configuration are stored in the <code>Laptop</code> folder of <code>C:\ClusterStorage\SampleData\Rack Management\Configurations</code>.
 +
You can easily switch to this configuration by executing the <code>D:\LaptopConfig.bat</code> script.
 +
 
 +
===2x5 Auto Publish===
 +
The files for this [[Loader Configuration#2x5|configuration]] are stored in the <code>2x5 Auto Publish</code> folder of <code>C:\ClusterStorage\SampleData\Rack Management\Configurations</code>.
 +
The [[Loader Configuration#2x5|2x5 configuration]] uses two machines to handle database and message bus, five servers for forensic exploitation.
 +
Once a piece of media has completed loading, the meta data for that media are automatically published into the [https://www.yellowbrick.com Yellowbrick] appliance.
 +
You can easily switch to this configuration by executing the <code>D:\2x5AutoPublish.bat</code> script.
 +
 
 +
{| class="wikitable"
 +
|-
 +
| U01
 +
| Message Bus, SOLR
 +
|-
 +
| U03
 +
| Load Database
 +
|-
 +
| U05
 +
| Node 1
 +
|-
 +
| U07
 +
| Node 2
 +
|-
 +
| U09
 +
| Node 3
 +
|-
 +
| U11
 +
| Node 4
 +
|-
 +
| U13
 +
| Node 5
 +
|-
 +
| U27
 +
| Yellowbrick Enterprise Database
 +
|}
 +
 
 +
The five exploitation nodes process data into the Load Database which is standard vanilla Postgres.
 +
When exploitation completes on a piece of media, the meta data is shipped from the Load Database in U03 to Yellowbrick in U27.
 +
The meta data remains in U03 until it is no longer needed or wanted.
  
 
=iDRAC=
 
=iDRAC=
−
The servers in the rack are [https://www.dell.com Dell] [https://www.dell.com/en-us/work/shop/cty/pdp/spd/poweredge-r740xd2/pe_r740xd2_13235_vi_vp Poweredge] machines that have Integraged Dell Remote Access Controller ([https://www.delltechnologies.com/en-us/solutions/openmanage/idrac.htm iDRAC]) support.
+
The servers in the rack are [https://www.dell.com Dell] [https://www.dell.com/en-us/work/shop/cty/pdp/spd/poweredge-r740xd2/pe_r740xd2_13235_vi_vp Poweredge] machines that have Integrated Dell Remote Access Controller ([https://www.delltechnologies.com/en-us/solutions/openmanage/idrac.htm iDRAC]) support.
 
This allows you to remotely manage the server as if you were physically there.
 
This allows you to remotely manage the server as if you were physically there.
  
Line 223: Line 271:
 
When you create a new drive on the SAN, it will appear as a sub-folder to <code>C:\ClusterStorage</code>
 
When you create a new drive on the SAN, it will appear as a sub-folder to <code>C:\ClusterStorage</code>
 
For example, a drive was allocated on the SAN and named <code>Depot</code>
 
For example, a drive was allocated on the SAN and named <code>Depot</code>
−
To access this drive, you simply navigate to the <code>C:\ClusterStorage\Depot</code> folder.
+
To access this drive, navigate to the <code>C:\ClusterStorage\Depot</code> folder.
  
 
==Shares==
 
==Shares==
Line 265: Line 313:
 
To test Truxton and get results:
 
To test Truxton and get results:
 
# Install Truxton - On each machine, run the <code>Truxton_Installer.exe</code>
 
# Install Truxton - On each machine, run the <code>Truxton_Installer.exe</code>
−
# Configure Truxton - <code>2x6config.bat</code>
+
# Configure Truxton - <code>2x5config.bat</code>
 
# Turn on Monitoring - [https://liveoptics.com/ Live Optics]
 
# Turn on Monitoring - [https://liveoptics.com/ Live Optics]
 
# Load Data - Load your [[Load List]]
 
# Load Data - Load your [[Load List]]
Line 278: Line 326:
 
## <code>net stop truxton</code>
 
## <code>net stop truxton</code>
 
## <code>net stop les</code>
 
## <code>net stop les</code>
−
## Close the Tray App
+
## Close the [[Truxton_Status|Tray App]]
 
# Configure Truxton
 
# Configure Truxton
 
# Start Truxton
 
# Start Truxton
Line 300: Line 348:
 
## <code>net stop truxton</code>
 
## <code>net stop truxton</code>
 
## <code>net stop les</code>
 
## <code>net stop les</code>
−
## Close the Tray App
+
## Close the [[Truxton_Status|Tray App]]
 
# Configure Truxton
 
# Configure Truxton
 
## Copy <code>D:\Truxton Data\Search</code> to <code>C:\ClusterStorage\Scratch\U11</code> (To your scratch directory)
 
## Copy <code>D:\Truxton Data\Search</code> to <code>C:\ClusterStorage\Scratch\U11</code> (To your scratch directory)
Line 329: Line 377:
 
### <code>net stop truxton</code>
 
### <code>net stop truxton</code>
 
### <code>net stop les</code>
 
### <code>net stop les</code>
−
### Close the Tray App
+
### Close the [[Truxton_Status|Tray App]]
 
# Configure Truxton
 
# Configure Truxton
 
## Copy <code>D:\Truxton Data\Search</code> to <code>C:\ClusterStorage\Scratch\U11</code> (To your scratch directory)
 
## Copy <code>D:\Truxton Data\Search</code> to <code>C:\ClusterStorage\Scratch\U11</code> (To your scratch directory)

Latest revision as of 13:53, 30 April 2023

The Truxton Forensic Rack is a 7 foot tall rack populated with hardware dedicated to forensic processing. It has a Yellowbrick database appliance, seven dedicated loader/exploitation machines and 1PB of usable depot storage. With everything separated, it gives us many knobs to turn for loading.

Hardware

Here's how the hardware is stacked:

U42 Network Switch
U27 Yellowbrick
U26 Yellowbrick Manager 0
U25 Yellowbrick Manager 1
U23 KVM Server
U22 Keyboard and Monitor
U13 Loader Node 7
U11 Loader Node 6
U09 Loader Node 5
U07 Loader Node 4
U05 Loader Node 3
U03 Loader Node 2
U01 Loader Node 1

Installing Hardware

Normally you won't have to do this but should you want to move the hardware out of the existing rack an into a different one, populate the new rack from the bottom up.

  1. Locate Server Node 1
    1. Install the rails for the server into the rack position
    2. Slide the server onto the rails
  2. Repeat for the remaining servers working your way up the rack
  3. Install rails for KVM
  4. Install KVM
  5. Install rails for Yellowbrick Management Node 1
  6. Install Yellowbrick Management Node 1
  7. Install rails for Yellowbrick Management Node 2
  8. Install Yellowbrick Management Node 2
  9. Install rails for Yellowbrick Server
  10. Install Yellowbrick Server
  11. Populate Yellowbrick Server with 9 blades
  12. Fill the empty spaces in Yellowbrick Server with empty blade enclosures
  13. Install rails for network switches
  14. Install network switches
  15. Add the hundreds of cables

Procedures

Software

Server Configuration

Before installing software on the server, you should make the following changes:

  • Stop virus scanning - When Truxton is processing media, it will encounter malware. Having a virus scanner running will cause Truxton to fail in mysterious ways. We have seen:
    • Processes freeze
    • Executables disappear from the installation folder
    • Services stop

Installing Software

Prior to software installation, several decisions must be made about where to store things on each node in the rack.

Where are temporary files going to go?

ETL processes may have to write files to a temporary location during forensic processing. This folder should ideally be on an SSD local to the node for speed. However, in your testing, having the temporary folder on the SAN may be fast enough.

Where are the hash sets going to go?

This should definitely be on an SSD local to the server. The hash set files are searched every time a file (a record in the [File] table) is added to Truxton. This will result in several IO operations being performed on the hash set file. Multiply this number of operations by the number of files being added and you can see why these operations must have as little latency as possible. Read/write speeds of the disk are far less important than seek time.

Where are the SOLR indexes going to go?

SOLR indexes are written when content is indexed in Truxton. If you are running the rack as a production server, SOLR should have its own storage volume on the SAN. If you are testing different configurations with the equipment in the rack, have SOLR store its contents on the Scratch drive.

Where are the software installation packages?

There will always be software that you want to install on the server nodes. It is best to put them on the SAN where you can maintain one location accessible by all nodes.

Where are the PostgreSQL databases going to go?

As with SOLR indexes, you must pick a location for the data files. You want to host them on an SSD because databases are notoriously IO intensive.

Defaults as Shipped

The Truxton Forensic Rack is delivered with the following defaults:

What Where
Temporary Files D:\Temp
Hash Sets D:\Hashsets
SOLR C:\Clusterstorage\Scratch\UXX\SOLR
Software Installers C:\Clusterstorage\SampleData\Rack Management\Software
Databases D:\Truxton Data\data

Each server node is pre-installed with Notepad++ and dBeaver tools.

Truxton was installed to write data to the D:\Truxton Data folder.

Populate Folders

Now that you have decided where to put files, take the time to populate the folders with your files. Copy whatever software you will want on the server nodes to the Software Installer folder. Copy hash sets to the Hash Sets folder, etc.

Helper Batch Files

There are two sets of batch files on the rack. The ones contained in C:\ClusterStorage\SampleData\Rack Management are designed to operate on the rack as a whole from any node. The ones in the D:\ folder are meant to operated on only the machine they are executed from.

The following are some batch files that ease the task of configuring Truxton on a single node. These batch files will be in the D:\ folder by default.

WARNING! These batch files presume you are trying different configurations and will wipe out data on the servers.

install.bat

This batch will start the installer process.

cleanup.bat

This batch file will prepare a server to have Truxton installed on it. It was designed to be run after Truxton was uninstalled from a machine. It will remove any lingering settings, temporary files and SOLR data.

ClearLocalDatabase.bat

This batch file will clear the [Truxton] database running on the local PostgreSQL server.

ClearLocalMessageBus.bat

This batch file will clear the [TruxtonMessageBus] database running on the local PostgreSQL server.

Initial Software Installation

Your first install of Truxton on a virgin rack server won't be like on a laptop. You should"

  1. Start the installer (install.bat)
  2. Do not use default options when installing Truxton
  3. When the installer shows the options window, set Data Depot Directory to D:\Truxton Data
  4. Finish the installer

Initial Postgres Installation

For the rack version of Truxton, we need to modify the default installation to expand the number of database connections. In the D:\Truxton Data\data folder, edit the postgresql.conf and change the following line to read: 

max_connections = 1000			# (change requires restart)

After saving your changes, restart the Postgres service. 

net stop postgresql-x64-12
net start postgresql-x64-12

Configuring Software

With this rack, you can test different configurations to meet your needs. There is no need to uninstall/reinstall Truxton in order to reconfigure it. The process of re-configuring a server is basically:

  1. Stop Truxton
  2. Replace configuration files in C:\ProgramData\Truxton\Settings
  3. Start Truxton

Default Truxton Configurations

The Truxton Forensic Rack comes with several different configurations on it. These are stored in folders in the C:\ClusterStorage\SampleData\Rack Management\Configurations folder. More details about configuring the rack can be found on the Loader Configuration page.

Sample configurations are stored in the C:\ClusterStorage\SampleData\Rack Management\Configurations folder.

Laptop

The laptop configuration is where all parts of Truxton are installed on a single machine. That would make for 7 "laptops" in the rack.

The files for this configuration are stored in the Laptop folder of C:\ClusterStorage\SampleData\Rack Management\Configurations. You can easily switch to this configuration by executing the D:\LaptopConfig.bat script.

2x5 Auto Publish

The files for this configuration are stored in the 2x5 Auto Publish folder of C:\ClusterStorage\SampleData\Rack Management\Configurations. The 2x5 configuration uses two machines to handle database and message bus, five servers for forensic exploitation. Once a piece of media has completed loading, the meta data for that media are automatically published into the Yellowbrick appliance. You can easily switch to this configuration by executing the D:\2x5AutoPublish.bat script.

U01 Message Bus, SOLR
U03 Load Database
U05 Node 1
U07 Node 2
U09 Node 3
U11 Node 4
U13 Node 5
U27 Yellowbrick Enterprise Database

The five exploitation nodes process data into the Load Database which is standard vanilla Postgres. When exploitation completes on a piece of media, the meta data is shipped from the Load Database in U03 to Yellowbrick in U27. The meta data remains in U03 until it is no longer needed or wanted.

iDRAC

The servers in the rack are Dell Poweredge machines that have Integrated Dell Remote Access Controller (iDRAC) support. This allows you to remotely manage the server as if you were physically there.

USB Connection

To access the iDRAC interface over the USB port:

  • Connect to the server through the USB port under the power button on the right of the front panel.
  • Turn off any wireless networks and disconnect from any other hard wired network.
  • Ensure that the USB port is enabled. For more information, see Configuring USB management port settings.
  • Wait for the laptop to acquire IP address 169.254.0.4. It may take several seconds for the IP addresses to be acquired. iDRAC acquires the IP address 169.254.0.3.
  • Start using iDRAC network interfaces such as the web interface, RACADM, Redfish or WSMan.

To access the iDRAC web interface, open a supported browser, and type the address 169.254.0.3 and press enter. When iDRAC is using the USB port, the LED blinks indicating activity. The blink frequency is four per second. After completing the desired actions, disconnect the USB cable from the system. The LED turns off.

Storage

The storage in the rack uses Microsoft Storage Spaces Direct to present SAN storage to each of the server nodes as a folder on their local file system.

Top Level Mount Point

The mount point for SAN storage is C:\ClusterStorage All allocated drives on the SAN will appear here.

Allocated Drives

When you create a new drive on the SAN, it will appear as a sub-folder to C:\ClusterStorage For example, a drive was allocated on the SAN and named Depot To access this drive, navigate to the C:\ClusterStorage\Depot folder.

Shares

The share exposed to allow clients access to depots is a Storage Spaces Direct resource. This means that the share can be serviced by any server node in the rack. The default share for Truxton is \\sofs-01.truxton.lab\Depot This share has been created for the C:\ClusterStorage\Depot\Depot (yes, two Depot) folder. You should not attempt to connect to the share using the IP address instead of the server name. Storage Spaces Direct will cycle through the server addresses to resolve sofs-01.truxton.lab

Gathering Performance Metrics

Tuning Truxton requires the ability to perform identical loads. The basic steps to do this are:

  1. Perform a load
  2. Gather metrics
  3. Adjust parameters
  4. Reset nodes
  5. Go to step 1

Perform a Load

Loading a single piece of media will not provide good enough data to test the system. Using a load list will allow you to load many pieces of media. You can keep adding entries to the list until you reach the amount of data you want to test. On one of the server nodes, load the load list.

Server Performance Data

Live Optics is used to gather performance metrics from the operating system. Run the client data gatherer on the node that will be used to load the data. Start a new session. Once data is being gathered, run [Load] to process the load list.

Truxton Performance Data

Truxton records quite a bit of data about a load in the [Statistics] table in the database. By using the information recorded here, you can see how performance changes over time or you can find anomalous loader machines.

In the Rack Management/Database Scripts/Load Statistics folder, there is a Powershell script named GatherLoadStatistics.ps1 which will perform a series of queries to gather data and output it in a CSV and plain English format. It will report on the total resources used to process all media in the test load.

Putting It All Together

To test Truxton and get results:

  1. Install Truxton - On each machine, run the Truxton_Installer.exe
  2. Configure Truxton - 2x5config.bat
  3. Turn on Monitoring - Live Optics
  4. Load Data - Load your Load List
  5. Gather load metrics - GatherLoadStatistics.ps1
  6. Gather monitoring metrics - Live Optics

Checklist

  1. Run Truxton_Installer.exe
    1. Do not use defaults
    2. Data Depot Directory: D:\Truxton Data
  2. Stop Truxton
    1. net stop truxton
    2. net stop les
    3. Close the Tray App
  3. Configure Truxton
  4. Start Truxton
    1. net start truxton
    2. net start les
  5. Load Something
    1. Load.exe LoadListName.txt
  6. Wait until all media in list have loaded
  7. GatherLoadStatistics.ps1

Single Node (Laptop Configuration)

The following steps are an example of turning the server at position U11 in the rack into an independent node.

  1. Stop and uninstall Truxton
  2. Run D:\Cleanup.bat
  3. Run D:\ClearLocalDatabase.bat
  4. Run D:\ClearLocalMessageBus.bat
  5. Run D:\Install.cmd
    1. Do not use defaults
    2. Data Depot Directory: D:\Truxton Data
  6. Stop Truxton
    1. net stop truxton
    2. net stop les
    3. Close the Tray App
  7. Configure Truxton
    1. Copy D:\Truxton Data\Search to C:\ClusterStorage\Scratch\U11 (To your scratch directory)
    2. Delete the files in C:\ProgramData\Truxton\Settings
    3. Copy the U11_* files from C:\ClusterStorage\SampleData\Rack Management\Configurations\Laptop to C:\ProgramData\Truxton\Settings
    4. Rename the files in C:\ProgramData\Truxton\Settings to get rid of the preambles (U11_137_)
  8. Ensure Virus Scanning won't interfere with Truxton
  9. Ensure PostgreSQL can handle the number of connections
  10. Start Truxton
    1. net start truxton
    2. net start les
  11. Load Something
    1. Load.exe LoadListName.txt
  12. Wait until all media in list have loaded
  13. GatherLoadStatistics.ps1

2x5 Publish Configuration

The following steps detail how to reconfigure the rack so that all servers process data and finished loads publish up to the Yellowbrick.

  1. On each server
    1. Stop and uninstall Truxton
    2. Run D:\Cleanup.bat
    3. Run D:\ClearLocalDatabase.bat
    4. Run D:\ClearLocalMessageBus.bat
    5. Run D:\Install.cmd
      1. Do not use defaults
      2. Data Depot Directory: D:\Truxton Data
    6. Stop Truxton
      1. net stop truxton
      2. net stop les
      3. Close the Tray App
  2. Configure Truxton
    1. Copy D:\Truxton Data\Search to C:\ClusterStorage\Scratch\U11 (To your scratch directory)
    2. Delete the files in C:\ProgramData\Truxton\Settings
    3. Copy the U11_* files from C:\ClusterStorage\SampleData\Rack Management\Configurations\Laptop to C:\ProgramData\Truxton\Settings
    4. Rename the files in C:\ProgramData\Truxton\Settings to get rid of the preambles (U11_137_)
  3. Ensure Virus Scanning won't interfere with Truxton
  4. Ensure PostgreSQL can handle the number of connections
  5. Start Truxton
    1. net start truxton
    2. net start les
  6. Load Something
    1. Load.exe LoadListName.txt
  7. Wait until all media in list have loaded
  8. GatherLoadStatistics.ps1
Retrieved from "https://truxwiki.com/index.php?title=Truxton_Forensic_Rack&oldid=6248"