Creating a Rack

If you have a plethora of equipment lying around, you can make your own Truxton Forensic Rack solution. This article will discuss how to configure the rack as a Truxton solution.

Equipment

The number and type of equipment that make up the rack is up to you. All Truxton needs is Windows OS.

Storage

Truxton ETL servers require a single view of depot storage. When an ETL process writes to a depot file, a process on a different machine must be able to read from it. ETL's require fast IO. For this reason we leave this to the operating system. One possible solution is to use Windows Storage Spaces Direct. The remainder of this article will assume that all rack servers will have the mount point for the shares file system to be C:\ClusterStorage.

Each server will have a rather small C: drive which will host the operating system and applications. They will also contain a larger D:\ drive to hold sets. Ideally these will SSDs. The goal is to never write to the internal drives of a server. Anything that Truxton writes should go onto the shared storage.

Careful attention needs to be paid to where things are stored. Hashsets need to be on fast media since they are accessed frequently during ETL processing. The log folder, temporary file folder, and text indexer folder should be on shared storage dedicated to the machine.

Sample Rack

Let's assume you have three servers lying around. You might populate the rack thusly:

Location	Equipment	IP Address
U18	Network Switch	10.10.1.200
U17	KVM	10.10.1.100
U13	Server 3	10.10.1.133
U09	Server 2	10.10.1.132
U05	Server 1	10.10.1.131
U01	UPS	10.10.1.50

Responsibilities

Decide which machines will have which responsibilities. There are a handful of singleton pieces of Truxton:

1. Database
2. Message Bus
3. Load Status Monitor

These do not need to be hosted on every server.

Configurations

You may want to play with different configurations of Truxton on the rack. To reconfigure Truxton, you just stop Truxton, replace the TruxtonSettings.xml file, replace the TruxtonService.xml file, and restart Truxton. You can make this process easier by using scripts and a configuration library.

Scripts

In the root of D:\, place batch files that will reconfigure that specific server by stopping the services, replacing the configuration files and restarting.

Configuration Library

In C:\ClusterStorage, create a folder called Configuration, then create, in this folder, another folder to hold the configuration files for each configuration. Create the settings and service files and place them here. The naming convention used by Truxton Forensics for machine specific files to begin with their rack location and IP address.

Managing Configuration

Keeping track of these different configurations can get out of hand quickly. After logging into a machine, you will ask yourself, "How is this machine configured?" We recommend using Truxton's [Configuration System|configuration system] to help you know how a server is configured. By adding your own items to TruxtonSettings.xml and using a script to display them, you can instantly determine that machine's configuration.

We recommend the following:

nodename	The name of the role this machine plays in the configuration
nodeconfig	The name of the configuration
nodedescription	A description of the context of this machine in the configuration
nodesettings	The full path to the file used to create the TruxtonSettings.xml on this machine
nodeservice	The full path to the file used to create the TruxtonService.xml on this machine

Once these are saved in the configuration file, we can use the following role.py Python script to show them. Running D:\role.py after logging onto the machine will tell you how it is configured.

import sys
sys.path.append('C:/Program Files/Truxton/SDK')
import truxton

def main():
  options = truxton.options()

  print( options.getstring("nodename") )
  print( options.getstring("nodeconfig") + " configuration" )
  print( options.getstring("nodedescription") )
  print( "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml came from " + options.getstring("nodesettings") )
  print( "C:\ProgramData\Truxton\Settings\TruxtonService.xml came from " + options.getstring("nodeservice") )

if __name__ == "__main__":
  main()

Samples

Here are a couple of examples of shared configuration as well as individual.

Shared Configuration

This sample is for a stand-alone instance of Truxton. Nothing is shared, similar to having three laptops in the rack.

The configuration files will be in the folder C:\ClusterStorage\Configurations\Laptop

Settings Files

The following is the contents for U05_131_TruxtonSettings.xml Notice the logdir, tempdir, and textindexerdir folders point to folders for this machine on shared storage because these folders are frequently written to. The hashsetdir setting points to the internal drive of the server because hashsets are read-only operations.

<?xml version="1.0" encoding="utf-8"?>
<truxton_options>
  <logdir>C:\ClusterStorage\Scratch\U05\Logs</logdir>
  <tempdir>C:\ClusterStorage\Scratch\U05\Temp</tempdir>
  <textindexerdir>C:\ClusterStorage\Scratch\U05\Search</textindexerdir>
  <hashsetdir>D:\Truxton Data\Hashsets</hashsetdir>
  <dbconnectionstring>Host=localhost;Port=5432;Database=Truxton;Username=postgres;Password=Truxton4n6;</dbconnectionstring>
  <mbconnectionstring>Host=localhost;Port=5432;Database=TruxtonMessageBus;Username=postgres;Password=Truxton4n6;</mbconnectionstring>
  <datadir>C:\ClusterStorage\Depot\Shares</datadir>
  <depotshare>\\sofs-02\Depot</depotshare>
  <purger>No</purger>
  <notray>Yes</notray>
  <fct>306</fct>
  <CreateTheDatabase>1</CreateTheDatabase>
  <solr_url>http://localhost:8983/solr/truxton-core</solr_url>
  <FileGroupsHaveBeenInitialized>1</FileGroupsHaveBeenInitialized>
  <!-- The interpretation of this machine id is:
       Rack Serial Number 1
       Server is in location U05
       Server contains 72 logical processors
       Server has 191GB RAM installed
       Server IP address is 10.10.1.131
       
       See https://truxwiki.com/Loader_Configuration#Machine_IDs for more details
  -->
  <machineid>54520001-5505-0072-0191-010010001131</machineid>
  
  <nodename>U05</nodename>
  <nodeconfig>Laptop</nodeconfig>
  <nodedescription>This machine hosts all of Truxton and operates independently from any other machine.</nodedescription>
  <nodesettings>C:\ClusterStorage\Configurations\Laptop\U05_131_TruxtonSettings.xml</nodesettings>
  <nodeservice>C:\ClusterStorage\Configurations\Laptop\Laptop_TruxtonService.xml</nodeservice>
</truxton_options>

Batch File

Here's the D:\Laptop.bat that will setup the server (U05) to a laptop configuration.

@ECHO off
ECHO Stopping services
net stop truxton
net stop les
ECHO Terminating Tray Application
taskkill /fi "IMAGENAME eq TruxtonStatus.exe" /F
ECHO Stopping SOLR
CALL "C:\ClusterStorage\Scratch\U05\Search\bin\solr.cmd" stop -p 8983
ECHO Giving ETL processes time to stop nicely
TIMEOUT /t 30
ECHO Terminating Truxton processes with extreme prejudice
TaskKill /fi "IMAGENAME eq Alert.exe" /F
TaskKill /fi "IMAGENAME eq Archives.exe" /F
TaskKill /fi "IMAGENAME eq Carve.exe" /F
TaskKill /fi "IMAGENAME eq ContactSheet.exe" /F
TaskKill /fi "IMAGENAME eq Email.exe" /F
TaskKill /fi "IMAGENAME eq Expand.exe" /F
TaskKill /fi "IMAGENAME eq Finished.exe" /F
TaskKill /fi "IMAGENAME eq Identify.exe" /F
TaskKill /fi "IMAGENAME eq Java.exe" /F
TaskKill /fi "IMAGENAME eq LangIDexe" /F
TaskKill /fi "IMAGENAME eq Load.exe" /F
TaskKill /fi "IMAGENAME eq Maintenance.exe" /F
TaskKill /fi "IMAGENAME eq Notify.exe" /F
TaskKill /fi "IMAGENAME eq Poly.exe" /F
TaskKill /fi "IMAGENAME eq PolyFileExpander.exe" /F
TaskKill /fi "IMAGENAME eq Registry.exe" /F
TaskKill /fi "IMAGENAME eq RegRipper.exe" /F
TaskKill /fi "IMAGENAME eq RemoteFileExpander.exe" /F
TaskKill /fi "IMAGENAME eq Report.exe" /F
TaskKill /fi "IMAGENAME eq SOLR.exe" /F
TaskKill /fi "IMAGENAME eq SOLRFile.exe" /F
TaskKill /fi "IMAGENAME eq Stitch.exe" /F
TaskKill /fi "IMAGENAME eq TextExtract.exe" /F
TaskKill /fi "IMAGENAME eq Thumbnail.exe" /F
TaskKill /fi "IMAGENAME eq Yara.exe" /F
ECHO Deleting old configurations
IF EXIST "C:\ProgramData\Truxton\Settings\TruxtonService.xml" DEL "C:\ProgramData\Truxton\Settings\TruxtonService.xml"
IF EXIST "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml" DEL "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
IF EXIST "C:\ProgramData\Truxton\Settings\TruxtonService.xml.backup" DEL "C:\ProgramData\Truxton\Settings\TruxtonService.xml.backup"
IF EXIST "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml.backup" DEL "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml.backup"
ECHO Installing Laptop Configuration Files
MKDIR "D:\Temp"
MKDIR "C:\ProgramData\Truxton\Settings"
COPY "C:\ClusterStorage\SampleData\Rack Management\Configurations\Laptop\Laptop_TruxtonService.xml" "C:\ProgramData\Truxton\Settings\TruxtonService.xml"
COPY "C:\ClusterStorage\SampleData\Rack Management\Configurations\Laptop\U05_131_TruxtonSettings.xml" "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
ECHO Clearing out old SOLR
IF EXIST "C:\ClusterStorage\Scratch\U05\Search" RMDIR /s /q "C:\ClusterStorage\Scratch\U05\Search"
ECHO Initializing New SOLR
MKDIR "C:\ClusterStorage\Scratch\U05\Search"
XCOPY "D:\Truxton Data\Search" "C:\ClusterStorage\Scratch\U05\Search" /E /H /C /I
ECHO Starting SOLR
CALL "C:\ClusterStorage\Scratch\U05\Search\bin\solr.cmd" start -p 8983 -m 20879m -a "-XX:-UsePerfData"
ECHO Starting Truxton Services
net start Truxton
ECHO Clearing Local Database
SET PGPASSWORD=Truxton4n6
"C:\Program Files\PostgreSQL\12\bin\psql.exe" --host=localhost --port=5432 --username=postgres --dbname=Truxton --tuples-only --file="C:\ClusterStorage\SampleData\Rack Management\Database Scripts\Load Test Resets\Reset Truxton Database.sql"
ECHO Clearing Local Message Bus
"C:\Program Files\PostgreSQL\12\bin\psql.exe" --host=localhost --port=5432 --username=postgres --dbname=TruxtonMessageBus --tuples-only --file="C:\ClusterStorage\SampleData\Rack Management\Database Scripts\Load Test Resets\Clear All Queues.sql"
ECHO Enabling the Load Status Monitor (Les) on this machine
sc config Les start=auto
net start Les
ECHO You are now ready to load data on this machine