Loader Configuration

From truxwiki.com
Revision as of 06:48, 13 September 2020 by Sam (talk | contribs) (→‎Install)
Jump to navigation Jump to search

Some pointers on tuning loader machines.

Loader Machine

Install

We recommend you install Truxton on each loader node as follows:

  1. On the D: drive, make the following folders:
    1. Hashsets
    2. Temp
    3. Truxton Data\Search (this is actually two folders)
  2. Run the PostgreSQL installer (postgresql-9.6.10-2-windows-x64.exe in the Prerequisites folder of the installation media). Use the following settings:
    1. Data Directory: D:\Truxton Data\data
    2. Password: Truxton4n6
    3. Locale to be used: C
  3. Run Truxton_Installer.exe, do not use the defaults
    1. Change the "Depot Data Directory" to D:\Truxton Data

Temporary Folder

The tempdir setting controls the folder where Truxton ETLs will write their temporary files. In the case of PST files, this can be huge. Choose a fast drive to hold this.

Hashsets

The hashsetdir setting controls the folder where hashsets are stored. Any ETL process that produces files will need fast access to the files in this folder. Many reads will be performed on these files so this folder should be on a fast drive.

Rack Configuration

The Truxton Forensic Rack has a Yellowbrick database appliance, dedicated loader/exploitation machines and depot storage. With everything separated, it gives us many knobs to turn for loading.

U42 Network Switch
U27 Yellowbrick
U26 Yellowbrick Manager 0
U25 Yellowbrick Manager 1
U23 KVM Server
U22 Keyboard and Monitor
U13 Loader Node 7
U11 Loader Node 6
U09 Loader Node 5
U07 Loader Node 4
U05 Loader Node 3
U03 Loader Node 2
U01 Loader Node 1

Loader Nodes

Truxton is made up of different kinds of processes running in the ETL layer. Some processes makes sense to have many instances of while others only require a single instance.

Loader Node 1

This machine hosts the singleton processes and low-volume processes.

  1. Load Status Monitor
  2. Message Bus
  3. SOLR Server
  4. Alert
  5. Finished
  6. Identify
  7. Poly

Loader Nodes 2-7

These are the machines with lots to do. They host:

  1. Expand
  2. Poly File Expander
  3. Archives
  4. File Carver
  5. Video Contact Sheet
  6. EMail Parsing
  7. Load

Machine IDs

Normally the machine identifier is derived from the operating system but in a rack situation we sacrifice some entropy to provide system administrators assistance. The Machine ID for rack configurations are made up of the following fields:

Field Length Meaning
Rack 2 bytes 5452 TR for Truxton Rack
Rack Serial Number 2 bytes The serial number of the rack in hex encoded decimal
Rack Address Indicator 1 byte 55 U
Rack Address 1 byte Unit address in the rack of the machine
Logical Processors 2 bytes The number of logical CPUs in the machine
RAM 2 bytes The number of gigabytes of RAM in the machine
IP Address 6 bytes The IP address of the machine

For Example a Rack Machine ID of 54520001-5501-0072-0191-010010182131 tells us:

Bytes Meaning
5452 ASCII TR, this is a Rack Machine ID
0001 Serial Number 1
55 Unit Address
01 The machine is in slot U01 in the rack
0072 The machine has 72 logical processors
0191 The machine has 191GB of RAM
010010182131 The IP address of the machine is 10.10.182.131

The Machine ID corresponds to the MachineID column of the Statistics table. This allows you to see how the different machines perform their loads.

Loader Node Configuration Files

A pair of configuration files TruxtonSettings.xml and TruxtonService.xml are created and saved to make installation easy on virgin machines. TruxtonSettings.xml identifies the loader node and where the database and depots are. TruxtonService.xml governs which ETL processes will run on that machine.

Low-Volume Host

Low-volume is defined as something that isn't used very often. When it is, there's not much impact on the performance of this host machine. A low-volume ETL won't produce very many of anything.

TruxtonService.xml

The following is a sample configuration file for low-volume host. 

<root>
  <truxton_options>
  <!-- This Truxton service configuration file was created 2020-07-09 17:30:09 by TruxtonService.exe running as SYSTEM from the machine named LOADER01 (10.10.10.131). -->
    <service_configuration_version>1840023666788</service_configuration_version>
  <!-- There should only be one purger of expired media on your network -->
    <purger>yes</purger>
    <work_schedule>
    <!-- If this service is not employed, the service will run but no ETLs will be allowed to run -->
      <employed>yes</employed>
      <monday>
        <on_duty>0000-2359</on_duty>
      </monday>
      <tuesday>
        <on_duty>0000-2359</on_duty>
      </tuesday>
      <wednesday>
        <on_duty>0000-2359</on_duty>
      </wednesday>
      <thursday>
        <on_duty>0000-2359</on_duty>
      </thursday>
      <friday>
        <on_duty>0000-2359</on_duty>
      </friday>
      <saturday>
        <on_duty>0000-2359</on_duty>
      </saturday>
      <sunday>
        <on_duty>0000-2359</on_duty>
      </sunday>
    </work_schedule>
    <emptyqueuethresholdseconds>3600</emptyqueuethresholdseconds>
    <etls>
      <etl>
        <description>This consumes BOLOs and creates Alerts.</description>
        <exe controllable="yes" instances="1" queue="alert">Alert</exe>
      </etl>
      <etl>
        <description>This expands archive files.</description>
        <exe controllable="yes" instances="4" queue="archives">Archives</exe>
      </etl>
      <etl>
        <description>This carves free space for files.</description>
        <exe controllable="yes" instances="1" queue="carve">Carve</exe>
        <arguments>-carve_threads 0</arguments>
      </etl>
      <etl>
        <description>This generates video contact sheets, the 10x10 grid of images taken throughout the video.</description>
        <exe controllable="yes" instances="4" queue="contactsheet">ContactSheet</exe>
      </etl>
      <etl>
        <description>This parses MIME email files.</description>
        <exe controllable="yes" instances="1" queue="email">EMail</exe>
      </etl>
      <etl>
        <description>This is the main file expander service.</description>
        <exe controllable="yes" instances="10" queue="expand">Expand</exe>
      </etl>
      <etl>
        <description>This performs final processing after all files are present. It performs count queries and updates statistics.</description>
        <exe controllable="yes" instances="4" queue="finishedstage">Finished</exe>
      </etl>
      <etl>
        <description>This identifies file contents and routes accordingly.</description>
        <exe controllable="yes" instances="1" queue="identify">Identify</exe>
      </etl>
      <etl>
        <description>This is a loader as an ETL. It has the responsibility to expand files and load media.</description>
        <exe controllable="yes" instances="4" queue="load">Load</exe>
        <arguments>-lq load</arguments>
      </etl>
      <etl>
        <description>This coordinates the poly file expansion process.</description>
        <exe controllable="yes" instances="1" queue="poly">Poly</exe>
      </etl>
      <etl>
        <description>This finds the all of the pieces of multi-part archives and expands them.</description>
        <exe controllable="yes" instances="1" queue="pfe">PolyFileExpander</exe>
      </etl>
      <etl>
        <description>This exploits Windows registry files.</description>
        <exe controllable="yes" instances="1" queue="registry">Registry</exe>
      </etl>
      <etl>
        <description>This exports registry files to the local filesystem, spawns RegRipper.exe, grabs the result and makes it a child file of the registry file.</description>
        <exe controllable="yes" instances="1" queue="regripper">RegRipper</exe>
      </etl>
      <etl>
        <description>This spawns executables to expand files then kills them when done.</description>
        <exe controllable="yes" instances="1" queue="remoteexpand">RemoteFileExpander</exe>
      </etl>
      <etl>
        <description>This generates the reports.</description>
        <exe controllable="yes" instances="1" queue="report">Report</exe>
      </etl>
      <etl>
        <description>This keeps the SOLR service running.</description>
        <exe controllable="yes" instances="1" queue="solrcontentstage">SOLR</exe>
      </etl>
      <etl>
        <description>This sends files to SOLR for content indexing.</description>
        <exe controllable="yes" instances="1" queue="solrfile">SOLRFile</exe>
      </etl>
      <etl>
        <description>This reassembles fragments of carved files into the correct order for viewing.</description>
        <exe controllable="yes" instances="1" queue="stitch">Stitch</exe>
      </etl>
      <etl>
        <description>This extracts text from files.</description>
        <exe controllable="yes" instances="1" queue="tqueue">TextExtract</exe>
      </etl>
      <etl>
        <description>This generates small thumbnail images from larger images.</description>
        <exe controllable="yes" instances="8" queue="thumbnail">Thumbnail</exe>
      </etl>
      <etl>
        <description>This uses Yara to scan files for the rules you specify. Normally this is a malware scanner.</description>
        <exe controllable="yes" instances="1" queue="yara">Yara</exe>
      </etl>
    </etls>
    <services>{Solr 5 Server||$TextIndexerData$Search/bin/solr.cmd|start -p 8983 -m 32533m -a &quot;-XX:-UsePerfData&quot;|$TextIndexerData$Search/bin/solr.cmd|stop -p 8983|java.exe|jetty.port=8983},</services>
  <!-- The shutdownmachine boolean value tells Truxton if it should power down the server once all ETLs go idle -->
    <shutdownmachine>false</shutdownmachine>
  </truxton_options>
</root>

TruxtonSettings.xml

The following is a sample low-volume configuration file. 

<truxton_options>
  <hashsetdir>D:\Hashsets</hashsetdir>
  <tempdir>D:\Temp</tempdir>
  <fct>1201</fct>
  <mqconcentrator>Yes</mqconcentrator>
  <mbp>MBCConnie.dll</mbp>  
  <xsyslog>127.0.0.1</xsyslog>
  <xsyslogport>514</xsyslogport>
  <FileGroupsHaveBeenInitialized>1</FileGroupsHaveBeenInitialized>
  <dbserver>10.10.10.103</dbserver>
  <dbport>5432</dbport>
  <dbname>Truxton</dbname>
  <dbuser>postgres</dbuser>
  <dbpassword>Truxton4n6</dbpassword>
  <mqserver>10.10.10.131</mqserver>
  <mqport>5432</mqport>
  <mqname>TruxtonMessageBus</mqname>
  <mquser>postgres</mquser>
  <mqpassword>Truxton4n6</mqpassword>
  <datadir>C:\ClusterStorage\Depot01\Depot</datadir>
  <machineid>54520001-5501-0072-0191-010010010131</machineid>
  <solr_url>http://10.10.10.131:8983/solr/truxton-core</solr_url>
  <CreateTheDatabase>1</CreateTheDatabase>
</truxton_options>

High-Volume Host

High-volume is when an ETL will consume significant host machine resources or produce lots of files.

TruxtonService.xml

TruxtonSettings.xml

<truxton_options>
  <hashsetdir>D:\Hashsets</hashsetdir>
  <tempdir>D:\Temp</tempdir>
  <fct>1201</fct>
  <mqconcentrator>Yes</mqconcentrator>
  <mbp>MBCConnie.dll</mbp>  
  <xsyslog>127.0.0.1</xsyslog>
  <xsyslogport>514</xsyslogport>
  <FileGroupsHaveBeenInitialized>1</FileGroupsHaveBeenInitialized>
<dbserver>10.10.10.103</dbserver>
<dbport>5432</dbport>
<dbname>Truxton</dbname>
<dbuser>postgres</dbuser>
<dbpassword>password</dbpassword>
  <mqserver>10.10.10.131</mqserver>
  <mqport>5432</mqport>
  <mqname>TruxtonMessageBus</mqname>
  <mquser>postgres</mquser>
  <mqpassword>Truxton4n6</mqpassword>
<datadir>C:\ClusterStorage\Depot01\Depot</datadir>
<machineid>54520001-5501-0072-0191-010010010131</machineid>
<CreateTheDatabase>1</CreateTheDatabase>
<solr_url>http://10.10.10.131:8983/solr/truxton-core</solr_url>
</truxton_options>

Loader Node Installation Checklist

Here are the steps for installing Truxton as a loader node on the rack. We recommend that each loader node have an SSD dedicated to Truxton. This SSD, we will call it the D: drive, can be in the 200GB range or larger. Truxton will use it to host the hashsets and for temporary files. We do this because hashsets require a lot of seek and reads. If it is on an SSD, the speed will be dramatically improved over having it on a shared folder. Likewise, the temporary files folder should be on an SSD for improved IO performance.

  1. Make sure FIPS mode is off
  2. Create C:\ProgramData\Truxton\Settings folder
  3. Copy the two configuration files into the Settings folder
  4. Modify the TruxtonSettings.xml file to set the machineid, set the rack and IP address fields
  5. Install Truxton
    1. Run Truxton_Installer.exe
    2. If asked "Should I install Java?" click the "No" button. Do not install Java
    3. If asked "Should I install PostgreSQL Database Server?" click the "No" button. Do not install PostgreSQL
    4. When asked "Would you like to use the default options when installing Truxton?" click the "Yes" button. Yes install with the default options (since we copied the settings files already)
  6. Stop the Truxton service
  7. Stop Les
  8. Set Les (Truxton Monitor) service to be Disabled from the Windows Services control panel.
  9. Create D:\Hashsets folder
  10. Create the D:\Temp folder
  11. Remove permissions from D:\Temp so the ETL processes can write to it.
  12. Copy the hashsets from the installation media to D:\Hashsets
  13. Restart the Truxton servicec
  14. Exit the tray app

Useful Batch Files

When working on the rack, life is made easier through the use of scripts.

Remove Permissions

Here's the Powershell script to remove security from a folder. 

$folder_path = "D:\Temp"
$security_descriptor = Get-Acl -Path $folder_path

# This is the SDDL for a NULL DACL
$sddl_for_a_null_dacl = ""

$security_descriptor.SetSecurityDescriptorSddlForm($sddl_for_a_null_dacl)
Set-Acl -Path $folder_path -AclObject $security_descriptor

Stop the Load Monitoring Service

This script will stop Les on the high-volume nodes. 

@ECHO OFF
ECHO U01 is the Truxton Load Status Monitor (Les) Service Singleton
ECHO See https://truxwiki.com/Les for more details
REM sc \\10.10.10.131 stop Les
ECHO Stopping Les service on U03
sc \\10.10.10.132 stop Les
ECHO Stopping Les service on U05
sc \\10.10.10.133 stop Les
ECHO Stopping Les service on U07
sc \\10.10.10.134 stop Les
ECHO Stopping Les service on U09
sc \\10.10.10.135 stop Les
ECHO Stopping Les service on U11
sc \\10.10.10.136 stop Les
ECHO Stopping Les service on U13
sc \\10.10.10.137 stop Les

Start Truxton Service

This script will start the Truxton service on the rack nodes. 

@ECHO OFF
ECHO Starting Truxton service on U01
sc \\10.10.10.131 start Truxton
ECHO Starting Truxton service on U03
sc \\10.10.10.132 start Truxton
ECHO Starting Truxton service on U05
sc \\10.10.10.133 start Truxton
ECHO Starting Truxton service on U07
sc \\10.10.10.134 start Truxton
ECHO Starting Truxton service on U09
sc \\10.10.10.135 start Truxton
ECHO Starting Truxton service on U11
sc \\10.10.10.136 start Truxton
ECHO Starting Truxton service on U13
sc \\10.10.10.137 start Truxton

Stop Truxton Service

This script will stop the Truxton service on the rack nodes. 

@ECHO OFF
ECHO Stopping Truxton service on U01
sc \\10.10.10.131 stop Truxton
ECHO Stopping Truxton service on U03
sc \\10.10.10.132 stop Truxton
ECHO Stopping Truxton service on U05
sc \\10.10.10.133 stop Truxton
ECHO Stopping Truxton service on U07
sc \\10.10.10.134 stop Truxton
ECHO Stopping Truxton service on U09
sc \\10.10.10.135 stop Truxton
ECHO Stopping Truxton service on U11
sc \\10.10.10.136 stop Truxton
ECHO Stopping Truxton service on U13
sc \\10.10.10.137 stop Truxton

Uninstall Truxton

This script will uninstall Truxton from the rack nodes. 

@ECHO OFF
ECHO Stopping Truxton service on U01
sc \\10.10.10.131 stop Truxton
sc \\10.10.10.131 stop Les
ECHO Stopping Truxton service on U03
sc \\10.10.10.132 stop Truxton
sc \\10.10.10.132 stop Les
ECHO Stopping Truxton service on U05
sc \\10.10.10.133 stop Truxton
sc \\10.10.10.133 stop Les
ECHO Stopping Truxton service on U07
sc \\10.10.10.134 stop Truxton
sc \\10.10.10.134 stop Les
ECHO Stopping Truxton service on U09
sc \\10.10.10.135 stop Truxton
sc \\10.10.10.135 stop Les
ECHO Stopping Truxton service on U11
sc \\10.10.10.136 stop Truxton
sc \\10.10.10.136 stop Les
ECHO Stopping Truxton service on U13
sc \\10.10.10.137 stop Truxton
sc \\10.10.10.137 stop Les
ECHO Truxton Service stopped on all nodes, starting uninstall process
start wmic /node:10.10.10.131 product where name="Truxton" call uninstall /nointeractive
start wmic /node:10.10.10.132 product where name="Truxton" call uninstall /nointeractive
start wmic /node:10.10.10.133 product where name="Truxton" call uninstall /nointeractive
start wmic /node:10.10.10.134 product where name="Truxton" call uninstall /nointeractive
start wmic /node:10.10.10.135 product where name="Truxton" call uninstall /nointeractive
start wmic /node:10.10.10.136 product where name="Truxton" call uninstall /nointeractive
start wmic /node:10.10.10.137 product where name="Truxton" call uninstall /nointeractive
ECHO All nodes told to uninstall, should be done in a couple of minutes

Storage Spaces Direct

The Truxton rack uses Microsoft's Storage Spaces Direct for managing disk space.

Creating a Volume

Here's how to create a 64TB volume for depots 

New-Volume -FriendlyName "GoldCopy" -Size 250TB -ResiliencySettingName Mirror -NumberOfDataCopies 2 -FileSystem CSVFS_ReFS -StoragePoolFriendlyName S2D*

Repairing a Volume

Sometimes you may see "Needs Repair" as its status.

Retrieved from "https://truxwiki.com/index.php?title=Loader_Configuration&oldid=2246"