Difference between revisions of "Loader Configuration"

From truxwiki.com
Jump to navigation Jump to search
 
(23 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
Some pointers on tuning loader machines.
 
Some pointers on tuning loader machines.
  
−
=Loader Machine=
+
=Single Machine=
 +
Also known as a "laptop" configuration.
 +
 
 
==Install==
 
==Install==
 
We recommend you install Truxton on each loader node as follows:
 
We recommend you install Truxton on each loader node as follows:
Line 8: Line 10:
 
## <code>Temp</code>
 
## <code>Temp</code>
 
## <code>Truxton Data\Search</code> (this is actually two folders)
 
## <code>Truxton Data\Search</code> (this is actually two folders)
−
# Run the PostgreSQL installer (<code>postgresql-9.6.10-2-windows-x64.exe</code> in the <code>Prerequisites</code> folder of the installation media). Use the following settings:
+
# Run the PostgreSQL installer (<code>postgresql-12.4.1-windows-x64.exe</code> in the <code>Prerequisites</code> folder of the installation media). Use the following settings:
 
## Data Directory: <code>D:\Truxton Data\data</code>
 
## Data Directory: <code>D:\Truxton Data\data</code>
 
## Password: <code>Truxton4n6</code>
 
## Password: <code>Truxton4n6</code>
Line 25: Line 27:
 
Many reads will be performed on these files so this folder should be on a fast drive.
 
Many reads will be performed on these files so this folder should be on a fast drive.
  
−
=Rack Configuration=
+
=Truxton Forensic Rack=
−
The Truxton Forensic Rack has a [https://www.yellowbrick.com/ Yellowbrick] database appliance, dedicated loader/exploitation machines and [[Depot|depot]] storage.
+
The Truxton Forensic Rack is a 7 foot tall rack populated with hardware dedicated to forensic processing.
−
With everything separated, it gives us many knobs to turn for loading.
+
It has a [https://www.yellowbrick.com/ Yellowbrick] database appliance, seven dedicated loader/exploitation machines and 1PB of usable [[Depot|depot]] storage.
 +
With everything separated, it gives us many ways to configure processing.
 +
 
 +
==Hardware==
 
Here's how the hardware is stacked:
 
Here's how the hardware is stacked:
  
Line 72: Line 77:
 
|}
 
|}
  
−
===Analyst Desktop===
+
==Analyst Desktop==
 
One of the things to turn off is the automatic <code>ANALYZE</code> when the desktop starts.
 
One of the things to turn off is the automatic <code>ANALYZE</code> when the desktop starts.
 
This is controlled by the <code>skip_update_db_stats_on_boot</code> desktop setting.
 
This is controlled by the <code>skip_update_db_stats_on_boot</code> desktop setting.
Line 93: Line 98:
 
</pre>
 
</pre>
  
−
===Loader Nodes===
+
==Server Architectures==
−
Truxton is made up of different kinds of processes running in the ETL layer.
+
With seven machines dedicated to forensic processing, we have a flexible platform to configure in a variety of ways.
−
Some processes makes sense to have many instances of while others only require a single instance.
+
 
 +
===Laptop===
 +
The "laptop" architecture uses the servers as seven individual machines.
 +
Each server hosts everything needed to run Truxton.
 +
The disadvantage of this configuration is that the other machines in the rack cannot help to process the data being loaded.
 +
 
 +
===2x5===
 +
This configuration will dedicate two machines to database and message bus.
 +
The remaining 5 machines cooperate in exploiting the media being loaded.
  
 
====Loader Node 1 (U01)====
 
====Loader Node 1 (U01)====
Line 118: Line 131:
 
They host:
 
They host:
 
# [[Expand]]
 
# [[Expand]]
−
# Poly File Expander
+
# [[Poly|Poly File Expander]]
 
# [[Archives]]
 
# [[Archives]]
 
# [[Carve|File Carver]]
 
# [[Carve|File Carver]]
Line 125: Line 138:
 
# [[Load]]
 
# [[Load]]
  
−
=====Steps to Update Truxton on a Loader Node=====
+
==Maintenance==
 +
===Steps to Update Truxton on a Loader Node===
 
# Stop the Truxton Service <source lang="bat">net stop truxton</source>
 
# Stop the Truxton Service <source lang="bat">net stop truxton</source>
 
# Uninstall Truxton
 
# Uninstall Truxton
Line 136: Line 150:
 
# Start the Truxton Service <source lang="bat">net start truxton</source>
 
# Start the Truxton Service <source lang="bat">net start truxton</source>
  
−
There should be only one [[Les]] on the rack and the Truxton Status Monitor tray application is something for a human to look at.
+
There should be only one [[Les]] on the rack and the [[Truxton_Status|Truxton Status Monitor tray application]] is something for a human to look at.
  
 
==Machine IDs==
 
==Machine IDs==
Line 207: Line 221:
 
The Machine ID corresponds to the <code>[MachineID]</code> column of the <code>[Statistics]</code> table.
 
The Machine ID corresponds to the <code>[MachineID]</code> column of the <code>[Statistics]</code> table.
 
This allows you to see how the different machines perform their loads.
 
This allows you to see how the different machines perform their loads.
 +
Rack serial numbers in the 9000 range are development racks where new configurations are tested.
  
 
==Loader Node Configuration Files==
 
==Loader Node Configuration Files==
Line 220: Line 235:
 
====TruxtonService.xml====
 
====TruxtonService.xml====
 
The following is a sample configuration file for low-volume host.
 
The following is a sample configuration file for low-volume host.
−
<syntaxhighlight lang="xml">
+
<source lang="xml">
 
<root>
 
<root>
 
   <truxton_options>
 
   <truxton_options>
Line 345: Line 360:
 
     <shutdownmachine>false</shutdownmachine>
 
     <shutdownmachine>false</shutdownmachine>
 
   </truxton_options>
 
   </truxton_options>
−
</root></syntaxhighlight>
+
</root></source>
  
 
====TruxtonSettings.xml====
 
====TruxtonSettings.xml====
 
The following is a sample low-volume configuration file.
 
The following is a sample low-volume configuration file.
−
<syntaxhighlight lang="xml">
+
<source lang="xml">
 
<truxton_options>
 
<truxton_options>
 
   <hashsetdir>D:\Hashsets</hashsetdir>
 
   <hashsetdir>D:\Hashsets</hashsetdir>
Line 374: Line 389:
 
   <CreateTheDatabase>1</CreateTheDatabase>
 
   <CreateTheDatabase>1</CreateTheDatabase>
 
</truxton_options>
 
</truxton_options>
−
</syntaxhighlight>
+
</source>
  
 
===High-Volume Host===
 
===High-Volume Host===
Line 381: Line 396:
 
====TruxtonService.xml====
 
====TruxtonService.xml====
 
====TruxtonSettings.xml====
 
====TruxtonSettings.xml====
−
<syntaxhighlight lang="xml">
+
<source lang="xml">
 
<truxton_options>
 
<truxton_options>
 
   <hashsetdir>D:\Hashsets</hashsetdir>
 
   <hashsetdir>D:\Hashsets</hashsetdir>
Line 406: Line 421:
 
<solr_url>http://10.10.10.131:8983/solr/truxton-core</solr_url>
 
<solr_url>http://10.10.10.131:8983/solr/truxton-core</solr_url>
 
</truxton_options>
 
</truxton_options>
−
</syntaxhighlight>
+
</source>
  
 
==Loader Node Installation Checklist==
 
==Loader Node Installation Checklist==
Line 417: Line 432:
 
Likewise, the temporary files folder should be on an SSD for improved IO performance.
 
Likewise, the temporary files folder should be on an SSD for improved IO performance.
  
−
# Make sure FIPS mode is off
+
# Make sure FIPS mode is off. [https://en.wikipedia.org/wiki/FIPS_140 FIPS] is a standard for secure computing. When Windows FIPS mode is on, it makes sure that only approved cryptographic algorithms are used for high level tasks like signing documents or encrypting files. Specifically, Windows will disable the use of non-approved algorithms. The problem is, Microsoft gave developers a [https://en.wikipedia.org/wiki/Microsoft_CryptoAPI Crypto API] that provided a common approach to low level tasks such as hashing. Other libraries (like [https://en.wikipedia.org/wiki/.NET_Framework .Net]) depend on the Crypto API for performing [https://en.wikipedia.org/wiki/MD5 MD5] hashing. When FIPS mode is on, MD5 is no longer part of the Crypto API and any code that relied on the Crypto API is now broken. This can lead to many hours of developer hilarity figuring out why programs stop working...
 
# Create <code>C:\ProgramData\Truxton\Settings</code> folder
 
# Create <code>C:\ProgramData\Truxton\Settings</code> folder
 
# Copy the two configuration files into the Settings folder
 
# Copy the two configuration files into the Settings folder
−
# Modify the <code>[[Configuration_System#Truxton_Settings_File|TruxtonSettings.xml]]</code> file to set the <code>[[#Machine IDs|machineid]]</code>, set the rack and IP address fields
+
# Modify the <code>[[Configuration_System#Truxton Settings File|TruxtonSettings.xml]]</code> file to set the <code>[[#Machine IDs|machineid]]</code>, set the rack and IP address fields
 
# Install Truxton
 
# Install Truxton
 
## Run <code>Truxton_Installer.exe</code>
 
## Run <code>Truxton_Installer.exe</code>
Line 433: Line 448:
 
# Remove permissions from <code>D:\Temp</code> so the ETL processes can write to it.
 
# Remove permissions from <code>D:\Temp</code> so the ETL processes can write to it.
 
# Copy the hashsets from the installation media to <code>D:\Hashsets</code>
 
# Copy the hashsets from the installation media to <code>D:\Hashsets</code>
−
# Restart the Truxton servicec
+
# Restart the Truxton service
−
# Exit the tray app
+
# Exit the [[Truxton_Status|tray app]]
  
 
==Rack Management==
 
==Rack Management==
Line 441: Line 456:
 
===Configurations===
 
===Configurations===
 
This sub-folder is home to the customized versions of <code>[[TruxtonService.xml]]</code> and <code>TruxtonSettings.xml</code> for each node.
 
This sub-folder is home to the customized versions of <code>[[TruxtonService.xml]]</code> and <code>TruxtonSettings.xml</code> for each node.
−
The rack address begins the filename.
+
The naming convention for the files is the rack address, underscore, last octet of the server's IP address, underscore, and XML filename.
−
For example, the <code>TruxtonSettings.xml</code> file to be used on the U01 machine will be named <code>U01_TruxtonSettings.xml</code>
+
For example, the <code>TruxtonSettings.xml</code> file to be used on the U01 machine at 10.10.182.131 will be named <code>U01_131_TruxtonSettings.xml</code>
  
−
Create a new sub-folder for each type of configuration you want the rack to use.
+
Create a new sub-folder for each configuration you want the rack to use.
 
The default configuration is in a sub-folder named <code>Nodes</code>
 
The default configuration is in a sub-folder named <code>Nodes</code>
  
Line 452: Line 467:
 
You can stop Truxton by using the <code>[[#Stopping Truxton|StopTruxton.cmd]]</code> batch file.
 
You can stop Truxton by using the <code>[[#Stopping Truxton|StopTruxton.cmd]]</code> batch file.
  
−
<syntaxhighlight lang="BAT">
+
<source lang="BAT">
 
@ECHO OFF
 
@ECHO OFF
 
start wmic /node:10.10.182.131 product call install true,"","C:\ClusterStorage\SampleData\Rack Management\Truxton Software\Truxton_Updates\Truxton.msi"
 
start wmic /node:10.10.182.131 product call install true,"","C:\ClusterStorage\SampleData\Rack Management\Truxton Software\Truxton_Updates\Truxton.msi"
Line 468: Line 483:
 
CALL ConfigTruxton.cmd
 
CALL ConfigTruxton.cmd
 
ECHO Finished.
 
ECHO Finished.
−
</syntaxhighlight>
+
</source>
  
 
===Starting Truxton===
 
===Starting Truxton===
 
The <code>StartTruxton.cmd</code> will start Truxton on all of the nodes of the rack.
 
The <code>StartTruxton.cmd</code> will start Truxton on all of the nodes of the rack.
  
−
<syntaxhighlight lang="BAT">
+
<source lang="BAT">
 
@ECHO OFF
 
@ECHO OFF
 
ECHO Starting Truxton service on U01
 
ECHO Starting Truxton service on U01
Line 489: Line 504:
 
ECHO Starting Truxton service on U13
 
ECHO Starting Truxton service on U13
 
sc \\10.10.182.138 start Truxton
 
sc \\10.10.182.138 start Truxton
−
</syntaxhighlight>
+
</source>
  
 
===Stopping Truxton===
 
===Stopping Truxton===
 
The <code>StopTruxton.cmd</code> batch file will stop Truxton on all of the nodes in the rack.
 
The <code>StopTruxton.cmd</code> batch file will stop Truxton on all of the nodes in the rack.
  
−
<syntaxhighlight lang="BAT">
+
<source lang="BAT">
 
@ECHO OFF
 
@ECHO OFF
 
ECHO Stopping Truxton service on U01
 
ECHO Stopping Truxton service on U01
Line 526: Line 541:
 
CALL KillETLProcesses.cmd 10.10.182.137
 
CALL KillETLProcesses.cmd 10.10.182.137
 
CALL KillETLProcesses.cmd 10.10.182.138
 
CALL KillETLProcesses.cmd 10.10.182.138
−
</syntaxhighlight>
+
</source>
  
 
===Configuring Truxton===
 
===Configuring Truxton===
 
The <code>ConfigTruxton.cmd</code> batch file will push the configuration files out to the nodes on the rack.
 
The <code>ConfigTruxton.cmd</code> batch file will push the configuration files out to the nodes on the rack.
−
This script will stop the tray application on all nodes and it will configure [[Les]] to be disabled on the worker nodes.
+
This script will stop the [[Truxton_Status|tray application]] on all nodes and it will configure [[Les]] to be disabled on the worker nodes.
  
−
<syntaxhighlight lang="BAT">
+
<source lang="BAT">
 
@ECHO OFF
 
@ECHO OFF
 
ECHO Configuring Truxton on U01
 
ECHO Configuring Truxton on U01
Line 568: Line 583:
 
wmic /node:10.10.182.138 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U13_TruxtonService.xml\"  \"C:\ProgramData\Truxton\Settings\TruxtonService.xml"
 
wmic /node:10.10.182.138 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U13_TruxtonService.xml\"  \"C:\ProgramData\Truxton\Settings\TruxtonService.xml"
 
wmic /node:10.10.182.138 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U13_TruxtonSettings.xml\" \"C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
 
wmic /node:10.10.182.138 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U13_TruxtonSettings.xml\" \"C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
−
</syntaxhighlight>
+
</source>
  
 
===Removing Truxton===
 
===Removing Truxton===
 
The <code>UninstallTruxton.cmd</code> batch file will remove Truxton from all the nodes on the rack.
 
The <code>UninstallTruxton.cmd</code> batch file will remove Truxton from all the nodes on the rack.
  
−
<syntaxhighlight lang="BAT">
+
<source lang="BAT">
 
@ECHO OFF
 
@ECHO OFF
 
CALL StopTruxton.cmd
 
CALL StopTruxton.cmd
Line 585: Line 600:
 
START wmic /node:10.10.182.138 product where name="Truxton" call uninstall /nointeractive
 
START wmic /node:10.10.182.138 product where name="Truxton" call uninstall /nointeractive
 
ECHO All nodes told to uninstall
 
ECHO All nodes told to uninstall
−
</syntaxhighlight>
+
</source>
  
 
===Updating Truxton===
 
===Updating Truxton===
 
When you receive a new version of Truxton, update the MSI file and run <code>UpdateTruxton.cmd</code>
 
When you receive a new version of Truxton, update the MSI file and run <code>UpdateTruxton.cmd</code>
−
<syntaxhighlight lang="BAT">
+
<source lang="BAT">
 
@ECHO OFF
 
@ECHO OFF
 
ECHO Updating Truxton
 
ECHO Updating Truxton
Line 595: Line 610:
 
CALL InstallTruxton.cmd
 
CALL InstallTruxton.cmd
 
CALL StartTruxton.cmd
 
CALL StartTruxton.cmd
−
</syntaxhighlight>
+
</source>
  
 
==Useful Batch Files==
 
==Useful Batch Files==
 
===Remove Permissions===
 
===Remove Permissions===
−
Here's the Powershell script to remove security from a folder.
+
Here's the [https://docs.microsoft.com/en-us/powershell/ PowerShell] script to remove security from a folder.
−
<syntaxhighlight lang="powershell">
+
<source lang="powershell">
 
$folder_path = "D:\Temp"
 
$folder_path = "D:\Temp"
 
$security_descriptor = Get-Acl -Path $folder_path
 
$security_descriptor = Get-Acl -Path $folder_path
Line 609: Line 624:
 
$security_descriptor.SetSecurityDescriptorSddlForm($sddl_for_a_null_dacl)
 
$security_descriptor.SetSecurityDescriptorSddlForm($sddl_for_a_null_dacl)
 
Set-Acl -Path $folder_path -AclObject $security_descriptor
 
Set-Acl -Path $folder_path -AclObject $security_descriptor
−
</syntaxhighlight>
+
</source>
  
 
==Storage Spaces Direct==
 
==Storage Spaces Direct==
−
The Truxton rack uses Microsoft's [https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview Storage Spaces Direct] for managing disk space.
+
The Truxton rack uses Microsoft's [https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview Storage Spaces Direct] (aka S2D) for managing disk space.
 +
 
 +
===Planning Volumes===
 +
[https://s2dcalc.blob.core.windows.net/www/index.html Here] is a calculator to use when planning on creating a volume.
  
 
===Creating a Volume===
 
===Creating a Volume===
−
Here's how to create a 64TB volume for depots
+
Here's how to create a 64TB volume for depots using the Powershell command <code>[https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/create-volumes New-Volume]</code>
−
<syntaxhighlight lang="powershell">
+
<source lang="powershell">
 
New-Volume -FriendlyName "GoldCopy" -Size 250TB -ResiliencySettingName Mirror -NumberOfDataCopies 2 -FileSystem CSVFS_ReFS -StoragePoolFriendlyName S2D*
 
New-Volume -FriendlyName "GoldCopy" -Size 250TB -ResiliencySettingName Mirror -NumberOfDataCopies 2 -FileSystem CSVFS_ReFS -StoragePoolFriendlyName S2D*
−
</syntaxhighlight>
+
</source>
  
 
===Repairing a Volume===
 
===Repairing a Volume===
 
Sometimes you may see "Needs Repair" as its status.
 
Sometimes you may see "Needs Repair" as its status.
 +
 +
===List Resources===
 +
To [https://docs.microsoft.com/en-us/powershell/module/failoverclusters/get-clusterresource?view=windowsserver2019-ps see] what's on the cluster:
 +
<source lang="powershell">
 +
Get-ClusterResource
 +
</source>

Latest revision as of 05:06, 18 August 2021

Some pointers on tuning loader machines.

Single Machine

Also known as a "laptop" configuration.

Install

We recommend you install Truxton on each loader node as follows:

  1. On the D: drive, make the following folders:
    1. Hashsets
    2. Temp
    3. Truxton Data\Search (this is actually two folders)
  2. Run the PostgreSQL installer (postgresql-12.4.1-windows-x64.exe in the Prerequisites folder of the installation media). Use the following settings:
    1. Data Directory: D:\Truxton Data\data
    2. Password: Truxton4n6
    3. Locale to be used: C
  3. Run Truxton_Installer.exe, do not use the defaults
    1. Change the "Depot Data Directory" to D:\Truxton Data

Temporary Folder

The tempdir setting controls the folder where Truxton ETLs will write their temporary files. In the case of PST files, this can be huge. Choose a fast drive to hold this.

Hashsets

The hashsetdir setting controls the folder where hashsets are stored. Any ETL process that produces files will need fast access to the files in this folder. Many reads will be performed on these files so this folder should be on a fast drive.

Truxton Forensic Rack

The Truxton Forensic Rack is a 7 foot tall rack populated with hardware dedicated to forensic processing. It has a Yellowbrick database appliance, seven dedicated loader/exploitation machines and 1PB of usable depot storage. With everything separated, it gives us many ways to configure processing.

Hardware

Here's how the hardware is stacked:

U42 Network Switch
U27 Yellowbrick
U26 Yellowbrick Manager 0
U25 Yellowbrick Manager 1
U23 KVM Server
U22 Keyboard and Monitor
U13 Loader Node 7
U11 Loader Node 6
U09 Loader Node 5
U07 Loader Node 4
U05 Loader Node 3
U03 Loader Node 2
U01 Loader Node 1

Analyst Desktop

One of the things to turn off is the automatic ANALYZE when the desktop starts. This is controlled by the skip_update_db_stats_on_boot desktop setting. Edit the %localappdata%\probity\truxton\client.settings file. Search for skip_update_db_stats_on_boot value. It should look like this: 

  "skip_update_db_stats_on_boot": {
    "OriginalType": "System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
    "Data": "false"
  },

Change the Data field to true 

  "skip_update_db_stats_on_boot": {
    "OriginalType": "System.Boolean, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
    "Data": "true"
  },

Server Architectures

With seven machines dedicated to forensic processing, we have a flexible platform to configure in a variety of ways.

Laptop

The "laptop" architecture uses the servers as seven individual machines. Each server hosts everything needed to run Truxton. The disadvantage of this configuration is that the other machines in the rack cannot help to process the data being loaded.

2x5

This configuration will dedicate two machines to database and message bus. The remaining 5 machines cooperate in exploiting the media being loaded.

Loader Node 1 (U01)

This machine hosts the singleton processes and low-volume processes.

  1. Load Status Monitor
  2. Message Bus
  3. SOLR Server
  4. Alert
  5. Identify
  6. Poly
  7. Report
  8. SOLR
  9. SOLRFile

Loader Node 2 (U03)

  1. Postgres Database Server - Configured for 1,000 connections
  2. Finished

Loader Nodes 3-7 (U05-U13)

These are the machines with lots to do. They host:

  1. Expand
  2. Poly File Expander
  3. Archives
  4. File Carver
  5. Video Contact Sheet
  6. EMail Parsing
  7. Load

Maintenance

Steps to Update Truxton on a Loader Node

  1. Stop the Truxton Service 
    net stop truxton
  2. Uninstall Truxton
  3. Install new Truxton
  4. Stop Les 
    net stop les
  5. Stop the Truxton Service 
    net stop truxton
  6. Click on the Truxton Status Monitor icon in the Windows Status Bar and select "Exit"
  7. Validate that TruxtonService.xml and TruxtonSettings.xml are correct
  8. In the Services application, select the properties of "Truxton Monitor (Les)" and change the Startup Type to "Disabled"
  9. Start the Truxton Service 
    net start truxton

There should be only one Les on the rack and the Truxton Status Monitor tray application is something for a human to look at.

Machine IDs

Normally the machine identifier is derived from the operating system but in a rack situation we sacrifice some entropy to provide system administrators assistance. The Machine ID for rack configurations are made up of the following fields:

Field Length Meaning
Rack 2 bytes 5452 TR for Truxton Rack
Rack Serial Number 2 bytes The serial number of the rack in hex encoded decimal
Rack Address Indicator 1 byte 55 U
Rack Address 1 byte Unit address in the rack of the machine
Logical Processors 2 bytes The number of logical CPUs in the machine
RAM 2 bytes The number of gigabytes of RAM in the machine
IP Address 6 bytes The IP address of the machine

For Example a Rack Machine ID of 54520001-5501-0072-0191-010010182131 tells us:

Bytes Meaning
5452 ASCII TR, this is a Rack Machine ID
0001 Serial Number 1
55 Unit Address
01 The machine is in slot U01 in the rack
0072 The machine has 72 logical processors
0191 The machine has 191GB of RAM
010010182131 The IP address of the machine is 10.10.182.131

The Machine ID corresponds to the [MachineID] column of the [Statistics] table. This allows you to see how the different machines perform their loads. Rack serial numbers in the 9000 range are development racks where new configurations are tested.

Loader Node Configuration Files

A pair of configuration files TruxtonSettings.xml and TruxtonService.xml are created and saved to make installation easy on virgin machines. TruxtonSettings.xml identifies the loader node and where the database and depots are. TruxtonService.xml governs which ETL processes will run on that machine.

Low-Volume Host

Low-volume is defined as something that isn't used very often. When it is, there's not much impact on the performance of this host machine. A low-volume ETL won't produce very many of anything.

TruxtonService.xml

The following is a sample configuration file for low-volume host. 

<root>
  <truxton_options>
  <!-- This Truxton service configuration file was created 2020-07-09 17:30:09 by TruxtonService.exe running as SYSTEM from the machine named LOADER01 (10.10.10.131). -->
    <service_configuration_version>1840023666788</service_configuration_version>
  <!-- There should only be one purger of expired media on your network -->
    <purger>yes</purger>
    <work_schedule>
    <!-- If this service is not employed, the service will run but no ETLs will be allowed to run -->
      <employed>yes</employed>
      <monday>
        <on_duty>0000-2359</on_duty>
      </monday>
      <tuesday>
        <on_duty>0000-2359</on_duty>
      </tuesday>
      <wednesday>
        <on_duty>0000-2359</on_duty>
      </wednesday>
      <thursday>
        <on_duty>0000-2359</on_duty>
      </thursday>
      <friday>
        <on_duty>0000-2359</on_duty>
      </friday>
      <saturday>
        <on_duty>0000-2359</on_duty>
      </saturday>
      <sunday>
        <on_duty>0000-2359</on_duty>
      </sunday>
    </work_schedule>
    <emptyqueuethresholdseconds>3600</emptyqueuethresholdseconds>
    <etls>
      <etl>
        <description>This consumes BOLOs and creates Alerts.</description>
        <exe controllable="yes" instances="1" queue="alert">Alert</exe>
      </etl>
      <etl>
        <description>This expands archive files.</description>
        <exe controllable="yes" instances="4" queue="archives">Archives</exe>
      </etl>
      <etl>
        <description>This carves free space for files.</description>
        <exe controllable="yes" instances="1" queue="carve">Carve</exe>
        <arguments>-carve_threads 0</arguments>
      </etl>
      <etl>
        <description>This generates video contact sheets, the 10x10 grid of images taken throughout the video.</description>
        <exe controllable="yes" instances="4" queue="contactsheet">ContactSheet</exe>
      </etl>
      <etl>
        <description>This parses MIME email files.</description>
        <exe controllable="yes" instances="1" queue="email">EMail</exe>
      </etl>
      <etl>
        <description>This is the main file expander service.</description>
        <exe controllable="yes" instances="10" queue="expand">Expand</exe>
      </etl>
      <etl>
        <description>This performs final processing after all files are present. It performs count queries and updates statistics.</description>
        <exe controllable="yes" instances="4" queue="finishedstage">Finished</exe>
      </etl>
      <etl>
        <description>This identifies file contents and routes accordingly.</description>
        <exe controllable="yes" instances="1" queue="identify">Identify</exe>
      </etl>
      <etl>
        <description>This is a loader as an ETL. It has the responsibility to expand files and load media.</description>
        <exe controllable="yes" instances="4" queue="load">Load</exe>
        <arguments>-lq load</arguments>
      </etl>
      <etl>
        <description>This coordinates the poly file expansion process.</description>
        <exe controllable="yes" instances="1" queue="poly">Poly</exe>
      </etl>
      <etl>
        <description>This finds the all of the pieces of multi-part archives and expands them.</description>
        <exe controllable="yes" instances="1" queue="pfe">PolyFileExpander</exe>
      </etl>
      <etl>
        <description>This exploits Windows registry files.</description>
        <exe controllable="yes" instances="1" queue="registry">Registry</exe>
      </etl>
      <etl>
        <description>This exports registry files to the local filesystem, spawns RegRipper.exe, grabs the result and makes it a child file of the registry file.</description>
        <exe controllable="yes" instances="1" queue="regripper">RegRipper</exe>
      </etl>
      <etl>
        <description>This spawns executables to expand files then kills them when done.</description>
        <exe controllable="yes" instances="1" queue="remoteexpand">RemoteFileExpander</exe>
      </etl>
      <etl>
        <description>This generates the reports.</description>
        <exe controllable="yes" instances="1" queue="report">Report</exe>
      </etl>
      <etl>
        <description>This keeps the SOLR service running.</description>
        <exe controllable="yes" instances="1" queue="solrcontentstage">SOLR</exe>
      </etl>
      <etl>
        <description>This sends files to SOLR for content indexing.</description>
        <exe controllable="yes" instances="1" queue="solrfile">SOLRFile</exe>
      </etl>
      <etl>
        <description>This reassembles fragments of carved files into the correct order for viewing.</description>
        <exe controllable="yes" instances="1" queue="stitch">Stitch</exe>
      </etl>
      <etl>
        <description>This extracts text from files.</description>
        <exe controllable="yes" instances="1" queue="tqueue">TextExtract</exe>
      </etl>
      <etl>
        <description>This generates small thumbnail images from larger images.</description>
        <exe controllable="yes" instances="8" queue="thumbnail">Thumbnail</exe>
      </etl>
      <etl>
        <description>This uses Yara to scan files for the rules you specify. Normally this is a malware scanner.</description>
        <exe controllable="yes" instances="1" queue="yara">Yara</exe>
      </etl>
    </etls>
    <services>{Solr 5 Server||$TextIndexerData$Search/bin/solr.cmd|start -p 8983 -m 32533m -a &quot;-XX:-UsePerfData&quot;|$TextIndexerData$Search/bin/solr.cmd|stop -p 8983|java.exe|jetty.port=8983},</services>
  <!-- The shutdownmachine boolean value tells Truxton if it should power down the server once all ETLs go idle -->
    <shutdownmachine>false</shutdownmachine>
  </truxton_options>
</root>

TruxtonSettings.xml

The following is a sample low-volume configuration file. 

<truxton_options>
  <hashsetdir>D:\Hashsets</hashsetdir>
  <tempdir>D:\Temp</tempdir>
  <fct>1201</fct>
  <mqconcentrator>Yes</mqconcentrator>
  <mbp>MBCConnie.dll</mbp>  
  <xsyslog>127.0.0.1</xsyslog>
  <xsyslogport>514</xsyslogport>
  <FileGroupsHaveBeenInitialized>1</FileGroupsHaveBeenInitialized>
  <dbserver>10.10.10.103</dbserver>
  <dbport>5432</dbport>
  <dbname>Truxton</dbname>
  <dbuser>postgres</dbuser>
  <dbpassword>Truxton4n6</dbpassword>
  <mqserver>10.10.10.131</mqserver>
  <mqport>5432</mqport>
  <mqname>TruxtonMessageBus</mqname>
  <mquser>postgres</mquser>
  <mqpassword>Truxton4n6</mqpassword>
  <datadir>C:\ClusterStorage\Depot01\Depot</datadir>
  <machineid>54520001-5501-0072-0191-010010010131</machineid>
  <solr_url>http://10.10.10.131:8983/solr/truxton-core</solr_url>
  <CreateTheDatabase>1</CreateTheDatabase>
</truxton_options>

High-Volume Host

High-volume is when an ETL will consume significant host machine resources or produce lots of files.

TruxtonService.xml

TruxtonSettings.xml

<truxton_options>
  <hashsetdir>D:\Hashsets</hashsetdir>
  <tempdir>D:\Temp</tempdir>
  <fct>1201</fct>
  <mqconcentrator>Yes</mqconcentrator>
  <mbp>MBCConnie.dll</mbp>  
  <xsyslog>127.0.0.1</xsyslog>
  <xsyslogport>514</xsyslogport>
  <FileGroupsHaveBeenInitialized>1</FileGroupsHaveBeenInitialized>
<dbserver>10.10.10.103</dbserver>
<dbport>5432</dbport>
<dbname>Truxton</dbname>
<dbuser>postgres</dbuser>
<dbpassword>password</dbpassword>
  <mqserver>10.10.10.131</mqserver>
  <mqport>5432</mqport>
  <mqname>TruxtonMessageBus</mqname>
  <mquser>postgres</mquser>
  <mqpassword>Truxton4n6</mqpassword>
<datadir>C:\ClusterStorage\Depot01\Depot</datadir>
<machineid>54520001-5501-0072-0191-010010010131</machineid>
<CreateTheDatabase>1</CreateTheDatabase>
<solr_url>http://10.10.10.131:8983/solr/truxton-core</solr_url>
</truxton_options>

Loader Node Installation Checklist

Here are the steps for installing Truxton as a loader node on the rack. We recommend that each loader node have an SSD dedicated to Truxton. This SSD, we will call it the D: drive, can be in the 200GB range or larger. Truxton will use it to host the hashsets and for temporary files. We do this because hashsets require a lot of seek and reads. If it is on an SSD, the speed will be dramatically improved over having it on a shared folder. Likewise, the temporary files folder should be on an SSD for improved IO performance.

  1. Make sure FIPS mode is off. FIPS is a standard for secure computing. When Windows FIPS mode is on, it makes sure that only approved cryptographic algorithms are used for high level tasks like signing documents or encrypting files. Specifically, Windows will disable the use of non-approved algorithms. The problem is, Microsoft gave developers a Crypto API that provided a common approach to low level tasks such as hashing. Other libraries (like .Net) depend on the Crypto API for performing MD5 hashing. When FIPS mode is on, MD5 is no longer part of the Crypto API and any code that relied on the Crypto API is now broken. This can lead to many hours of developer hilarity figuring out why programs stop working...
  2. Create C:\ProgramData\Truxton\Settings folder
  3. Copy the two configuration files into the Settings folder
  4. Modify the TruxtonSettings.xml file to set the machineid, set the rack and IP address fields
  5. Install Truxton
    1. Run Truxton_Installer.exe
    2. If asked "Should I install Java?" click the "No" button. Do not install Java
    3. If asked "Should I install PostgreSQL Database Server?" click the "No" button. Do not install PostgreSQL
    4. When asked "Would you like to use the default options when installing Truxton?" click the "Yes" button. Yes install with the default options (since we copied the settings files already)
  6. Stop the Truxton service
  7. Stop Les
  8. Set Les (Truxton Monitor) service to be Disabled from the Windows Services control panel.
  9. Create D:\Hashsets folder
  10. Create the D:\Temp folder
  11. Remove permissions from D:\Temp so the ETL processes can write to it.
  12. Copy the hashsets from the installation media to D:\Hashsets
  13. Restart the Truxton service
  14. Exit the tray app

Rack Management

There is a folder in the clustered storage named Rack Management which contains scripts and configuration files to ease the management of the rack.

Configurations

This sub-folder is home to the customized versions of TruxtonService.xml and TruxtonSettings.xml for each node. The naming convention for the files is the rack address, underscore, last octet of the server's IP address, underscore, and XML filename. For example, the TruxtonSettings.xml file to be used on the U01 machine at 10.10.182.131 will be named U01_131_TruxtonSettings.xml

Create a new sub-folder for each configuration you want the rack to use. The default configuration is in a sub-folder named Nodes

Installing Truxton

The InstallTruxton.cmd batch file will install the latest version of Truxton onto the machines in the rack. It assumes that Truxton is not running on the rack. You can stop Truxton by using the StopTruxton.cmd batch file. 

@ECHO OFF
start wmic /node:10.10.182.131 product call install true,"","C:\ClusterStorage\SampleData\Rack Management\Truxton Software\Truxton_Updates\Truxton.msi"
start wmic /node:10.10.182.132 product call install true,"","C:\ClusterStorage\SampleData\Rack Management\Truxton Software\Truxton_Updates\Truxton.msi"
start wmic /node:10.10.182.134 product call install true,"","C:\ClusterStorage\SampleData\Rack Management\Truxton Software\Truxton_Updates\Truxton.msi"
start wmic /node:10.10.182.135 product call install true,"","C:\ClusterStorage\SampleData\Rack Management\Truxton Software\Truxton_Updates\Truxton.msi"
start wmic /node:10.10.182.136 product call install true,"","C:\ClusterStorage\SampleData\Rack Management\Truxton Software\Truxton_Updates\Truxton.msi"
start wmic /node:10.10.182.137 product call install true,"","C:\ClusterStorage\SampleData\Rack Management\Truxton Software\Truxton_Updates\Truxton.msi"
start wmic /node:10.10.182.138 product call install true,"","C:\ClusterStorage\SampleData\Rack Management\Truxton Software\Truxton_Updates\Truxton.msi"
ECHO All nodes told to install, should be done in a couple of minutes
TIMEOUT /T 103
REM At this point, the default installations have been done
REM All nodes are local, we need them to team together
CALL StopTruxton.cmd
CALL ConfigTruxton.cmd
ECHO Finished.

Starting Truxton

The StartTruxton.cmd will start Truxton on all of the nodes of the rack. 

@ECHO OFF
ECHO Starting Truxton service on U01
sc \\10.10.182.131 start Truxton
ECHO Starting Truxton service on U03
sc \\10.10.182.132 start Truxton
ECHO Starting Truxton service on U05
sc \\10.10.182.134 start Truxton
ECHO Starting Truxton service on U07
sc \\10.10.182.135 start Truxton
ECHO Starting Truxton service on U09
sc \\10.10.182.136 start Truxton
ECHO Starting Truxton service on U11
sc \\10.10.182.137 start Truxton
ECHO Starting Truxton service on U13
sc \\10.10.182.138 start Truxton

Stopping Truxton

The StopTruxton.cmd batch file will stop Truxton on all of the nodes in the rack. 

@ECHO OFF
ECHO Stopping Truxton service on U01
sc \\10.10.182.131 stop Les
sc \\10.10.182.131 stop Truxton
ECHO Stopping Truxton service on U03
sc \\10.10.182.132 stop Les
sc \\10.10.182.132 stop Truxton
ECHO Stopping Truxton service on U05
sc \\10.10.182.134 stop Les
sc \\10.10.182.134 stop Truxton
ECHO Stopping Truxton service on U07
sc \\10.10.182.135 stop Les
sc \\10.10.182.135 stop Truxton
ECHO Stopping Truxton service on U09
sc \\10.10.182.136 stop Les
sc \\10.10.182.136 stop Truxton
ECHO Stopping Truxton service on U11
sc \\10.10.182.137 stop Les
sc \\10.10.182.137 stop Truxton
ECHO Stopping Truxton service on U13
sc \\10.10.182.138 stop Les
sc \\10.10.182.138 stop Truxton
TIMEOUT /T 7
REM Now go kill any hung processes
CALL KillETLProcesses.cmd 10.10.182.131
CALL KillETLProcesses.cmd 10.10.182.132
CALL KillETLProcesses.cmd 10.10.182.134
CALL KillETLProcesses.cmd 10.10.182.135
CALL KillETLProcesses.cmd 10.10.182.136
CALL KillETLProcesses.cmd 10.10.182.137
CALL KillETLProcesses.cmd 10.10.182.138

Configuring Truxton

The ConfigTruxton.cmd batch file will push the configuration files out to the nodes on the rack. This script will stop the tray application on all nodes and it will configure Les to be disabled on the worker nodes. 

@ECHO OFF
ECHO Configuring Truxton on U01
taskkill /s \\10.10.182.131 /FI "IMAGENAME eq TruxtonStatus.exe"
wmic /node:10.10.182.131 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U01_TruxtonService.xml\"  \"C:\ProgramData\Truxton\Settings\TruxtonService.xml"
wmic /node:10.10.182.131 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U01_TruxtonSettings.xml\" \"C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
ECHO Configuring Truxton on U03
taskkill /s \\10.10.182.132 /FI "IMAGENAME eq TruxtonStatus.exe"
sc \\10.10.182.132 config Les start=disabled
wmic /node:10.10.182.132 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U03_TruxtonService.xml\"  \"C:\ProgramData\Truxton\Settings\TruxtonService.xml"
wmic /node:10.10.182.132 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U03_TruxtonSettings.xml\" \"C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
ECHO Configuring Truxton on U05
taskkill /s \\10.10.182.134 /FI "IMAGENAME eq TruxtonStatus.exe"
sc \\10.10.182.134 config Les start=disabled
wmic /node:10.10.182.134 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U05_TruxtonService.xml\"  \"C:\ProgramData\Truxton\Settings\TruxtonService.xml"
wmic /node:10.10.182.134 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U05_TruxtonSettings.xml\" \"C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
ECHO Configuring Truxton on U07
taskkill /s \\10.10.182.135 /FI "IMAGENAME eq TruxtonStatus.exe"
sc \\10.10.182.135 config Les start=disabled
wmic /node:10.10.182.135 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U07_TruxtonService.xml\"  \"C:\ProgramData\Truxton\Settings\TruxtonService.xml"
wmic /node:10.10.182.135 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U07_TruxtonSettings.xml\" \"C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
ECHO Configuring Truxton on U09
taskkill /s \\10.10.182.136 /FI "IMAGENAME eq TruxtonStatus.exe"
sc \\10.10.182.136 config Les start=disabled
wmic /node:10.10.182.136 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U09_TruxtonService.xml\"  \"C:\ProgramData\Truxton\Settings\TruxtonService.xml"
wmic /node:10.10.182.136 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U09_TruxtonSettings.xml\" \"C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
ECHO Configuring Truxton on U11
taskkill /s \\10.10.182.137 /FI "IMAGENAME eq TruxtonStatus.exe"
sc \\10.10.182.137 config Les start=disabled
wmic /node:10.10.182.137 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U11_TruxtonService.xml\"  \"C:\ProgramData\Truxton\Settings\TruxtonService.xml"
wmic /node:10.10.182.137 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U11_TruxtonSettings.xml\" \"C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
ECHO Configuring Truxton on U13
taskkill /s \\10.10.182.138 /FI "IMAGENAME eq TruxtonStatus.exe"
sc \\10.10.182.138 config Les start=disabled
wmic /node:10.10.182.138 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U13_TruxtonService.xml\"  \"C:\ProgramData\Truxton\Settings\TruxtonService.xml"
wmic /node:10.10.182.138 process call create "cmd.exe /C copy \"C:\ClusterStorage\SampleData\Rack Management\Configurations\Nodes\U13_TruxtonSettings.xml\" \"C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"

Removing Truxton

The UninstallTruxton.cmd batch file will remove Truxton from all the nodes on the rack. 

@ECHO OFF
CALL StopTruxton.cmd
ECHO Uninstalling Truxton
START wmic /node:10.10.182.131 product where name="Truxton" call uninstall /nointeractive
START wmic /node:10.10.182.132 product where name="Truxton" call uninstall /nointeractive
START wmic /node:10.10.182.134 product where name="Truxton" call uninstall /nointeractive
START wmic /node:10.10.182.135 product where name="Truxton" call uninstall /nointeractive
START wmic /node:10.10.182.136 product where name="Truxton" call uninstall /nointeractive
START wmic /node:10.10.182.137 product where name="Truxton" call uninstall /nointeractive
START wmic /node:10.10.182.138 product where name="Truxton" call uninstall /nointeractive
ECHO All nodes told to uninstall

Updating Truxton

When you receive a new version of Truxton, update the MSI file and run UpdateTruxton.cmd 

@ECHO OFF
ECHO Updating Truxton
CALL UninstallTruxton.cmd
CALL InstallTruxton.cmd
CALL StartTruxton.cmd

Useful Batch Files

Remove Permissions

Here's the PowerShell script to remove security from a folder. 

$folder_path = "D:\Temp"
$security_descriptor = Get-Acl -Path $folder_path

# This is the SDDL for a NULL DACL
$sddl_for_a_null_dacl = ""

$security_descriptor.SetSecurityDescriptorSddlForm($sddl_for_a_null_dacl)
Set-Acl -Path $folder_path -AclObject $security_descriptor

Storage Spaces Direct

The Truxton rack uses Microsoft's Storage Spaces Direct (aka S2D) for managing disk space.

Planning Volumes

Here is a calculator to use when planning on creating a volume.

Creating a Volume

Here's how to create a 64TB volume for depots using the Powershell command New-Volume 

New-Volume -FriendlyName "GoldCopy" -Size 250TB -ResiliencySettingName Mirror -NumberOfDataCopies 2 -FileSystem CSVFS_ReFS -StoragePoolFriendlyName S2D*

Repairing a Volume

Sometimes you may see "Needs Repair" as its status.

List Resources

To see what's on the cluster: 

Get-ClusterResource
Retrieved from "https://truxwiki.com/index.php?title=Loader_Configuration&oldid=5179"