Configuration System

Description

The way to communicate settings to Truxton is through a class called Options. This is an interpretation of the Ye Olde "Command Line Options" of which much has been written. In Truxton, an option is prefaced by a name followed by the value. There is only one value per option. In this article, we may refer to an option as a parameter or setting.

Sources

The following are the nine different places that a parameter can come from. The lower the number, the greater the precedence that option has. The harder it is for a user to define an option, the greater weight that option will be given. It is harder for a user to specify an option on a command line because they must type it out. The user exerts no effort whatsoever to use a default value compiled into programs. Therefore, the command line option will be used instead of the compiled default.

1. Command Line Parameters
2. Command Line Configuration File
3. Command Line Database
4. Environment Variable
5. ETL Configuration File
6. ETL Machine File
7. Database by Machine
8. Truxton Settings File
9. Compiled Defaults

Command Line Parameters

If a human takes the time to physically type parameters on a command line, they must really mean that they want this value to be used. Anything specified on a command line takes precedence over any other location. The format of command line parameters is parameter name followed by a space and the parameter value.

load.exe -w 1 --xx abc /z "3 4 and 5" disk.dat

In the example above there are three options specified and one non-option.

Option Name	Option Value
w	1
xx	abc
z	3 4 and 5

The non-option is "disk.dat" A command line is the only way to specify a non-option. Option names can be prefaced with a single dash, double dash or a slash depending on your command line religion.

Command Line Configuration File

You can put the name of a configuration file on the command line.

Identify.exe my_settings.xml

The my_settings.xml would be assumed to be a file name since it was not part of an option. That file would then be opened, parsed as XML and anything found in a truxton_options section would be used as configuration options. All XML element names used in Truxton are lower case.

The settings would look like this:

<truxton_options>
  <w>1</w>
  <xx>abc</xx>
  <z>3 4 and 5</z>
</truxton_options>

Command Line Database

If you specify a settings identifier on the command line, Truxton will retrieve that setting from the database and use it. The GUID you specify should match the [ID] column of the [Setting] table. If a record is found, it will then parse the XML contents of the [Value] column.

Identify.exe 51256fb4-c7f6-327a-2703-325e219ecc2b

If you actually use a random value you will have no end of mistakes. We suggest making fake ids for settings and using those. It will make typing it in far easier. Here's a fake setting that can be easily typed and remembered by a human.

Identify.exe 12345678-1234-1234-1234-123456789abc

Environment Variable

Configuration options can also be set using environment variables. These variables must have names that begin with Truxton_ For example:

set Truxton_w=1
set Truxton_xx=abc
set Truxton_z="3 4 and 5"
Load.exe disk.dat

ETL Configuration File

Another location is an XML configuration file specifically for this type of ETL process. Truxton will look for a file in the same directory as the executable with config as the extension. For example, the Identify.exe ETL process will look for a file named Identify.config and parse it if found.

ETL Machine File

This is a configuration file with customized settings valid for the machine that runs the ETL processes. Truxton will look for a file in the same directory as the TruxtonSettings.xml file named TruxtonSettings.machine as the extension.

Machine configuration is usually used for specifying common settings that are unique to a machine. For example, every ETL process needs some sort of temporary directory. Some of your ETL machines may have RAM disks, others may have hard drives with differing drive letters. Machine configuration files allow you to customize where the temp folder is for each machine.

Database by Machine

Each installation of Truxton has a globally unique machine identifier. This can be found in the TruxtonObject#machineid:_str element of the TruxtonSettings.XML file. Truxton will connect to the database and query the [Setting] table for a record with the machine identifier in the [ID] column. If found, the XML in the Value column will be parsed for options.

Truxton Settings File

Each installation has a TruxtonSettings.xml file. It is found in the %programdata%\Truxton\Settings folder which usually expands to C:\ProgramData\Truxton\Settings

NOTE: The ProgramData folder is normally hidden and won't show up in directory listings (or File Explorer).

Its primary purpose is to store the globally unique identifier for the machine which was generated during installation. The default database connection parameters are also stored here.

As you can see, the TruxtonSettings.xml file has very low priority.

Compiled Defaults

Each ETL member can have default options built into it. If the option is not specified anywhere above, whatever was compiled into the executable will be used. This is the last resort.

Debugging the Options

With nine different places to get options from, how could anything go wrong? If you need help debugging why an option isn't getting the value you want, use the debugoptions option.

Identify.exe -debugoptions 1

This will cause an installation log to be created and the options written to it. This log file is found in the Truxton\OptionsDebugging directory under the %programdata% folder (which usually expands to C:\ProgramData) folder. An installation log is the only log that can be created safely if everything else fails.

Sample Options Log

Log file created 2022-07-25 12:26:07 by "C:\Program Files\Truxton\Loader\Identify.exe" running as Sam on the machine named DESKTOP-5IFOKKS (192.168.1.88).
2022-07-25T16:26:06Z 2CAC-84D4 : Parsing 3 command line arguments
2022-07-25T16:26:06Z 2CAC-84D4 : First command line argument was something that ended in .exe, skipping it
2022-07-25T16:26:06Z 2CAC-84D4 : Beginning to parse the command line arguments at array index 1
2022-07-25T16:26:06Z 2CAC-84D4 : command_line_argument[ 1 ] = "-debugoptions"
2022-07-25T16:26:06Z 2CAC-84D4 : command_line_argument[ 2 ] = "1" is the value of option "debugoptions"
2022-07-25T16:26:06Z 2CAC-84D4 : No existing option named "debugoptions" was found, creating a new one
2022-07-25T16:26:06Z 2CAC-84D4 : Finished parsing 3 command line arguments, there are now 0 NotOptions
2022-07-25T16:26:06Z 2CAC-84D4 : Processing "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml" as TruxtonSettings.xml(C:\ProgramData\Truxton\Settings\TruxtonSettings.xml)
2022-07-25T16:26:06Z 2CAC-84D4 : Processing "C:\Program Files\Truxton\Loader\Identify.config" as the ETL.config file
2022-07-25T16:26:06Z 2CAC-84D4 : Can't parse "C:\Program Files\Truxton\Loader\Identify.config" as XML
2022-07-25T16:26:06Z 2CAC-84D4 : Processing "C:\ProgramData\Truxton\Settings\TruxtonSettings.machine" as the ETL.machine file
2022-07-25T16:26:06Z 2CAC-84D4 : Can't parse "C:\ProgramData\Truxton\Settings\TruxtonSettings.machine" as XML
2022-07-25T16:26:06Z 2CAC-84D4 : Getting Options from Truxton Database
2022-07-25T16:26:06Z 2CAC-84D4 : All Option Dump (The first entry of duplicate lines is the value that will be used by the program.)
Process "C:\Program Files\Truxton\Loader\Identify.exe"
createthedatabase : "1" from TruxtonSettings.xml XML element name "truxton_options.CreateTheDatabase" beginning at line 6 column 3 byte 382 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
datadir : "C:\Truxton Data" from TruxtonSettings.xml XML element name "truxton_options.datadir" beginning at line 4 column 3 byte 281 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbconnectionstring : "Host=localhost;Port=5432;Database=Truxton;Username=postgres;Password=Truxton4n6;" from TruxtonSettings.xml XML element name "truxton_options.dbconnectionstring" beginning at line 2 column 3 byte 21 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
debugoptions : "1" from Command Line argument 2 (zero is the first)
filegroupshavebeeninitialized : "1" from TruxtonSettings.xml XML element name "truxton_options.FileGroupsHaveBeenInitialized" beginning at line 16 column 3 byte 833 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
machineid : "671B7D4B-0D2E-4A00-84D6-FD9F6CA1DEBA" from TruxtonSettings.xml XML element name "truxton_options.machineid" beginning at line 5 column 3 byte 319 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mbconnectionstring : "Host=localhost;Port=5432;Database=TruxtonMessageBus;Username=postgres;Password=Truxton4n6;" from TruxtonSettings.xml XML element name "truxton_options.mbconnectionstring" beginning at line 3 column 3 byte 146 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
solr_url : "http://localhost:8983/solr/truxton-core" from TruxtonSettings.xml XML element name "truxton_options.solr_url" beginning at line 7 column 3 byte 426 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
who_wrote_this : "Written by SYSTEM at 2022-07-24 08:29:10 (local time) on DESKTOP-5IFOKKS process id 3432, version 4.2.1.724, command line: "C:\Program Files\Truxton\Loader\Load.exe" --create_truxton_settings_xml Yes --dbconnectionstring "Host=localhost;Port=5432;Database=Truxton;Username=postgres;Password=Truxton4n6;" --mbconnectionstring "Host=localhost;Port=5432;Database=TruxtonMessageBus;Username=postgres;Password=Truxton4n6;" --datadir "C:\Truxton Data" --machineid "671B7D4B-0D2E-4A00-84D6-FD9F6CA1DEBA" --solr_url "http://localhost:8983/solr/truxton-core" --CreateTheDatabase 1" from TruxtonSettings.xml XML element name "truxton_options.who_wrote_this" beginning at line 53 column 3 byte 3993 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"

End of Options, Start of Non-Options

End of All Option Dump

2022-07-25T16:26:06Z 2CAC-84D4 : Parsing 3 command line arguments
2022-07-25T16:26:06Z 2CAC-84D4 : First command line argument was something that ended in .exe, skipping it
2022-07-25T16:26:06Z 2CAC-84D4 : Beginning to parse the command line arguments at array index 1
2022-07-25T16:26:06Z 2CAC-84D4 : command_line_argument[ 1 ] = "-debugoptions"
2022-07-25T16:26:06Z 2CAC-84D4 : command_line_argument[ 2 ] = "1" is the value of option "debugoptions"
2022-07-25T16:26:06Z 2CAC-84D4 : No existing option named "debugoptions" was found, creating a new one
2022-07-25T16:26:06Z 2CAC-84D4 : Finished parsing 3 command line arguments, there are now 0 NotOptions
2022-07-25T16:26:06Z 2CAC-84D4 : Processing "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml" as TruxtonSettings.xml(C:\ProgramData\Truxton\Settings\TruxtonSettings.xml)
2022-07-25T16:26:06Z 2CAC-84D4 : Processing "C:\Program Files\Truxton\Loader\Identify.config" as the ETL.config file
2022-07-25T16:26:06Z 2CAC-84D4 : Can't parse "C:\Program Files\Truxton\Loader\Identify.config" as XML
2022-07-25T16:26:06Z 2CAC-84D4 : Processing "C:\ProgramData\Truxton\Settings\TruxtonSettings.machine" as the ETL.machine file
2022-07-25T16:26:06Z 2CAC-84D4 : Can't parse "C:\ProgramData\Truxton\Settings\TruxtonSettings.machine" as XML
2022-07-25T16:26:06Z 2CAC-84D4 : Getting Options from Truxton Database
2022-07-25T16:26:06Z 2CAC-84D4 : All Option Dump (The first entry of duplicate lines is the value that will be used by the program.)
Process "C:\Program Files\Truxton\Loader\Identify.exe"
createthedatabase : "1" from TruxtonSettings.xml XML element name "truxton_options.CreateTheDatabase" beginning at line 6 column 3 byte 382 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
datadir : "C:\Truxton Data" from TruxtonSettings.xml XML element name "truxton_options.datadir" beginning at line 4 column 3 byte 281 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbconnectionstring : "Host=localhost;Port=5432;Database=Truxton;Username=postgres;Password=Truxton4n6;" from TruxtonSettings.xml XML element name "truxton_options.dbconnectionstring" beginning at line 2 column 3 byte 21 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
debugoptions : "1" from Command Line argument 2 (zero is the first)
filegroupshavebeeninitialized : "1" from TruxtonSettings.xml XML element name "truxton_options.FileGroupsHaveBeenInitialized" beginning at line 16 column 3 byte 833 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
machineid : "671B7D4B-0D2E-4A00-84D6-FD9F6CA1DEBA" from TruxtonSettings.xml XML element name "truxton_options.machineid" beginning at line 5 column 3 byte 319 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mbconnectionstring : "Host=localhost;Port=5432;Database=TruxtonMessageBus;Username=postgres;Password=Truxton4n6;" from TruxtonSettings.xml XML element name "truxton_options.mbconnectionstring" beginning at line 3 column 3 byte 146 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
solr_url : "http://localhost:8983/solr/truxton-core" from TruxtonSettings.xml XML element name "truxton_options.solr_url" beginning at line 7 column 3 byte 426 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
who_wrote_this : "Written by SYSTEM at 2022-07-24 08:29:10 (local time) on DESKTOP-5IFOKKS process id 3432, version 4.2.1.724, command line: "C:\Program Files\Truxton\Loader\Load.exe" --create_truxton_settings_xml Yes --dbconnectionstring "Host=localhost;Port=5432;Database=Truxton;Username=postgres;Password=Truxton4n6;" --mbconnectionstring "Host=localhost;Port=5432;Database=TruxtonMessageBus;Username=postgres;Password=Truxton4n6;" --datadir "C:\Truxton Data" --machineid "671B7D4B-0D2E-4A00-84D6-FD9F6CA1DEBA" --solr_url "http://localhost:8983/solr/truxton-core" --CreateTheDatabase 1" from TruxtonSettings.xml XML element name "truxton_options.who_wrote_this" beginning at line 53 column 3 byte 3993 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"

End of Options, Start of Non-Options

End of All Option Dump

2022-07-25T16:26:06Z 2CAC-84D4 : Parsing 3 command line arguments
2022-07-25T16:26:06Z 2CAC-84D4 : First command line argument was something that ended in .exe, skipping it
2022-07-25T16:26:06Z 2CAC-84D4 : Beginning to parse the command line arguments at array index 1
2022-07-25T16:26:06Z 2CAC-84D4 : command_line_argument[ 1 ] = "-debugoptions"
2022-07-25T16:26:06Z 2CAC-84D4 : command_line_argument[ 2 ] = "1" is the value of option "debugoptions"
2022-07-25T16:26:06Z 2CAC-84D4 : No existing option named "debugoptions" was found, creating a new one
2022-07-25T16:26:06Z 2CAC-84D4 : Finished parsing 3 command line arguments, there are now 0 NotOptions
2022-07-25T16:26:06Z 2CAC-84D4 : Processing "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml" as TruxtonSettings.xml(C:\ProgramData\Truxton\Settings\TruxtonSettings.xml)
2022-07-25T16:26:06Z 2CAC-84D4 : Processing "C:\Program Files\Truxton\Loader\Identify.config" as the ETL.config file
2022-07-25T16:26:06Z 2CAC-84D4 : Can't parse "C:\Program Files\Truxton\Loader\Identify.config" as XML
2022-07-25T16:26:06Z 2CAC-84D4 : Processing "C:\ProgramData\Truxton\Settings\TruxtonSettings.machine" as the ETL.machine file
2022-07-25T16:26:06Z 2CAC-84D4 : Can't parse "C:\ProgramData\Truxton\Settings\TruxtonSettings.machine" as XML
2022-07-25T16:26:06Z 2CAC-84D4 : Getting Options from Truxton Database
2022-07-25T16:26:06Z 2CAC-84D4 : All Option Dump (The first entry of duplicate lines is the value that will be used by the program.)
Process "C:\Program Files\Truxton\Loader\Identify.exe"
aci : "10007" from Default definitions
alert_email_from_address : "Truxton@localhost" from Default definitions
alert_email_subject : "[Truxton Alert]" from Default definitions
carve : "1" from Default definitions
carve_threads : "1" from Default definitions
carvemode : "0" from Default definitions
carveoffset : "65536" from Default definitions
carveunknown : "0" from Default definitions
cmid : "0" from Default definitions
controlledby : "0" from Default definitions
createthedatabase : "1" from TruxtonSettings.xml XML element name "truxton_options.CreateTheDatabase" beginning at line 6 column 3 byte 382 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
cunk : "1" from Default definitions
cws : "10485760" from Default definitions
datadir : "C:\Truxton Data" from TruxtonSettings.xml XML element name "truxton_options.datadir" beginning at line 4 column 3 byte 281 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbconnectionstring : "Host=localhost;Port=5432;Database=Truxton;Username=postgres;Password=Truxton4n6;" from TruxtonSettings.xml XML element name "truxton_options.dbconnectionstring" beginning at line 2 column 3 byte 21 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbflavor : "1" from Default definitions
debugfileidmapping : "0" from Default definitions
debugoptions : "1" from Command Line argument 2 (zero is the first)
dedupemessages : "0" from Default definitions
deepsleep : "45007" from Default definitions
deletedfiles : "1" from Default definitions
depotmaxsize : "1099511627776" from Default definitions
depotpagesize : "4096" from Default definitions
ds : "0" from Default definitions
ee : "0" from Default definitions
excludefilesbytype : "0" from Default definitions
excludes : "0" from Default definitions
fct : "306" from Default definitions
filegroupshavebeeninitialized : "1" from TruxtonSettings.xml XML element name "truxton_options.FileGroupsHaveBeenInitialized" beginning at line 16 column 3 byte 833 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
fileslack : "1" from Default definitions
fqhi : "7500" from Default definitions
fqlo : "100" from Default definitions
hashsetall : "0" from Default definitions
iei : "1" from Default definitions
imagelength : "-1" from Default definitions
imageoffset : "0" from Default definitions
includefilesbytype : "0" from Default definitions
includes : "0" from Default definitions
istatus : "1" from Default definitions
itype : "0" from Default definitions
loadas : "0" from Default definitions
lq : "load" from Default definitions
machineid : "671B7D4B-0D2E-4A00-84D6-FD9F6CA1DEBA" from TruxtonSettings.xml XML element name "truxton_options.machineid" beginning at line 5 column 3 byte 319 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
maxsleep : "600101" from Default definitions
mbconnectionstring : "Host=localhost;Port=5432;Database=TruxtonMessageBus;Username=postgres;Password=Truxton4n6;" from TruxtonSettings.xml XML element name "truxton_options.mbconnectionstring" beginning at line 3 column 3 byte 146 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
metacarve : "1" from Default definitions
minsleep : "3011" from Default definitions
mlat : "0" from Default definitions
mlong : "0" from Default definitions
mqconcentrator : "0" from Default definitions
mr : "0" from Default definitions
mt : "0" from Default definitions
nocontents : "0" from Default definitions
nohashsets : "0" from Default definitions
norouting : "0" from Default definitions
nostartdelay : "0" from Default definitions
nostore : "0" from Default definitions
numberofbuffers : "2048" from Default definitions
numberofbytesperbuffer : "65536" from Default definitions
numberoffiles : "9223372036854775807" from Default definitions
pfiles : "1" from Default definitions
pfree : "1" from Default definitions
priority : "1000" from Default definitions
rapid : "0" from Default definitions
recovery : "0" from Default definitions
reprocess : "0" from Default definitions
skipdupmedia : "0" from Default definitions
smtp_server : "localhost" from Default definitions
solr_url : "http://localhost:8983/solr/truxton-core" from TruxtonSettings.xml XML element name "truxton_options.solr_url" beginning at line 7 column 3 byte 426 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
startatmft : "0" from Default definitions
syslogport : "514" from Default definitions
timeout : "120" from Default definitions
triage : "0" from Default definitions
tsk_verbose : "0" from Default definitions
updatemedianumberofchildren : "1" from Default definitions
updatepathid : "1" from Default definitions
verbose : "0" from Default definitions
wait : "none" from Default definitions
who_wrote_this : "Written by SYSTEM at 2022-07-24 08:29:10 (local time) on DESKTOP-5IFOKKS process id 3432, version 4.2.1.724, command line: "C:\Program Files\Truxton\Loader\Load.exe" --create_truxton_settings_xml Yes --dbconnectionstring "Host=localhost;Port=5432;Database=Truxton;Username=postgres;Password=Truxton4n6;" --mbconnectionstring "Host=localhost;Port=5432;Database=TruxtonMessageBus;Username=postgres;Password=Truxton4n6;" --datadir "C:\Truxton Data" --machineid "671B7D4B-0D2E-4A00-84D6-FD9F6CA1DEBA" --solr_url "http://localhost:8983/solr/truxton-core" --CreateTheDatabase 1" from TruxtonSettings.xml XML element name "truxton_options.who_wrote_this" beginning at line 53 column 3 byte 3993 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
xorcarve : "1" from Default definitions

End of Options, Start of Non-Options

End of All Option Dump

Data Types

There are only three types of data for options.

String

String data is any series of characters.

Integer

The string data is interpreted as a sixty-four bit unsigned integer. The option system will automatically convert hexadecimal strings for you if they begin with x or 0x. For example, of you have an integer field specified as x2a and read it as an integer value, 42 will be returned.

Boolean

Booleans are either true or false. The string data will be interpreted according to these rules:

• If the string value is true or yes, the returned value will be true.
• If the string value interpreted as an integer is not zero, the returned value will be true.
• Return value is false in all other cases.

ReFS Deduplication

It looks like Windows Storage Spaces has limitations on deduplication of volumes. It appears that the maximum volume size is 64TB while only the first 4TB of a file will be deduplicated.