Difference between revisions of "Configuration System"
| Line 71: | Line 71: | ||
==Command Line Database== | ==Command Line Database== | ||
If you specify a settings identifier on the command line, Truxton will retrieve that setting from the database and use it. | If you specify a settings identifier on the command line, Truxton will retrieve that setting from the database and use it. | ||
| − | The [https://en.wikipedia.org/wiki/Universally_unique_identifier GUID] you specify should match the <code>ID</code> column of the <code>Setting</code> table. | + | The [https://en.wikipedia.org/wiki/Universally_unique_identifier GUID] you specify should match the <code>[ID]</code> column of the <code>[Setting]</code> table. |
| − | If a record is found, it will then parse the XML contents of the <code>Value</code> column. | + | If a record is found, it will then parse the XML contents of the <code>[Value]</code> column. |
| − | < | + | <source lang="bat"> |
Identify.exe 51256fb4-c7f6-327a-2703-325e219ecc2b | Identify.exe 51256fb4-c7f6-327a-2703-325e219ecc2b | ||
| − | </ | + | </source> |
If you actually use a random value you will have no end of mistakes. | If you actually use a random value you will have no end of mistakes. | ||
| Line 82: | Line 82: | ||
It will make typing it in far easier. Here's a fake setting that can be easily typed and remembered by a human. | It will make typing it in far easier. Here's a fake setting that can be easily typed and remembered by a human. | ||
| − | < | + | <source lang="bat"> |
Identify.exe 12345678-1234-1234-1234-123456789abc | Identify.exe 12345678-1234-1234-1234-123456789abc | ||
| − | </ | + | </source> |
==Environment Variable== | ==Environment Variable== | ||
Revision as of 03:36, 4 December 2020
Description
The way to communicate settings to Truxton is through a class called Options.
This is an interpretation of the Ye Olde "Command Line Options" of which much has been written.
In Truxton, an option is prefaced by a name followed by the value.
There is only one value per option.
In this article, we may refer to an option as a parameter or setting.
Sources
The following are the nine different places that a parameter can come from. The lower the number, the greater the precedence that option has. The harder it is for a user to define an option, the greater weight that option will be given. It is harder for a user to specify an option on a command line because they must type it out. The user exerts no effort whatsoever to use a default value compiled into programs. Therefore, the command line option will be used instead of the compiled default.
- Command Line Parameters
- Command Line Configuration File
- Command Line Database
- Environment Variable
- ETL Configuration File
- ETL Machine File
- Database by Machine
- Truxton Settings File
- Compiled Defaults
Command Line Parameters
If a human takes the time to physically type parameters on a command line, they must really mean that they want this value to be used. Anything specified on a command line takes precedence over any other location. The format of command line parameters is parameter name followed by a space and the parameter value.
load.exe -w 1 --xx abc /z "3 4 and 5" disk.dat
In the example above there are three options specified and one non-option.
| Option Name | Option Value |
|---|---|
| w | 1 |
| xx | abc |
| z | 3 4 and 5 |
The non-option is "disk.dat" A command line is the only way to specify a non-option. Option names can be prefaced with a single dash, double dash or a slash depending on your command line religion.
Command Line Configuration File
You can put the name of a configuration file on the command line.
Identify.exe my_settings.xml
The my_settings.xml would be assumed to be a file name since it was not part of an option.
That file would then be opened, parsed as XML and anything found in a truxton_options section would be used as configuration options.
All XML element names used in Truxton are lower case.
The settings would look like this:
<truxton_options>
<w>1</w>
<xx>abc</xx>
<z>3 4 and 5</z>
</truxton_options>
Command Line Database
If you specify a settings identifier on the command line, Truxton will retrieve that setting from the database and use it.
The GUID you specify should match the [ID] column of the [Setting] table.
If a record is found, it will then parse the XML contents of the [Value] column.
Identify.exe 51256fb4-c7f6-327a-2703-325e219ecc2b
If you actually use a random value you will have no end of mistakes. We suggest making fake ids for settings and using those. It will make typing it in far easier. Here's a fake setting that can be easily typed and remembered by a human.
Identify.exe 12345678-1234-1234-1234-123456789abc
Environment Variable
Configuration options can also be set using environment variables.
These variables must have names that begin with Truxton_
For example:
set Truxton_w=1
set Truxton_xx=abc
set Truxton_z="3 4 and 5"
Load.exe disk.dat
ETL Configuration File
Another location is an XML configuration file specifically for this type of ETL process.
Truxton will look for a file in the same directory as the executable with config as the extension.
For example, the Identify.exe ETL process will look for a file named Identify.config and parse it if found.
ETL Machine File
This is a configuration file with customized settings valid for the machine that runs the ETL processes.
Truxton will look for a file in the same directory as the TruxtonSettings.xml file named TruxtonSettings.machine as the extension.
Machine configuration is usually used for specifying common settings that are unique to a machine. For example, every ETL process needs some sort of temporary directory. Some of your ETL machines may have RAM disks, others may have hard drives with differing drive letters. Machine configuration files allow you to customize where the temp folder is for each machine.
Database by Machine
Each installation of Truxton has a globally unique machine identifier.
This can be found in the machineid element of the TruxtonSettings.XML file.
Truxton will connect to the database and query the Setting table for a record with the machine identifier in the ID column.
If found, the XML in the Value column will be parsed for options.
Truxton Settings File
Each installation has a TruxtonSettings.xml file.
It is found in the %programdata%\Truxton\Settings folder which usually expands to C:\ProgramData\Truxton\Settings
NOTE: The ProgramData folder is normally hidden and won't show up in directory listings (or File Explorer).
Its primary purpose is to store the globally unique identifier for the machine which was generated during installation. The default database connection parameters are also stored here.
As you can see, the TruxtonSettings.xml file has very low priority.
Compiled Defaults
Each ETL member can have default options built into it. If the option is not specified anywhere above, whatever was compiled into the executable will be used. This is the last resort.
Debugging the Options
With nine different places to get options from, how could anything go wrong? If you need help debugging why an option isn't getting the value you want, use the debugoptions option.
Identify.exe -debugoptions 1
This will cause an installation log to be created and the options written to it.
This log file is found in the Truxton\OptionsDebugging directory under the %programdata% folder (which usually expands to C:\ProgramData) folder.
An installation log is the only log that can be created safely if everything else fails.
Sample Options Log
Log file created by "C:\Program Files\Truxton\Loader\Identify.exe"
2019-10-16T12:31:21Z 4454-6528 : Processing "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml" as TruxtonSettings.xml(C:\ProgramData\Truxton\Settings\TruxtonSettings.xml)
2019-10-16T12:31:21Z 4454-6528 : Getting Options from Truxton Database
2019-10-16T12:31:21Z 4454-6528 : Parsing 3 command line arguments
2019-10-16T12:31:21Z 4454-6528 : Beginning to parse the command line arguments at array index 0
2019-10-16T12:31:21Z 4454-6528 : command_line_argument[ 0 ] = "identify"
2019-10-16T12:31:21Z 4454-6528 : command_line_argument[ 0 ] ("identify") is being added as a NotOption.
2019-10-16T12:31:21Z 4454-6528 : command_line_argument[ 0 ] ("identify") was added as a NotOption number 0.
2019-10-16T12:31:21Z 4454-6528 : command_line_argument[ 1 ] = "-debugoptions"
2019-10-16T12:31:21Z 4454-6528 : command_line_argument[ 2 ] = "1" is the value of option "debugoptions"
2019-10-16T12:31:21Z 4454-6528 : No existing option named "debugoptions" was found, creating a new one
2019-10-16T12:31:21Z 4454-6528 : Finished parsing 3 command line arguments, there are now 1 NotOptions
2019-10-16T12:31:21Z 4454-6528 : Processing "C:\Program Files\Truxton\Loader\Identify.config" as the ETL.config file
2019-10-16T12:31:21Z 4454-6528 : Can't parse "C:\Program Files\Truxton\Loader\Identify.config" as XML
2019-10-16T12:31:21Z 4454-6528 : Processing "C:\ProgramData\Truxton\Settings\TruxtonSettings.machine" as the ETL.machine file
2019-10-16T12:31:21Z 4454-6528 : Can't parse "C:\ProgramData\Truxton\Settings\TruxtonSettings.machine" as XML
2019-10-16T12:31:21Z 4454-6528 : All Option Dump (The first entry of duplicate lines is the value that will be used by the program.)
Process "C:\Program Files\Truxton\Loader\Identify.exe"
createthedatabase : "1" from TruxtonSettings.xml XML element name truxton_options.CreateTheDatabase beginning at line 15 column 3 byte 485 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
datadir : "C:\Truxton Data" from TruxtonSettings.xml XML element name truxton_options.datadir beginning at line 13 column 3 byte 384 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbname : "Truxton" from TruxtonSettings.xml XML element name truxton_options.dbname beginning at line 5 column 3 byte 123 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbpassword : "Truxton4n6" from TruxtonSettings.xml XML element name truxton_options.dbpassword beginning at line 7 column 3 byte 180 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbport : "5432" from TruxtonSettings.xml XML element name truxton_options.dbport beginning at line 4 column 3 byte 98 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbserver : "localhost" from TruxtonSettings.xml XML element name truxton_options.dbserver beginning at line 3 column 3 byte 64 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbsr1 :
dbuser : "postgres" from TruxtonSettings.xml XML element name truxton_options.dbuser beginning at line 6 column 3 byte 151 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
debugoptions : "1" from Command Line argument 2 (zero is the first)
filegroupshavebeeninitialized : "1" from TruxtonSettings.xml XML element name truxton_options.FileGroupsHaveBeenInitialized beginning at line 17 column 3 byte 593 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
machineid : "64B30278-F250-4896-82EB-1AC416DB30B2" from TruxtonSettings.xml XML element name truxton_options.machineid beginning at line 14 column 3 byte 422 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mqname : "TruxtonMessageBus" from TruxtonSettings.xml XML element name truxton_options.mqname beginning at line 10 column 3 byte 278 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mqpassword : "Truxton4n6" from TruxtonSettings.xml XML element name truxton_options.mqpassword beginning at line 12 column 3 byte 345 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mqport : "5432" from TruxtonSettings.xml XML element name truxton_options.mqport beginning at line 9 column 3 byte 253 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mqserver : "localhost" from TruxtonSettings.xml XML element name truxton_options.mqserver beginning at line 8 column 3 byte 219 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mquser : "postgres" from TruxtonSettings.xml XML element name truxton_options.mquser beginning at line 11 column 3 byte 316 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
solr_url : "http://localhost:8983/solr/truxton-core" from TruxtonSettings.xml XML element name truxton_options.solr_url beginning at line 16 column 3 byte 529 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
writesettings :
End of Options, Start of Non-Options
"identify"
End of All Option Dump
2019-10-16T12:31:21Z 4454-6528 : Processing "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml" as TruxtonSettings.xml(C:\ProgramData\Truxton\Settings\TruxtonSettings.xml)
2019-10-16T12:31:21Z 4454-6528 : Getting Options from Truxton Database
2019-10-16T12:31:21Z 4454-6528 : Parsing 3 command line arguments
2019-10-16T12:31:21Z 4454-6528 : Beginning to parse the command line arguments at array index 0
2019-10-16T12:31:21Z 4454-6528 : command_line_argument[ 0 ] = "identify"
2019-10-16T12:31:21Z 4454-6528 : command_line_argument[ 0 ] ("identify") is being added as a NotOption.
2019-10-16T12:31:21Z 4454-6528 : command_line_argument[ 0 ] ("identify") was added as a NotOption number 0.
2019-10-16T12:31:21Z 4454-6528 : command_line_argument[ 1 ] = "-debugoptions"
2019-10-16T12:31:21Z 4454-6528 : command_line_argument[ 2 ] = "1" is the value of option "debugoptions"
2019-10-16T12:31:21Z 4454-6528 : No existing option named "debugoptions" was found, creating a new one
2019-10-16T12:31:21Z 4454-6528 : Finished parsing 3 command line arguments, there are now 1 NotOptions
2019-10-16T12:31:21Z 4454-6528 : Processing "C:\Program Files\Truxton\Loader\Identify.config" as the ETL.config file
2019-10-16T12:31:21Z 4454-6528 : Can't parse "C:\Program Files\Truxton\Loader\Identify.config" as XML
2019-10-16T12:31:21Z 4454-6528 : Processing "C:\ProgramData\Truxton\Settings\TruxtonSettings.machine" as the ETL.machine file
2019-10-16T12:31:21Z 4454-6528 : Can't parse "C:\ProgramData\Truxton\Settings\TruxtonSettings.machine" as XML
2019-10-16T12:31:21Z 4454-6528 : All Option Dump (The first entry of duplicate lines is the value that will be used by the program.)
Process "C:\Program Files\Truxton\Loader\Identify.exe"
aci : "10007" from Default definitions
alert_email_from_address : "Truxton@localhost" from Default definitions
alert_email_subject : "[Truxton Alert]" from Default definitions
carve : "1" from Default definitions
carve_threads : "1" from Default definitions
carvemode : "0" from Default definitions
carveoffset : "65536" from Default definitions
carveunknown : "0" from Default definitions
cmid : "0" from Default definitions
controlledby : "0" from Default definitions
createthedatabase : "1" from TruxtonSettings.xml XML element name truxton_options.CreateTheDatabase beginning at line 15 column 3 byte 485 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
cunk : "1" from Default definitions
cws : "10485760" from Default definitions
datadir : "C:\Truxton Data" from TruxtonSettings.xml XML element name truxton_options.datadir beginning at line 13 column 3 byte 384 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbname : "Truxton" from TruxtonSettings.xml XML element name truxton_options.dbname beginning at line 5 column 3 byte 123 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbname : "Truxton" from Default definitions
dbpassword : "Truxton4n6" from TruxtonSettings.xml XML element name truxton_options.dbpassword beginning at line 7 column 3 byte 180 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbport : "5432" from TruxtonSettings.xml XML element name truxton_options.dbport beginning at line 4 column 3 byte 98 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbport : "5432" from Default definitions
dbserver : "localhost" from TruxtonSettings.xml XML element name truxton_options.dbserver beginning at line 3 column 3 byte 64 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbserver : "127.0.0.1" from Default definitions
dbsr1 :
dbuser : "postgres" from TruxtonSettings.xml XML element name truxton_options.dbuser beginning at line 6 column 3 byte 151 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
dbuser : "postgres" from Default definitions
debugfileidmapping : "0" from Default definitions
debugoptions : "1" from Command Line argument 2 (zero is the first)
dedupemessages : "0" from Default definitions
deepsleep : "45007" from Default definitions
deletedfiles : "1" from Default definitions
depotpagesize : "4096" from Default definitions
ds : "0" from Default definitions
ee : "0" from Default definitions
excludefilesbytype : "0" from Default definitions
excludes : "0" from Default definitions
fct : "300" from Default definitions
filegroupshavebeeninitialized : "1" from TruxtonSettings.xml XML element name truxton_options.FileGroupsHaveBeenInitialized beginning at line 17 column 3 byte 593 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
fileslack : "1" from Default definitions
fqhi : "7500" from Default definitions
fqlo : "100" from Default definitions
hashsetall : "0" from Default definitions
iei : "1" from Default definitions
imagelength : "-1" from Default definitions
imageoffset : "0" from Default definitions
includefilesbytype : "0" from Default definitions
includes : "0" from Default definitions
istatus : "1" from Default definitions
itype : "0" from Default definitions
loadas : "0" from Default definitions
machineid : "64B30278-F250-4896-82EB-1AC416DB30B2" from TruxtonSettings.xml XML element name truxton_options.machineid beginning at line 14 column 3 byte 422 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
maxsleep : "600101" from Default definitions
metacarve : "1" from Default definitions
minsleep : "3011" from Default definitions
mlat : "0" from Default definitions
mlong : "0" from Default definitions
mqname : "TruxtonMessageBus" from TruxtonSettings.xml XML element name truxton_options.mqname beginning at line 10 column 3 byte 278 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mqname : "TruxtonMessageBus" from Default definitions
mqpassword : "Truxton4n6" from TruxtonSettings.xml XML element name truxton_options.mqpassword beginning at line 12 column 3 byte 345 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mqport : "5432" from TruxtonSettings.xml XML element name truxton_options.mqport beginning at line 9 column 3 byte 253 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mqport : "0" from Default definitions
mqserver : "localhost" from TruxtonSettings.xml XML element name truxton_options.mqserver beginning at line 8 column 3 byte 219 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mqserver : "127.0.0.1" from Default definitions
mquser : "postgres" from TruxtonSettings.xml XML element name truxton_options.mquser beginning at line 11 column 3 byte 316 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
mquser : "postgres" from Default definitions
mr : "0" from Default definitions
nocontents : "0" from Default definitions
nohashsets : "0" from Default definitions
norouting : "0" from Default definitions
nostartdelay : "0" from Default definitions
nostore : "0" from Default definitions
numberofbuffers : "2048" from Default definitions
numberofbytesperbuffer : "65536" from Default definitions
numberoffiles : "9223372036854775807" from Default definitions
pfiles : "1" from Default definitions
pfree : "1" from Default definitions
priority : "1000" from Default definitions
rapid : "0" from Default definitions
recovery : "0" from Default definitions
reprocess : "0" from Default definitions
skipdupmedia : "0" from Default definitions
smtp_server : "localhost" from Default definitions
solr_url : "http://localhost:8983/solr/truxton-core" from TruxtonSettings.xml XML element name truxton_options.solr_url beginning at line 16 column 3 byte 529 of "C:\ProgramData\Truxton\Settings\TruxtonSettings.xml"
startatmft : "0" from Default definitions
syslogport : "514" from Default definitions
timeout : "120" from Default definitions
triage : "0" from Default definitions
tsk_verbose : "0" from Default definitions
updatemedianumberofchildren : "1" from Default definitions
updatepathid : "1" from Default definitions
verbose : "0" from Default definitions
wait : "none" from Default definitions
writesettings :
xorcarve : "1" from Default definitions
End of Options, Start of Non-Options
"identify"
End of All Option Dump
2019-10-16T12:31:21Z 4454-6528 : application_name='Truxton Identify' dbname='Truxton' user='postgres' password='Truxton4n6' host='localhost' port=5432
2019-10-16T12:31:21Z 4454-6528 : application_name='Truxton Identify' dbname='TruxtonMessageBus' user='postgres' password='Truxton4n6' host='localhost' port=5432
2019-10-16T12:31:21Z 4454-6528 : Truxton Identify can't get connection string variable dbserver variable "logserver"
2019-10-16T12:31:21Z 4454-6528 : application_name='Truxton Identify' dbname='Truxton' user='postgres' password='Truxton4n6' host='localhost' port=5432
Random Notes
Lessons learned from the field...
ReFS Deduplication
It looks like Windows Storage Spaces has limitations on deduplication of volumes. It appears that the maximum volume size is 64TB while only the first 4TB of a file will be deduplicated.
