Difference between revisions of "Depot"

From truxwiki.com
Jump to navigation Jump to search
(Created page with "Truxton stores file contents in large files called depots. When you store a stream of bytes in a depot, the offset of the beginning of the stream in the depot and the depot le...")
 
 
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
Truxton stores file contents in large files called depots.
 
Truxton stores file contents in large files called depots.
−
When you store a stream of bytes in a depot, the offset of the beginning of the stream in the depot and the depot length are recorded in the database.
+
When you store a stream of bytes in a depot, the offset of the beginning of the stream in the depot and the stream length are recorded in the <code><nowiki>[</nowiki>[[Content Table|Content]]<nowiki>]</nowiki></code> table in the database.
  
−
=General Format=
+
=File Format=
 
The depot file format is very simple.
 
The depot file format is very simple.
 
The header of the file is 64KB.
 
The header of the file is 64KB.
Line 8: Line 8:
  
 
==Header==
 
==Header==
−
 
+
The depot file header is a 32-byte signature followed by [https://en.wikipedia.org/wiki/XML XML] padded with enough spaces to fill out the first 64KB of the file.
−
The depot file header is a 32-byte signature followed by XML padded with enough spaces to fill out the 64KB beginning region.
+
Data stored in a depot file begins at offset 65536.
 
The signature is:
 
The signature is:
  
−
<syntaxhighlight lang="text">
+
<source lang="text">
 
This is a Truxton Depot File!!<CR><LF>
 
This is a Truxton Depot File!!<CR><LF>
−
</syntaxhighlight>
+
</source>
  
 
===Sample===
 
===Sample===
−
<syntaxhighlight lang="xml">
+
<source lang="xml">
 
This is a Truxton Depot File!!
 
This is a Truxton Depot File!!
 
<?xml version="1.0" standalone="yes"?>
 
<?xml version="1.0" standalone="yes"?>
Line 38: Line 38:
 
  <customer>D0814FE2-7794-5E33-DDD4-B6BB588CB6D4</customer>
 
  <customer>D0814FE2-7794-5E33-DDD4-B6BB588CB6D4</customer>
 
</truxton>
 
</truxton>
−
</syntaxhighlight>
+
</source>
  
 
===XML Fields===
 
===XML Fields===
−
We store a lot of information in the XML to aid anyone who has the unenviable task of recovering a depot file.
+
We store a lot of information in the [https://en.wikipedia.org/wiki/XML XML] to aid anyone who has the unenviable task of recovering a depot file.
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 48: Line 48:
 
|-
 
|-
 
| <code>depot_id</code>
 
| <code>depot_id</code>
−
| The identifier for this depot file. This should correspond to the <code>ID</code> column of the <code>[[Depot Table|Depot]]</code> table in the database.
+
| The identifier for this depot file. This should correspond to the <code>[ID]</code> column of the <code><nowiki>[</nowiki>[[Depot Table|Depot]]<nowiki>]</nowiki></code> table in the database.
 
|-
 
|-
 
| <code>page_size</code>
 
| <code>page_size</code>
Line 69: Line 69:
 
|-
 
|-
 
| <code>process_id</code>
 
| <code>process_id</code>
−
| The id of the process that created this file.
+
| The integer identifier of the process that created this file.
 
|-
 
|-
 
| <code>bios_id</code>
 
| <code>bios_id</code>
Line 100: Line 100:
  
 
====bios_id====
 
====bios_id====
−
This [https://en.wikipedia.org/wiki/Universally_unique_identifier uuid] is retrieved from the [https://en.wikipedia.org/wiki/BIOS BIOS] using an [https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/nf-sysinfoapi-getsystemfirmwaretable API] call.
+
This [https://en.wikipedia.org/wiki/Universally_unique_identifier uuid] is retrieved from the [https://en.wikipedia.org/wiki/BIOS BIOS] using an [https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/nf-sysinfoapi-getsystemfirmwaretable Windows API] call.
 
Specifically, the [https://www.dmtf.org/sites/default/files/standards/documents/DSP0134_2.7.1.pdf 'RSMB'] firmware table is retrieved and the <code>UUID</code> field of the System Information (Type 1) structure is used.
 
Specifically, the [https://www.dmtf.org/sites/default/files/standards/documents/DSP0134_2.7.1.pdf 'RSMB'] firmware table is retrieved and the <code>UUID</code> field of the System Information (Type 1) structure is used.
  
 
It can also be retrieved via:
 
It can also be retrieved via:
−
<syntaxhighlight lang="bat">
+
<source lang="bat">
 
wmic path Win32_ComputerSystemProduct get UUID  
 
wmic path Win32_ComputerSystemProduct get UUID  
−
</syntaxhighlight>
+
</source>
  
 
====bios_serial_number====
 
====bios_serial_number====
Line 112: Line 112:
 
Specifically, the [https://www.dmtf.org/sites/default/files/standards/documents/DSP0134_2.7.1.pdf 'RSMB'] firmware table is retrieved and the <code>SerialNumber</code> field of the System Information (Type 1) structure is used.
 
Specifically, the [https://www.dmtf.org/sites/default/files/standards/documents/DSP0134_2.7.1.pdf 'RSMB'] firmware table is retrieved and the <code>SerialNumber</code> field of the System Information (Type 1) structure is used.
  
−
Dell will store their service tags in this field.
+
[https://www.dell.com Dell] will store their service tags in this field.
  
 
==Data Alignment==
 
==Data Alignment==
−
While data is normally aligned on a 4KB boundary, it can be modified by the <code>depotpagesize</code> setting which can be put into <code>TruxtonSettings.xml</code>
+
While data is normally aligned on a 4KB boundary, it can be modified by the <code>depotpagesize</code> setting which can be put into <code>[[Configuration System#Truxton Settings File|TruxtonSettings.xml]]</code>
  
 
A special case of alignment is free space depots.
 
A special case of alignment is free space depots.
 
They are aligned on a 512 byte boundary as this most closely matches hard drive sectors.
 
They are aligned on a 512 byte boundary as this most closely matches hard drive sectors.
−
If we were to align on 4KB or 8KB boundaries, carving would be nearly impossible.
+
If we were to align on 4KB or 8KB boundaries, [https://en.wikipedia.org/wiki/File_carving carving] would be nearly impossible.
 +
 
 +
When a file is added to a depot, it begins on the page boundary.
 +
Any space left after the end of a file but before the next page boundary is filled with zeroes.
 +
 
 +
=Depot Deletion=
 +
When Truxton deletes media, it will not delete the depot files by default.
 +
There are two settings that govern depot deletion:
 +
* <code>enabledeletedepots</code> - A boolean value that must be set to true to enable the deletion of depot files
 +
* <code>softdeletedepots</code> - Instead of deleting the depot file, rename it giving it an extension of <code>.ToBeDeleted</code>
 +
 
 +
=Retrieving a File=
 +
To retrieve a file from a depot, you must first know where the file's contents exist in the depot.
 +
When anything in Truxton needs a file's contents it will follow these steps:
 +
# Retrieve the file's [https://en.wikipedia.org/wiki/Universally_unique_identifier GUID]. This comes from the <code>[ID]</code> column of the <code><nowiki>[</nowiki>[[File Table|File]]<nowiki>]</nowiki></code> table.
 +
# Retrieve the [https://en.wikipedia.org/wiki/MD5 MD5]. This comes from the <code>[HashID]</code> column of the <code><nowiki>[</nowiki>[[File Table|File]]<nowiki>]</nowiki></code> table.
 +
# Retrieve the <code>[DepotID]</code>, <code>[Offset]</code> and <code>[Length]</code> values from the <code><nowiki>[</nowiki>[[Content Table|Content]]<nowiki>]</nowiki></code> table when the <code>[Hash]</code> column matches the MD5.
 +
# Retrieve the <code>[Filename]</code> and <code>[URI]</code> columns from the <code><nowiki>[</nowiki>[[Depot Table|Depot]]<nowiki>]</nowiki></code> table when the <code>[ID]</code> column matches the depot identifier.
 +
# Open the file, seek to the offset and read bytes for the given length. If the file cannot be opened, open the URI and read the byte range.
 +
 
 +
When using the <code>[[truxton_message_get_file]]</code> C API, it performs the steps above.
 +
The Python methods [[TruxtonObject#getfileid.28file_id:_str_.7C_UUID.29_-.3E_TruxtonFileIO|getfileid()]] and [[TruxtonMessage#file.28.29_-.3E_TruxtonFileIO|file()]] also do this for you.

Latest revision as of 07:21, 30 September 2024

Truxton stores file contents in large files called depots. When you store a stream of bytes in a depot, the offset of the beginning of the stream in the depot and the stream length are recorded in the [Content] table in the database.

File Format

The depot file format is very simple. The header of the file is 64KB. The remainder of the file is streams of bytes aligned on a page boundary, 4KB by default.

Header

The depot file header is a 32-byte signature followed by XML padded with enough spaces to fill out the first 64KB of the file. Data stored in a depot file begins at offset 65536. The signature is: 

This is a Truxton Depot File!!<CR><LF>

Sample

This is a Truxton Depot File!!
<?xml version="1.0" standalone="yes"?>
<truxton>
 <depot_id>5D72324E-A079-1A6A-0E80-D81600000010</depot_id>
 <page_size>4096</page_size>
 <createdby>C:\Program Files\Truxton\Loader\Expand.exe</createdby>
 <created>2019-09-06T10:17:50.744432Z</created>
 <ipaddress>172.18.92.241</ipaddress>
 <version>3.1.0.902</version>
 <command_line>&quot;Expand.exe&quot; -controlledby 9622956017505175439</command_line>
 <process_id>4244</process_id>
 <bios_id>4C4C4544-0046-5310-804D-B2C04F515132</bios_id>
 <machine_sid_hash>F7CD1C0A-F558-41EB-8994-8DC9F83F5896</machine_sid_hash>
 <bios_serial_number>2FSMCC2</bios_serial_number>
 <user_name>SYSTEM</user_name>
 <computer_name>DESKTOP-5NRI5PO</computer_name>
 <filename>5D72324E-A079-1A6A-0E80-D81600000010_expand.file.depot</filename>
 <type>1</type>
 <customer>D0814FE2-7794-5E33-DDD4-B6BB588CB6D4</customer>
</truxton>

XML Fields

We store a lot of information in the XML to aid anyone who has the unenviable task of recovering a depot file.

Name Meaning
depot_id The identifier for this depot file. This should correspond to the [ID] column of the [Depot] table in the database.
page_size The boundary on which entries in the depot will be aligned. It can be altered via the depotpagesize setting.
createdby The full path to the executable that created this file.
created The time the depot file was created.
ipaddress The TCP/IP v4 address of the machine where the executable that created this file was running.
version The version number of the process that created this file.
command_line The contents of the command line of the process that created this file.
process_id The integer identifier of the process that created this file.
bios_id The unique id retrieved from the BIOS firmware.
machine_sid_hash This is the MachineGuid value retrieved from the registry key HKLM/SOFTWARE/Microsoft/Cryptography
bios_serial_number The serial number of the machine running the process that created this file as reported by the BIOS.
user_name The name of the account the process was running under.
computer_name The name of the computer the process that created this file was running on.
filename The original name of the depot file.
type The type of depot file.
customer The identifier (license) of the customer that created this depot file.

BIOS Fields

Here's where Truxton gets the BIOS fields.

bios_id

This uuid is retrieved from the BIOS using an Windows API call. Specifically, the 'RSMB' firmware table is retrieved and the UUID field of the System Information (Type 1) structure is used.

It can also be retrieved via: 

wmic path Win32_ComputerSystemProduct get UUID

bios_serial_number

This string is retrieved from the BIOS using an API call. Specifically, the 'RSMB' firmware table is retrieved and the SerialNumber field of the System Information (Type 1) structure is used.

Dell will store their service tags in this field.

Data Alignment

While data is normally aligned on a 4KB boundary, it can be modified by the depotpagesize setting which can be put into TruxtonSettings.xml

A special case of alignment is free space depots. They are aligned on a 512 byte boundary as this most closely matches hard drive sectors. If we were to align on 4KB or 8KB boundaries, carving would be nearly impossible.

When a file is added to a depot, it begins on the page boundary. Any space left after the end of a file but before the next page boundary is filled with zeroes.

Depot Deletion

When Truxton deletes media, it will not delete the depot files by default. There are two settings that govern depot deletion:

  • enabledeletedepots - A boolean value that must be set to true to enable the deletion of depot files
  • softdeletedepots - Instead of deleting the depot file, rename it giving it an extension of .ToBeDeleted

Retrieving a File

To retrieve a file from a depot, you must first know where the file's contents exist in the depot. When anything in Truxton needs a file's contents it will follow these steps:

  1. Retrieve the file's GUID. This comes from the [ID] column of the [File] table.
  2. Retrieve the MD5. This comes from the [HashID] column of the [File] table.
  3. Retrieve the [DepotID], [Offset] and [Length] values from the [Content] table when the [Hash] column matches the MD5.
  4. Retrieve the [Filename] and [URI] columns from the [Depot] table when the [ID] column matches the depot identifier.
  5. Open the file, seek to the offset and read bytes for the given length. If the file cannot be opened, open the URI and read the byte range.

When using the truxton_message_get_file C API, it performs the steps above. The Python methods getfileid() and file() also do this for you.

Retrieved from "https://truxwiki.com/index.php?title=Depot&oldid=9860"