Difference between revisions of "Creating NSRL Hashset"

From truxwiki.com
Jump to navigation Jump to search
Line 21: Line 21:
 
# Unzip the database
 
# Unzip the database
 
# Download [https://sqlite.org/download.html sqlite tools]
 
# Download [https://sqlite.org/download.html sqlite tools]
 +
# Run the <code>Truxton\Tools\NSRLMinimalMD5.exe</code> program to create a [[Hash Set|hash set]] file <source lang="bat">"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt</source>
 +
# Rename the output file <source lang="bat">ren Out.hashset 1.hashset</source>
 +
# Repeat the above steps until you have produced all of the [[Hash Set|hash set]] files from the NSRL downloads.
 +
# Merge the hashset files into a single hashset file <source lang="bat">"C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes 1.hashset 2.hashset 3.hashset 4.hashset 5.hashset 6.hashset</source>
 +
# Rename the merged hashset into what you want <source lang="bat">ren merged.hashset LatestNSRL.hashset</source>
 +
# Replace the existing [[Hash Set|hash set]] file for the ETL layer with this new one.
  
 
==Example==
 
==Example==
Line 26: Line 32:
 
WARNING! You will need a LOT of disk space.
 
WARNING! You will need a LOT of disk space.
  
−
# Download <code>RDS_2023.03.1_modern_minimal.zip</code> (13.5GB)
+
# Download <code>RDS_2023.03.1_modern_minimal.zip</code>
−
# Unzip it to produce <code>RDS_2023.03.1_modern_minimal.db</code> (113GB)
+
# Unzip it to produce <code>RDS_2023.03.1_modern_minimal.db</code>
 
# From a command window, execute <source lang="txt">
 
# From a command window, execute <source lang="txt">
 
sqlite3.exe
 
sqlite3.exe
Line 35: Line 41:
 
.exit
 
.exit
 
</source>
 
</source>
 +
# <source lang="bat">"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt</source>
 +
# <source lang="bat">ren Out.hashset 1.hashset</source>
 +
 +
===Statistics===
 +
Size of the RDS zip file: 13.5GB<br />
 +
Size of the decompressed SQLite database: 113GB<br />
 +
Size of the extracted MD5 hashes: 9GB<br />
 +
Size of the resulting hash set: 946MB<br />
 +
Number of extracted hashes: 293,889,314<br />
 +
Number of unique hashes in hash set: 62,007,285

Revision as of 05:36, 12 May 2023

The National Software Reference Library is a project from the National Institute of Standards and Technology. It consists of several file downloads of CSV files containing hashes of file contents. You can use this library to determine the source of a hash (which software package it came from) or, in the case of Truxton, eliminate files that will have no evidence in them.

How to Create NSRL Hash Set (v2)

Follow these steps:

  1. Download the data from NIST
  2. Unzip or mount the ISO and unzip the NSRLFile.txt
  3. Run the Truxton\Tools\NSRLMinimalMD5.exe program to create a hash set file 
    "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" NSRLFile.txt
  4. Rename the output file 
    ren Out.hashset 1.hashset
  5. Repeat the above steps until you have produced all of the hash set files from the NSRL downloads.
  6. Merge the hashset files into a single hashset file 
    "C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes 1.hashset 2.hashset 3.hashset 4.hashset 5.hashset 6.hashset
  7. Rename the merged hashset into what you want 
    ren merged.hashset LatestNSRL.hashset
  8. Replace the existing hash set file for the ETL layer with this new one.

How to Create NSRL Hash Set (v3)

Follow these steps:

  1. Download the data from NIST
  2. Unzip the database
  3. Download sqlite tools
  4. Run the Truxton\Tools\NSRLMinimalMD5.exe program to create a hash set file 
    "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt
  5. Rename the output file 
    ren Out.hashset 1.hashset
  6. Repeat the above steps until you have produced all of the hash set files from the NSRL downloads.
  7. Merge the hashset files into a single hashset file 
    "C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes 1.hashset 2.hashset 3.hashset 4.hashset 5.hashset 6.hashset
  8. Rename the merged hashset into what you want 
    ren merged.hashset LatestNSRL.hashset
  9. Replace the existing hash set file for the ETL layer with this new one.

Example

Here's how to process one of the downloads. WARNING! You will need a LOT of disk space.

  1. Download RDS_2023.03.1_modern_minimal.zip
  2. Unzip it to produce RDS_2023.03.1_modern_minimal.db
  3. From a command window, execute 
    sqlite3.exe
.open RDS_2023.03.1_modern_minimal.db
.output rds1.txt
select md5 from FILE;
.exit
  4. "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt

  5. ren Out.hashset 1.hashset

Statistics

Size of the RDS zip file: 13.5GB
Size of the decompressed SQLite database: 113GB
Size of the extracted MD5 hashes: 9GB
Size of the resulting hash set: 946MB
Number of extracted hashes: 293,889,314
Number of unique hashes in hash set: 62,007,285

Retrieved from "https://truxwiki.com/index.php?title=Creating_NSRL_Hashset&oldid=6308"