Creating NSRL Hashset

The National Software Reference Library is a project from the National Institute of Standards and Technology. It consists of several file downloads of CSV files containing hashes of file contents. You can use this library to determine the source of a hash (which software package it came from) or, in the case of Truxton, eliminate files that will have no evidence in them.

How to Create NSRL Hash Set (v2)

Follow these steps:

1. Download the data from NIST
2. Unzip or mount the ISO and unzip the NSRLFile.txt
3. Run the Truxton\Tools\NSRLMinimalMD5.exe program to create a hash set file

   "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" NSRLFile.txt
   

4. Rename the output file

   ren Out.hashset 1.hashset
   

5. Repeat the above steps until you have produced all of the hash set files from the NSRL downloads.
6. Merge the hashset files into a single hashset file

   "C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes 1.hashset 2.hashset 3.hashset 4.hashset 5.hashset 6.hashset
   

7. Rename the merged hashset into what you want

   ren merged.hashset LatestNSRL.hashset
   

8. Replace the existing hash set file for the ETL layer with this new one.

How to Create NSRL Hash Set (v3)

Follow these steps:

1. Download the data from NIST
2. Unzip the database
3. Download sqlite tools
4. Run the Truxton\Tools\NSRLMinimalMD5.exe program to create a hash set file

   "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt
   

5. Rename the output file

   ren Out.hashset 1.hashset
   

6. Repeat the above steps until you have produced all of the hash set files from the NSRL downloads.
7. Merge the hashset files into a single hashset file

   "C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes 1.hashset 2.hashset 3.hashset 4.hashset 5.hashset 6.hashset
   

8. Rename the merged hashset into what you want

   ren merged.hashset LatestNSRL.hashset
   

9. Replace the existing hash set file for the ETL layer with this new one.

Example

Here's how to process one of the downloads. WARNING! You will need a LOT of disk space.

1. Download RDS_2023.03.1_modern_minimal.zip
2. Unzip it to produce RDS_2023.03.1_modern_minimal.db
3. From a command window, execute

   sqlite3.exe
   .open RDS_2023.03.1_modern_minimal.db
   .output rds1.txt
   select md5 from FILE;
   .exit

4. "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt
   

5. ren Out.hashset 1.hashset
   

Statistics

Size of the RDS zip file: 13.5GB
Size of the decompressed SQLite database: 113GB
Size of the extracted MD5 hashes: 9GB
Size of the resulting hash set: 946MB
Number of extracted hashes: 293,889,314
Number of unique hashes in hash set: 62,007,285