Creating NSRL Hashset

The National Software Reference Library is a project from the National Institute of Standards and Technology. It consists of several file downloads of CSV files containing hashes of file contents. You can use this library to determine the source of a hash (which software package it came from) or, in the case of Truxton, eliminate files that will have no evidence in them.

How to Create NSRL Hash Set (v2)

Follow these steps:

1. Download the data from NIST
2. Unzip or mount the ISO and unzip the NSRLFile.txt
3. Run the Truxton\Tools\NSRLMinimalMD5.exe program to create a hash set file

   "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" NSRLFile.txt
   

4. Repeat the above steps until you have produced all of the hash set files from the NSRL downloads.
5. Merge the hashset files into a single hashset file

   "C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes rds1.txt.hashset rds2.txt.hashset rds3.txt.hashset rds4.txt.hashset rds5.txt.hashset rds6.txt.hashset
   

6. Rename the merged hashset into what you want

   ren merged.hashset LatestNSRL.hashset
   

7. Replace the existing hash set file for the ETL layer with this new one.

How to Create NSRL Hash Set (v3)

Follow these steps:

1. Download the data from NIST
2. Unzip the database
3. Download sqlite tools
4. Run the Truxton\Tools\NSRLMinimalMD5.exe program to create a hash set file

   "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt
   

5. Repeat the above steps until you have produced all of the hash set files from the NSRL downloads.
6. Merge the hashset files into a single hashset file

   "C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes rds1.txt.hashset rds2.txt.hashset rds3.txt.hashset rds4.txt.hashset rds5.txt.hashset rds6.txt.hashset
   

7. Rename the merged hashset into what you want

   ren merged.hashset LatestNSRL.hashset
   

8. Replace the existing hash set file for the ETL layer with this new one.

Example

Here's how to process one of the downloads. WARNING! You will need a LOT of disk space.

1. Download RDS_2023.03.1_modern_minimal.zip
2. Unzip it to produce RDS_2023.03.1_modern_minimal.db
3. From a command window, execute

   sqlite3.exe
   .open RDS_2023.03.1_modern_minimal.db
   .output rds1.txt
   select md5 from FILE;
   .exit

4. "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt
   

Statistics

Size of the RDS zip file	13.5GB
Size of the decompressed SQLite database	113GB
Size of the extracted MD5 hashes	9GB
Size of the resulting hash set	946MB
Number of extracted hashes	293,889,314
Number of unique hashes in hash set	62,007,285

How to Create NSRL Delta Hash Set (v3)

Yet another way NIST distributes the NSRL is called the "Delta" dumps. This is the additions to the NSRL since the previous release. It is distributed as ASCII SQL INSERT statements.

sqlite3.exe rds1
.read RDS_2023.12.1_modern.schema.sql
.read RDS_2023.12.1_modern_delta.sql
.output rds1.txt
select md5 from FILE;
.exit