Difference between revisions of "Creating NSRL Hashset"
| (4 intermediate revisions by the same user not shown) | |||
| Line 32: | Line 32: | ||
# Download <code>RDS_2023.03.1_modern_minimal.zip</code> | # Download <code>RDS_2023.03.1_modern_minimal.zip</code> | ||
# Unzip it to produce <code>RDS_2023.03.1_modern_minimal.db</code> | # Unzip it to produce <code>RDS_2023.03.1_modern_minimal.db</code> | ||
| − | # From a command window, execute <source lang=" | + | # From a command window, execute <source lang="text"> |
sqlite3.exe | sqlite3.exe | ||
.open RDS_2023.03.1_modern_minimal.db | .open RDS_2023.03.1_modern_minimal.db | ||
| Line 39: | Line 39: | ||
.exit | .exit | ||
</source> | </source> | ||
| − | # <source lang="bat">"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt</source> | + | # <source lang="bat" inline>"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt</source> |
===Statistics=== | ===Statistics=== | ||
| Line 61: | Line 61: | ||
| style="text-align:right;" | 62,007,285 | | style="text-align:right;" | 62,007,285 | ||
|} | |} | ||
| + | |||
| + | =Unzipping The Download= | ||
| + | The NSRL is getting a little out of hand. | ||
| + | The March 2025 download was a 118GB zip file that expanded to 428GB. | ||
| + | When things get that big, Windows GUI tools choke on it. | ||
| + | For some unknown reason, other than crappy coding, GUI tools first unzip a file to a temporary folder then copy the file to the final destination. | ||
| + | Normally this is find for small files but 428GB becomes a real problem. | ||
| + | Fortunately, command line tools can decompress files directly to their final destination. | ||
| + | Here's how to use 7Zip to unzip this monstrous archive. | ||
| + | <source lang="bat"> | ||
| + | "C:\Program Files\7-Zip\7z.exe" x D:\NSRL\RDS_2025.03.1_modern.zip -oE:\NSRL | ||
| + | </source> | ||
=How to Create NSRL Delta Hash Set (v3)= | =How to Create NSRL Delta Hash Set (v3)= | ||
| Line 66: | Line 78: | ||
This is the additions to the NSRL since the previous release. | This is the additions to the NSRL since the previous release. | ||
It is distributed as ASCII SQL INSERT statements. | It is distributed as ASCII SQL INSERT statements. | ||
| − | <source lang=" | + | <source lang="text"> |
sqlite3.exe rds1 | sqlite3.exe rds1 | ||
.read RDS_2023.12.1_modern.schema.sql | .read RDS_2023.12.1_modern.schema.sql | ||
| Line 72: | Line 84: | ||
.output rds1.txt | .output rds1.txt | ||
select md5 from FILE; | select md5 from FILE; | ||
| + | .exit | ||
| + | </source> | ||
| + | |||
| + | ==OR== | ||
| + | The schema is a moving target.... | ||
| + | <source lang="text"> | ||
| + | sqlite3.exe rds1 | ||
| + | .read RDS_2023.12.1_modern.schema.sql | ||
| + | .read RDS_2023.12.1_modern_delta.sql | ||
| + | .output rds1.txt | ||
| + | select md5 from METADATA; | ||
.exit | .exit | ||
</source> | </source> | ||
Latest revision as of 03:37, 1 July 2025
The National Software Reference Library is a project from the National Institute of Standards and Technology. It consists of several file downloads of CSV files containing hashes of file contents. You can use this library to determine the source of a hash (which software package it came from) or, in the case of Truxton, eliminate files that will have no evidence in them.
Contents
How to Create NSRL Hash Set (v2)
Follow these steps:
- Download the data from NIST
- Unzip or mount the ISO and unzip the
NSRLFile.txt - Run the
Truxton\Tools\NSRLMinimalMD5.exeprogram to create a hash set file"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" NSRLFile.txt - Repeat the above steps until you have produced all of the hash set files from the NSRL downloads.
- Merge the hashset files into a single hashset file
"C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes rds1.txt.hashset rds2.txt.hashset rds3.txt.hashset rds4.txt.hashset rds5.txt.hashset rds6.txt.hashset - Rename the merged hashset into what you want
ren merged.hashset LatestNSRL.hashset - Replace the existing hash set file for the ETL layer with this new one.
How to Create NSRL Hash Set (v3)
Follow these steps:
- Download the data from NIST
- Unzip the database
- Download sqlite tools
- Run the
Truxton\Tools\NSRLMinimalMD5.exeprogram to create a hash set file"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt - Repeat the above steps until you have produced all of the hash set files from the NSRL downloads.
- Merge the hashset files into a single hashset file
"C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes rds1.txt.hashset rds2.txt.hashset rds3.txt.hashset rds4.txt.hashset rds5.txt.hashset rds6.txt.hashset - Rename the merged hashset into what you want
ren merged.hashset LatestNSRL.hashset - Replace the existing hash set file for the ETL layer with this new one.
Example
Here's how to process one of the downloads. WARNING! You will need a LOT of disk space.
- Download
RDS_2023.03.1_modern_minimal.zip - Unzip it to produce
RDS_2023.03.1_modern_minimal.db - From a command window, execute
sqlite3.exe .open RDS_2023.03.1_modern_minimal.db .output rds1.txt select md5 from FILE; .exit
"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt
Statistics
| Size of the RDS zip file | 13.5GB |
| Size of the decompressed SQLite database | 113GB |
| Size of the extracted MD5 hashes | 9GB |
| Size of the resulting hash set | 946MB |
| Number of extracted hashes | 293,889,314 |
| Number of unique hashes in hash set | 62,007,285 |
Unzipping The Download
The NSRL is getting a little out of hand. The March 2025 download was a 118GB zip file that expanded to 428GB. When things get that big, Windows GUI tools choke on it. For some unknown reason, other than crappy coding, GUI tools first unzip a file to a temporary folder then copy the file to the final destination. Normally this is find for small files but 428GB becomes a real problem. Fortunately, command line tools can decompress files directly to their final destination. Here's how to use 7Zip to unzip this monstrous archive.
"C:\Program Files\7-Zip\7z.exe" x D:\NSRL\RDS_2025.03.1_modern.zip -oE:\NSRL
How to Create NSRL Delta Hash Set (v3)
Yet another way NIST distributes the NSRL is called the "Delta" dumps. This is the additions to the NSRL since the previous release. It is distributed as ASCII SQL INSERT statements.
sqlite3.exe rds1
.read RDS_2023.12.1_modern.schema.sql
.read RDS_2023.12.1_modern_delta.sql
.output rds1.txt
select md5 from FILE;
.exit
OR
The schema is a moving target....
sqlite3.exe rds1
.read RDS_2023.12.1_modern.schema.sql
.read RDS_2023.12.1_modern_delta.sql
.output rds1.txt
select md5 from METADATA;
.exit
