Difference between revisions of "Creating NSRL Hashset"

From truxwiki.com
Jump to navigation Jump to search
 
(11 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
You can use this library to determine the source of a hash (which software package it came from) or, in the case of Truxton, eliminate files that will have no evidence in them.
 
You can use this library to determine the source of a hash (which software package it came from) or, in the case of Truxton, eliminate files that will have no evidence in them.
  
−
=How to Create NSRL Hashset=
+
=How to Create NSRL Hash Set (v2)=
 
Follow these steps:
 
Follow these steps:
  
 
# [https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl/nsrl-download/current-rds Download] the data from NIST
 
# [https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl/nsrl-download/current-rds Download] the data from NIST
 
# Unzip or mount the ISO and unzip the <code>NSRLFile.txt</code>
 
# Unzip or mount the ISO and unzip the <code>NSRLFile.txt</code>
−
# Run the <code>Truxton\Tools\NSRLMinimalMD5.exe</code> program to create a hashset file <source lang="bat">"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" NSRLFile.txt</source>
+
# Run the <code>Truxton\Tools\NSRLMinimalMD5.exe</code> program to create a [[Hash Set|hash set]] file <source lang="bat">"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" NSRLFile.txt</source>
−
# Rename the output file <source lang="bat">ren Out.hashset 1.hashset</source>
+
# Repeat the above steps until you have produced all of the [[Hash Set|hash set]] files from the NSRL downloads.
−
# Repeat the above steps until you have produced all of the hashset files from the NSRL downloads.
+
# Merge the hashset files into a single hashset file <source lang="bat">"C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes rds1.txt.hashset rds2.txt.hashset rds3.txt.hashset rds4.txt.hashset rds5.txt.hashset rds6.txt.hashset</source>
−
# Merge the hashset files into a single hashset file <source lang="bat">"C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes 1.hashset 2.hashset 3.hashset 4.hashset 5.hashset 6.hashset</source>
 
 
# Rename the merged hashset into what you want <source lang="bat">ren merged.hashset LatestNSRL.hashset</source>
 
# Rename the merged hashset into what you want <source lang="bat">ren merged.hashset LatestNSRL.hashset</source>
−
# Replace the existing hashset file for the ETL layer with this new one.
+
# Replace the existing [[Hash Set|hash set]] file for the ETL layer with this new one.
 +
 
 +
=How to Create NSRL Hash Set (v3)=
 +
Follow these steps:
 +
 
 +
# [https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl/nsrl-download/current-rds Download] the data from NIST
 +
# Unzip the database
 +
# Download [https://sqlite.org/download.html sqlite tools]
 +
# Run the <code>Truxton\Tools\NSRLMinimalMD5.exe</code> program to create a [[Hash Set|hash set]] file <source lang="bat">"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt</source>
 +
# Repeat the above steps until you have produced all of the [[Hash Set|hash set]] files from the NSRL downloads.
 +
# Merge the hashset files into a single hashset file <source lang="bat">"C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes rds1.txt.hashset rds2.txt.hashset rds3.txt.hashset rds4.txt.hashset rds5.txt.hashset rds6.txt.hashset</source>
 +
# Rename the merged hashset into what you want <source lang="bat">ren merged.hashset LatestNSRL.hashset</source>
 +
# Replace the existing [[Hash Set|hash set]] file for the ETL layer with this new one.
 +
 
 +
==Example==
 +
Here's how to process one of the downloads.
 +
WARNING! You will need a LOT of disk space.
 +
 
 +
# Download <code>RDS_2023.03.1_modern_minimal.zip</code>
 +
# Unzip it to produce <code>RDS_2023.03.1_modern_minimal.db</code>
 +
# From a command window, execute <source lang="text">
 +
sqlite3.exe
 +
.open RDS_2023.03.1_modern_minimal.db
 +
.output rds1.txt
 +
select md5 from FILE;
 +
.exit
 +
</source>
 +
# <source lang="bat" inline>"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt</source>
 +
 
 +
===Statistics===
 +
{| class="wikitable"
 +
| Size of the RDS zip file
 +
| style="text-align:right;" | 13.5GB
 +
|-
 +
| Size of the decompressed SQLite database
 +
| style="text-align:right;" | 113GB
 +
|-
 +
| Size of the extracted MD5 hashes
 +
| style="text-align:right;" | 9GB
 +
|-
 +
| Size of the resulting hash set
 +
| style="text-align:right;" | 946MB
 +
|-
 +
| Number of extracted hashes
 +
| style="text-align:right;" | 293,889,314
 +
|-
 +
| Number of unique hashes in hash set
 +
| style="text-align:right;" | 62,007,285
 +
|}
 +
 
 +
=Unzipping The Download=
 +
The NSRL is getting a little out of hand.
 +
The March 2025 download was a 118GB zip file that expanded to 428GB.
 +
When things get that big, Windows GUI tools choke on it.
 +
For some unknown reason, other than crappy coding, GUI tools first unzip a file to a temporary folder then copy the file to the final destination.
 +
Normally this is find for small files but 428GB becomes a real problem.
 +
Fortunately, command line tools can decompress files directly to their final destination.
 +
Here's how to use 7Zip to unzip this monstrous archive.
 +
<source lang="bat">
 +
"C:\Program Files\7-Zip\7z.exe" x D:\NSRL\RDS_2025.03.1_modern.zip -oE:\NSRL
 +
</source>
 +
 
 +
=How to Create NSRL Delta Hash Set (v3)=
 +
Yet another way NIST distributes the NSRL is called the "Delta" dumps.
 +
This is the additions to the NSRL since the previous release.
 +
It is distributed as ASCII SQL INSERT statements.
 +
<source lang="text">
 +
sqlite3.exe rds1
 +
.read RDS_2023.12.1_modern.schema.sql
 +
.read RDS_2023.12.1_modern_delta.sql
 +
.output rds1.txt
 +
select md5 from FILE;
 +
.exit
 +
</source>
 +
 
 +
==OR==
 +
The schema is a moving target....
 +
<source lang="text">
 +
sqlite3.exe rds1
 +
.read RDS_2023.12.1_modern.schema.sql
 +
.read RDS_2023.12.1_modern_delta.sql
 +
.output rds1.txt
 +
select md5 from METADATA;
 +
.exit
 +
</source>

Latest revision as of 03:37, 1 July 2025

The National Software Reference Library is a project from the National Institute of Standards and Technology. It consists of several file downloads of CSV files containing hashes of file contents. You can use this library to determine the source of a hash (which software package it came from) or, in the case of Truxton, eliminate files that will have no evidence in them.

How to Create NSRL Hash Set (v2)

Follow these steps:

  1. Download the data from NIST
  2. Unzip or mount the ISO and unzip the NSRLFile.txt
  3. Run the Truxton\Tools\NSRLMinimalMD5.exe program to create a hash set file 
    "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" NSRLFile.txt
  4. Repeat the above steps until you have produced all of the hash set files from the NSRL downloads.
  5. Merge the hashset files into a single hashset file 
    "C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes rds1.txt.hashset rds2.txt.hashset rds3.txt.hashset rds4.txt.hashset rds5.txt.hashset rds6.txt.hashset
  6. Rename the merged hashset into what you want 
    ren merged.hashset LatestNSRL.hashset
  7. Replace the existing hash set file for the ETL layer with this new one.

How to Create NSRL Hash Set (v3)

Follow these steps:

  1. Download the data from NIST
  2. Unzip the database
  3. Download sqlite tools
  4. Run the Truxton\Tools\NSRLMinimalMD5.exe program to create a hash set file 
    "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt
  5. Repeat the above steps until you have produced all of the hash set files from the NSRL downloads.
  6. Merge the hashset files into a single hashset file 
    "C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes rds1.txt.hashset rds2.txt.hashset rds3.txt.hashset rds4.txt.hashset rds5.txt.hashset rds6.txt.hashset
  7. Rename the merged hashset into what you want 
    ren merged.hashset LatestNSRL.hashset
  8. Replace the existing hash set file for the ETL layer with this new one.

Example

Here's how to process one of the downloads. WARNING! You will need a LOT of disk space.

  1. Download RDS_2023.03.1_modern_minimal.zip
  2. Unzip it to produce RDS_2023.03.1_modern_minimal.db
  3. From a command window, execute 
    sqlite3.exe
.open RDS_2023.03.1_modern_minimal.db
.output rds1.txt
select md5 from FILE;
.exit
  4. "C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt

Statistics

Size of the RDS zip file 13.5GB
Size of the decompressed SQLite database 113GB
Size of the extracted MD5 hashes 9GB
Size of the resulting hash set 946MB
Number of extracted hashes 293,889,314
Number of unique hashes in hash set 62,007,285

Unzipping The Download

The NSRL is getting a little out of hand. The March 2025 download was a 118GB zip file that expanded to 428GB. When things get that big, Windows GUI tools choke on it. For some unknown reason, other than crappy coding, GUI tools first unzip a file to a temporary folder then copy the file to the final destination. Normally this is find for small files but 428GB becomes a real problem. Fortunately, command line tools can decompress files directly to their final destination. Here's how to use 7Zip to unzip this monstrous archive. 

"C:\Program Files\7-Zip\7z.exe" x D:\NSRL\RDS_2025.03.1_modern.zip -oE:\NSRL

How to Create NSRL Delta Hash Set (v3)

Yet another way NIST distributes the NSRL is called the "Delta" dumps. This is the additions to the NSRL since the previous release. It is distributed as ASCII SQL INSERT statements. 

sqlite3.exe rds1
.read RDS_2023.12.1_modern.schema.sql
.read RDS_2023.12.1_modern_delta.sql
.output rds1.txt
select md5 from FILE;
.exit

OR

The schema is a moving target.... 

sqlite3.exe rds1
.read RDS_2023.12.1_modern.schema.sql
.read RDS_2023.12.1_modern_delta.sql
.output rds1.txt
select md5 from METADATA;
.exit
Retrieved from "https://truxwiki.com/index.php?title=Creating_NSRL_Hashset&oldid=10328"