Creating NSRL Hashset - Revision history

Sam: /* Unzipping The Download */

2025-07-01T07:37:11Z

Unzipping The Download

Sam: /* How to Create NSRL Delta Hash Set (v3) */

2025-06-28T11:39:52Z

How to Create NSRL Delta Hash Set (v3)

Sam at 10:04, 28 June 2025

2025-06-28T10:04:42Z

Sam at 19:26, 3 April 2025

2025-04-03T19:26:19Z

Sam at 19:18, 3 April 2025

2025-04-03T19:18:56Z

Sam at 10:21, 6 February 2024

2024-02-06T10:21:14Z

Sam at 09:36, 6 February 2024

2024-02-06T09:36:38Z

Sam at 09:23, 6 February 2024

2024-02-06T09:23:19Z

Sam: /* Statistics */

2023-06-08T11:08:31Z

Statistics

Sam at 09:36, 12 May 2023

2023-05-12T09:36:39Z

@@ Line 71: / Line 71: @@
 Here's how to use 7Zip to unzip this monstrous archive.
 <source lang="bat">
-z.exe x D:\NSRL\RDS_2025.03.1_modern.zip -oE:\NSRL
+"C:\Program Files\7-Zip\7z.exe" x D:\NSRL\RDS_2025.03.1_modern.zip -oE:\NSRL
 </source>

@@ Line 84: / Line 84: @@
 .output rds1.txt
 select md5 from FILE;
 .exit
 </source>

@@ Line 61: / Line 61: @@
 | style="text-align:right;" | 62,007,285
 |}
 =How to Create NSRL Delta Hash Set (v3)=

@@ Line 32: / Line 32: @@
 # Download <code>RDS_2023.03.1_modern_minimal.zip</code>
 # Unzip it to produce <code>RDS_2023.03.1_modern_minimal.db</code>
-# From a command window, execute <source lang="txt">
+# From a command window, execute <source lang="text">
 sqlite3.exe
 .open RDS_2023.03.1_modern_minimal.db
@@ Line 39: / Line 39: @@
 .exit
 </source>
-# <source lang="bat">"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt</source>
+# <source lang="bat" inline>"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt</source>
 ===Statistics===

@@ Line 66: / Line 66: @@
 This is the additions to the NSRL since the previous release.
 It is distributed as ASCII SQL INSERT statements.
-<source lang="txt">
+<source lang="text">
 sqlite3.exe rds1
 .read RDS_2023.12.1_modern.schema.sql

@@ Line 9: / Line 9: @@
 # Unzip or mount the ISO and unzip the <code>NSRLFile.txt</code>
 # Run the <code>Truxton\Tools\NSRLMinimalMD5.exe</code> program to create a [[Hash Set|hash set]] file <source lang="bat">"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" NSRLFile.txt</source>
 # Repeat the above steps until you have produced all of the [[Hash Set|hash set]] files from the NSRL downloads.
-# Merge the hashset files into a single hashset file <source lang="bat">"C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes 1.hashset 2.hashset 3.hashset 4.hashset 5.hashset 6.hashset</source>
+# Merge the hashset files into a single hashset file <source lang="bat">"C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes rds1.txt.hashset rds2.txt.hashset rds3.txt.hashset rds4.txt.hashset rds5.txt.hashset rds6.txt.hashset</source>
 # Rename the merged hashset into what you want <source lang="bat">ren merged.hashset LatestNSRL.hashset</source>
 # Replace the existing [[Hash Set|hash set]] file for the ETL layer with this new one.
@@ Line 22: / Line 21: @@
 # Download [https://sqlite.org/download.html sqlite tools]
 # Run the <code>Truxton\Tools\NSRLMinimalMD5.exe</code> program to create a [[Hash Set|hash set]] file <source lang="bat">"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt</source>
 # Repeat the above steps until you have produced all of the [[Hash Set|hash set]] files from the NSRL downloads.
-# Merge the hashset files into a single hashset file <source lang="bat">"C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes 1.hashset 2.hashset 3.hashset 4.hashset 5.hashset 6.hashset</source>
+# Merge the hashset files into a single hashset file <source lang="bat">"C:\Program Files\Truxton\Loader\Load.exe" -mergehashsets Yes rds1.txt.hashset rds2.txt.hashset rds3.txt.hashset rds4.txt.hashset rds5.txt.hashset rds6.txt.hashset</source>
 # Rename the merged hashset into what you want <source lang="bat">ren merged.hashset LatestNSRL.hashset</source>
 # Replace the existing [[Hash Set|hash set]] file for the ETL layer with this new one.
@@ Line 42: / Line 40: @@
 </source>
 # <source lang="bat">"C:\Program Files\Truxton\Tools\NSRLMinimalMD5.exe" rds1.txt</source>
 ===Statistics===

← Older revision		Revision as of 09:36, 6 February 2024
Line 69:		Line 69:
	This is the additions to the NSRL since the previous release.		This is the additions to the NSRL since the previous release.
	It is distributed as ASCII SQL INSERT statements.		It is distributed as ASCII SQL INSERT statements.
		+	<source lang="txt">
		+	sqlite3.exe rds1
		+	.read RDS_2023.12.1_modern.schema.sql
		+	.read RDS_2023.12.1_modern_delta.sql
		+	.output rds1.txt
		+	select md5 from FILE;
		+	.exit
		+	</source>

← Older revision		Revision as of 09:23, 6 February 2024
Line 64:		Line 64:
	\| style="text-align:right;" \| 62,007,285		\| style="text-align:right;" \| 62,007,285
	\|}		\|}
		+
		+	=How to Create NSRL Delta Hash Set (v3)=
		+	Yet another way NIST distributes the NSRL is called the "Delta" dumps.
		+	This is the additions to the NSRL since the previous release.
		+	It is distributed as ASCII SQL INSERT statements.

@@ Line 45: / Line 45: @@
 ===Statistics===
-Size of the RDS zip file: 13.5GB<br />
+{| class="wikitable"
-Size of the decompressed SQLite database: 113GB<br />
+| Size of the RDS zip file
-Size of the extracted MD5 hashes: 9GB<br />
+| style="text-align:right;" | 13.5GB
-Size of the resulting hash set: 946MB<br />
+|-
-Number of extracted hashes: 293,889,314<br />
+| Size of the decompressed SQLite database
-Number of unique hashes in hash set: 62,007,285
+| style="text-align:right;" | 113GB

@@ Line 21: / Line 21: @@
 # Unzip the database
 # Download [https://sqlite.org/download.html sqlite tools]
 ==Example==
@@ Line 26: / Line 32: @@
 WARNING! You will need a LOT of disk space.
-# Download <code>RDS_2023.03.1_modern_minimal.zip</code> (13.5GB)
+# Download <code>RDS_2023.03.1_modern_minimal.zip</code>
-# Unzip it to produce <code>RDS_2023.03.1_modern_minimal.db</code> (113GB)
+# Unzip it to produce <code>RDS_2023.03.1_modern_minimal.db</code>
 # From a command window, execute <source lang="txt">
 sqlite3.exe
@@ Line 35: / Line 41: @@
 .exit
 </source>