Visit the NIST Main Home Page

National Software Reference Library Logo

HOME

GENERAL INFORMATION

TECHNICAL INFORMATION

DOWNLOADS

CFTT Website

Privacy Policy/Security Notice
Disclaimer | FOIA

NIST is an agency of the
U.S. Commerce Department

Date created: 1/20/2010
Last updated: 1/25/2010

Technical comments: nsrl@nist.gov

Website comments: web897@nist.gov

 

 

 

Non-RDS Algorithms

On this page, we will make links available to data sets that use hashing
or digest algorithms not contained in the RDS release.

If there is an algorithm or process that you think would be interesting to run on the 40,000,000 files
found in the RDS (or on a subset of files), please contact nsrl@nist.gov .


Data sets using the ssdeep algorithm (aka "fuzzy hashes") are found here.


Data sets using the sdhash algorithm (aka "similarity digest") are found here.


Data sets using the bulk_extractor tool are found here.


A text file which relates SHA-1 to SHA-256 hashes is available.
A 1.1 GB Zip file can be downloaded which contains a 1.9 GB text file with 16,801,737 rows.
Each row has a SHA-1 hash, a tab character, a SHA-256 hash, a tab and a filename.
The SHA-1 values can be matched to SHA-1 values in RDS 2.41. Note: not all SHA-1 values in RDS 2.41 can be matched to SHA-256 values.

Example rows:

00000C818DD9203A38348D260CDB6D7D1A84326A	B3CAFC6AE323C837BFAB5C3B0E9B1C8FC3602581623C7AC4396DE740858608C2	listing53.html
00000E294B89F381C95890A54417CFC6296C15A5	7336B50E69C456EBA58B28912E49E3EFB9025F6393D501796EE1C37B4EDB961C	8c88fe9d-2d33-42af-9d4c-ed00a6e60901.jpg
00000FF9D0ED9A6B53BC6A9364C07074DE1565F3	19E88A5365D6A54E92446A9A12DB16510FF37376FCB25E034C505373B2015EE8	cmnres.pdb.dll

File signatures:
SHA1(rds241-sha256.zip)= 3aa5dc0230444545a817cb9368bbfc28f4105822
MD5(rds241-sha256.zip)= 87542ab5610dc673f4406d0f1c333c9e

SHA1(rds241-sha256.txt)= 1580ecea496a34042f5b9db56e5f02307ffa68b9
MD5(rds241-sha256.txt)= 83234d8f550dd32f8b3706e47d5dd3d9

Note: SHA-256 test data is available.


NSRL has hashes of blocks of files.

A text file is available which relates the MD5 of a complete file to the MD5 of the first 4096 bytes in the file (provided the file is that large).
A 426 MB Zip file can be downloaded which contains a 823 MB text file with 13,112,687 rows.
Each row has an MD5 hash, a tab character, and an MD5 hash.

Example rows:

FFFFF06ED6CEB23B016780FD24DFE620        4CF962C2E3815626FE28DE7B2918D8CF
FFFFF19D706C0A754EBB553FEB03C724        D0CDC4D740BDF851D61E049D1DCFA139
FFFFF517FFD3E7425F3F15498595DA89        F5DCF00B42DFE803BD8114798FD67033

File signatures :
SHA1(md5b4096.zip)= dc70683f911d6a6254f37f09418faf0636267f3d
MD5(md5b4096.zip)= c458e36feef54c22226e2ae445e443d0

SHA1(md5b4096.txt)= 57445b7f4dcff3ad62fd05d170bfcc1e80b38b7f
MD5(md5b4096.txt)= ead485222f9f4f7f4a08b050b85d63ef


NSRL has hashes of blocks of files.

A text file is available which relates the MD5 of a complete file to the MD5 of the first 8192 bytes in the file (provided the file is that large).
A 358 MB Zip file can be downloaded which contains a 638 MB text file with 10,164,753 rows.
Each row has an MD5 hash, a tab character, and an MD5 hash.

Example rows:

FFFFF06ED6CEB23B016780FD24DFE620        41805D5E6AEC565158DEEBB5AD8762B5
FFFFF19D706C0A754EBB553FEB03C724        AA8A7BFF6361797B1A06B05F66059644
FFFFF517FFD3E7425F3F15498595DA89        4B8F27CE545C766DF6E62309EF2A4A9F

File signatures :
SHA1(md5b8192.zip)= 5a6ffcf73f1303caf9b49e35867cca913c173d98
MD5(md5b8192.zip)= 1f810bfa6103ef96c436a749a3d3bc7a

SHA1(md5b8192.txt)= f81203fabb968a5f2f0eec1e228c380798bfbdb1
MD5(md5b8192.txt)= d09c1248ce6780a5cf4e9cc5946e1ebf


NSRL has a SHA-1 - to - filename mapping available (Jan 2011).

A text file is available which relates the SHA-1 of a file to a file name.
A 600 MB Zip file can be downloaded which contains a 1 GB text file with 18,840,521 rows.
Each row has a SHA-1 hash, a space character, and a filename string.

Example rows:

0000004DA6391F7F5D2F7FCCF36CEBDA60C6EA02 00br2026.gif
000000A9E47BD385A0A3685AA12C2DB6FD727A20 femvo523.wav
00000142988AFA836117B1B572FAE4713F200567 J0180794.JPG

File signatures :
SHA1(SHA_name.txt)= adc0c597e97fe224bbbe8b366b5cf6b941fd6aca
MD5(SHA_name.txt)= a95157eb30acfe39cdfaa7729de4eef5