NSRL: The Next Generation

Summer 2002 we decided to switch over to the NIST environment for data collection

Still hashing only distribution media, so we did some research into how the media hashes compared to installed hashes - 75% to 90% similar

Continued hashing distribution media only - best ROI at the time

At YAPC::Europe:2002 I attended a talk by Nicholas Clark of London.pm, "When Perl is not Quite Fast Enough"
http://www.ccl4.org/~nick/P/YAPC2002/

• try: better algorithm, better hardware, mod_perl, rewrite in (C, ...)
• banish demons of stupidity: hashes, regexps, un/pack, undef
• benchmarking: perl -d:DProf foo.pl and dprofpp
• benchmarking: use Benchmark ':all'
• perl -MO=terse -e 'command' to count operations
• most significant: use fast features, give interpreter hints, less ops, less CPU, less RAM

I dug deeper into Perl and rewrote most of the hashing code

Hashing was still done by a .EXE

In the back of my mind was the aim for portability to other platforms

Late 2002 moved the hashing constellation onto 2GHz, 2GB RAM Win2K PCs

Moved the file server onto a separate NT box with 180 GB RAID 5

Still ran MS SQL 7 on original NT 4 box

December 2002 moved into a real lab room

Early 2003 attended a Damian Conway class sponsored by Boston.pm on "Data Munging" which provided a (dangerous) lot of inspiration

Code could still run in "monolith" mode

March 2003 released code to public for Win2K platforms

Started looking at Red Hat Linux and Mac OS X for hashing platforms