I recently finished consulting on a rather high profile case, and once again found myself spending almost as much time correcting reports from third party forensic tools vendors as I did analyzing actual evidence. It’s even sadder that I charged less for my services than these tools manufacturers charge for a single license of their buggy software. I don’t say high profile to sound important, I say it because these types of cases are generally of great importance themselves, and you absolutely need the evidence to be accurate. Many in the law enforcement community have learned to “trust the tools”, citing scientific method and all that. The problem I’ve found throughout my entire career in forensics, however, has shown me quite the opposite. When it comes to forensic software, the judge should not automatically trust the forensic tools as part of the scientific process, and neither should the forensic examiners using them. Let me explain why…
In forensics, we often misplace our trust in tools that, unlike tried and true scientific methods, are usually closed source. While true scientific process relies on making our findings repeatable and verifiable, the methods to analyze data are sometimes patented, and almost always considered trade secrets. This is the complete opposite of the scientific method, where methods are fully explained and documented. In the software industry, repeatable is exactly what you don’t want your methods to be – especially by your competitors. The nature of secrecy in the software industry doesn’t rub well against the open scientific nature that you’d expect to find in forensic, or other scientific disciplines.As such, “software” is not scientific in nature, and should not be trusted using the same rules as science. Sure, we have some validation experts out there. NIST does a good job of validating logical data acquired from a number of devices and has struck some good and interesting results that have helped the industry. Even still, such tests are only a single data point on an ever evolving software manufacturing process riddled with regression bugs and programming errors that only show up in certain specific data sets.