An Example of Forensic Science at its Worst: US v. Brig. Gen. Jeffrey Sinclair

In early 2014, I provided material support in what would end up turning around what was, in their own words, the US Army’s biggest case in a generation, and much to the dismay of the prosecution team that brought me in to assist them. In the process, it seems I also prevented what the evidence pointed to as an innocent man, facing 25 years in prison, from becoming a political scapegoat. Strangely, I was not officially contracted on the books, nor was I asked to sign any kind of NDA or exposed to any materials marked classified (nor did I have a clearance at that time), so it seems safe to talk about this, and I think it’s an important case.

While I would have thought other cases like US v. Manning would have been considered more important than this to the Army (and certainly to the public), this case – US v. Brig. Gen. Jeffrey Sinclar with the 18th Airborne Corps – could have seriously affected the Army directly, and in a more severe way. It was during this case that President Obama was doing his usual thing of making strongly worded comments with no real ideas about how to fix anything – this time against sexual abuse in the military. Simultaneously, however, the United States Congress was getting ramped up to vote on a military sexual harassment bill. At stake was a massive power grab from congress that would have resulted in stripping the Army of its authority to prosecute sexual harassment cases and other felonies. The Army maintaining their court martial powers in this area seemed to be the driving cause that made this case vastly more important to them than any other in recent history. At the heart of prosecuting Sinclair was the need to prove that the Army was competent enough to run their own courts. With that came what appeared to be a very strong need to make an example out of someone. I didn’t have a dog in this fight at all, but when the US Army comes asking for your help, of course you want to do what you can to serve your country. I made it clear, however, that I would deliver unbiased findings whether they favored the prosecution or not. After finishing my final reports and looking at all of the evidence, followed by the internal US Army drama that went with it, it became clear that this whole thing had – up until this point – involved too much politics and not enough fair trial.

My involvement in US v. BG Jeffrey Sinclair came late in the game, just before trial was scheduled to start. I’d been told that over a hundred thousand dollars had already been blown on so-called computer forensics experts. (I didn’t charge them nearly that much). I was asked to look at the evidence from an iPhone. It was both strange and distasteful to me that I was told what others had charged, as if I was expected to charge the same inflated price. I didn’t; I charged them an honest invoice.

As is almost always the case with sexual harassment (and other sexual crimes), Sinclair was thoroughly demonized by the media before trial even started. Don’t get me wrong, the guy’s actions, based on his final plea deal, made him a scumbag by my standards, and I’m glad we never met; we likely would have if the case had gone to trial. Sinclair eventually pled guilty to having a three-year affair with a subordinate, along with a number of other sleazy activities. The more felonious rape and sodomy charges made against him, however, were tied to evidence I analyzed, which would later prove damaging to their case, so much in fact that I was shocked at what a slap on the wrist Sinclair ended up getting for everything else he had admitted to.

This case was being tried out of Fort Bragg, and they contacted me because of my expertise in the field. In 2008, I pioneered the very first forensic techniques to extract evidence from devices and engineered the first tools for law enforcement to perform acquisitions, bypass the PIN, and analyze evidence; I provided these freely for years. Since then, I’ve written many books, trained hundreds of federal agents internationally, and assisted on numerous high profile cases. I’ve got a pile full of swag from agencies around the world, and have gotten the chance to walk on a number of seals I used to only see on TV. It’s been a cool ride, but I am much more cautious these days about who I’m willing to work with, with recent events. I’ve published much of my research, and so it’s since shown up in a number of third party forensics tools, so I’ve effectively outmoded myself, which was the plan all along.

While the media would love to demonize all of law enforcement, and while I do agree there are bad agents and bad agencies out there, there are far more good men and women serving this country. I’ve had dinner with many of them, met their families, and even gone shooting with them. Some of the highlights of the camaraderie for me were shooting one of Al Capone’s original Thompson submachine guns at the Chicago Police Training Academy, late night homicide cases with fully stocked wine cabinets, barbecue parties, and shooting restricted handguns in Canada with the RCMP (they were so excited about these new M&Ps, I didn’t have the heart to tell them that I already owned several back home in the US). The downside has been that I’ve also watched some of my LE friends become so heavily invested in catching their victim’s attacker that they struggled with depression; I’ve taken some of the guys out for drinks and had conversations about the problems they struggle with, seen what a bad day at work can do to their home relationships, and wanted to come home and hug my own kids after some nights myself. While what’s going on at certain echelons of government to destroy our constitution breaks my heart, I’ve continued doing what I can to assist because there are still incredible people out there that would make you proud of your country if the media ever let you see what they did to serve. I am fortunate to have gotten to see that, and even more fortunate to have gotten to play a role in that.

The Call

In mid-January, I received an email from the lead prosecutor at the time, LTC William M. Helixon (“Special Victim Prosecutor”).

Mr. Zdziarski:I am the lead prosecutor in the case of United States v. BG Jeffrey Sinclair.  The US Army is prosecuting BG Sinclair for several offenses, including sexual assault.Recently, an iPhone3 was discovered and turned over to the government for forensic analysis. [redacted…]

Helixon was a very sound minded and professional guy, and really impressed me. I often find prosecutors to be overly hell bent on finding any way possible to make their case. Helixon, on the other hand, was genuinely interested in getting to the truth; the existence of this device had already, it seemed, raise some concerns about this case, and he wanted to bring me in to analyze it. Helixon seemed very grounded, and showed no signs of the mental instability that the Army would later use to discredit him.

As I mentioned on the phone, this is the most important case facing the Army in a generation. [redacted…]Bottom line. The jurist in me does not care about your findings – truth is truth. Please make sure you inform me of the truth. Good, bad, indifferent.I am charged with ensuring justice is done. I cannot do that wo your expertise and advice, regardless of whether it hurts or helps my case.Understand?Will

Helixon had expressed to me his coming promotion to judge. This was supposedly his last case, in fact, before he left the prosecutor seat. He seemed to have a bright future ahead of him, and yet somehow appeared to remain unbiased, and genuinely interested in getting to the truth of this case and in justice being served. Most guys looking for this kind of promotion would feel an urgency to “hit it home” with the last case, or even feel pressured to. Helixon seemed willing to back out of the case if necessary.

The Evidence

I sent my fee schedule to Helixon and his assistants, got a federal CAGE code for billing, and a few days later a DVD and memory card showed up on my doorstep. Most of the time, chain of custody or other policy prohibits me from handling evidence at my home office. In the past, an agency would fly a couple of agents out to me in my little New England town. The former police chief was a great guy, and let us use some offices at the local station back in the day when I was one of the only people in the world who knew how to perform forensically sound iOS acquisitions (just try walking into a police department and explaining that – it goes over much better when the other department calls instead). The small police station in this little NH town had helped me to assist agencies as far away as LAPD’s Gang Justice and Internal Affairs units, ICAC and Organized Crime Units from other west coast agencies, and many others. In this case, the military apparently didn’t have any problem with me analyzing a copy of the evidence from one of the RCFLs (FBI’s Regional Computer Forensics Labs) they worked with. I didn’t handle the iPhone directly. I worked from a physical image dump created by a commercial forensics tool, and three reports from various tools which, as it would turn out, appeared to be misreporting (or at best “under explaining”) at least some of data that the case would later hinge on. What the tools didn’t report turned out to be much more interesting than what they did, and this – combined with whatever other evidence the Army had gathered – eventually led to the turnaround of the case.

When I analyze evidence, I try to do so without knowing exactly what kind of “smoking gun” I’m looking for; often times, I generate a long report with sets of dates and activities, and then afterwards discuss the details of the case with the attorneys to see how relevant my findings are, and we figure out the context that best explains the artifacts. This seems more honest to me than going on a hunt for specific data. I know a number of “professional” forensic examiners who only search for what they’re looking for to prosecute a case – an image, a text, what have you, and then ignore the rest of the evidence on the device, which may exonerate the suspect. I’ve also testified as an expert in certain cases where demonstrating a dedication to science first – as opposed to trying to prove the prosecution’s case – has carried a lot of weight with the jury. I’ve never done this full time (only under special circumstances) and so I’ve kept my nose clean in not selling out to one side.

As it happened, I found myself spending almost as much time correcting reports from third party forensic tools vendors as I did analyzing actual evidence. It’s even sadder that I charged less for my services than these tools manufacturers charge for a single license of their software. I said before that I assisted on a number of high profile cases. I don’t say high profile to sound important, I say it because these types of cases are generally of great importance themselves, and you absolutely need the evidence to be accurate. The military doesn’t just pick up the phone and call people unless they need absolutely critical expertise, and so you’d better be detailed and accurate, maybe even wear pants. Many in the law enforcement community have been trained to “trust the tools”, citing scientific method and all that. Bullshit. Throughout my entire career in forensics, the tools have shown me quite the opposite of anything scientific. When it comes to forensic software, examiners should default to distrusting the tools, and prove the most critical results themselves. Similarly, the court should be looking at the actual raw data, and determining whether or not the examiners can even explain it. Let me explain why.

Forensics Tools are Terrible

In forensics, we often misplace our trust in tools that, unlike tried and true scientific methods, are closed source. The most widely accepted tools out there are run by software that is protected as a trade secret, and not some open academic project available for scrutiny. While true scientific process relies on making our findings repeatable and verifiable, the methods to analyze data are either patented, or almost always involve guarded trade secrets. This is the complete opposite of the scientific method, where methods are fully explained and documented. In the software industry, repeatable is exactly what you don’t want your methods to be – especially by your competitors. The nature of secrecy in the software industry doesn’t rub well against the open scientific nature that you’d expect to find in forensic, or other scientific disciplines. As such, “software” is not scientific in nature (as far as forensics is concerned), and should not be trusted using the same rules as science. Sure, we have some validation experts out there. NIST does a good job at some things, notsomuch with others. Even still, such tests are only a single data point on an ever-evolving software manufacturing process riddled with regression bugs, race conditions, and other, often un-reproducible programming errors that only show up at that one time when you most need them to be reliable.

A recent court ruling, United States v. Daniel Harry Milzman, highlighted some of the technical issues that the criminal justice community is finally starting to wake up to with regards to this closed source “scientific” world we live in. The judge basically said that, until you can explain how you’re going to conduct your search only for data that you have a warrant for, I’ll consider his phone protected under the 4th Amendment (such a rarity to see coming from a judge these days). This whole matter seems that it could have been resolved if the forensic examiner could have just explained technically how his tools work… unfortunately, not many can. While there are many smart investigators out there, the user interface and workflows in today’s tools are making many investigators dumb, and what’s even sadder is that the tools are not even as smart as the investigator would be if he did his job without them.

There are some good forensics tools out there, don’t get me wrong, but these aren’t the mainstream tools you see in every police department. More often than not, the popular tools have components that are poorly written, and part of this is due to the fact that most software engineers are not forensic scientists or pen-testers, and have no solid methodology to validate the evidence they’re reporting on. Having worked on commercial forensics software first hand, I can attest to this: most of the software engineers working on forensics tools have little experience with forensics or forensic methodology. As accuracy of data is critical, this presents a bit of a problem (more than a bit, actually). Tools validation is critical to the healthy development cycle of a forensics tool, and unfortunately many companies don’t do enough of it – or any of it. Simply running a dozen QA units through the software doesn’t guarantee that it’s parsing or reporting all of the data correctly for the other billion devices out there. Since it’s closed source, the community at large can’t validate the tools either. If investigators aren’t doing their homework to individually validate the artifacts on every case (and subsequently provide feedback to the software manufacturer), the consequences could mean an innocent person goes to jail, or a guilty one goes free. This particular case could have turned out quite differently for this very reason.

Without getting into the details of this case, establishing a timeline of device use and interaction was critical. Determining whether or not the device had been used, accessed, and whether or not any information had been added or deleted during a certain time frame would materially impact this case. As my findings would later reflect, the commercial tools that had been used to initially evaluate the evidence on the device had either misreported key evidence, or failed to acknowledge its existence entirely.

There are reasons I’m not going to dig into the details of the case. Certain people that were involved could easily be pinpointed by revealing technical details that could be pieced together with news reports, and help build a story in your mind that would probably be inaccurate. The details aren’t so important as the errors that were made. All you need to know from a technical perspective is right here: some of the types of information that these commercial tools were (and likey still are) misreporting is significant. Evidence and timestamps of a device erasure event. Evidence of a backup restore event. Application usage dates. Application deletion events and timestamps. File access times. This, and many other types of artifacts are often either completely overlooked by numerous commercially sold, expensive-as-hell tools, or in the case of at least one tool – seemingly made up data. All of these came into play in this case and would later play a role in its outcome.

Helixon’s Departure

While I remained in close contact with LTC Helixon during the beginning of my involvement, I had read of his abrupt departure in the news before I even got news of it myself through Army channels. Reports of finding him drunk and suicidal in a hotel room, or having a breakdown, didn’t seem like him – at all. It didn’t sound like the well put together professional prosecutor – soon to be judge – that I had worked with. As I presented him with my findings, which ultimately destroyed the credibility of this case, his emotional state hadn’t degraded, nor had he shown any signs of stress or anger. In fact, Helixon was quite content to be given a very clear and concise explanation of the truth based on the evidence. His abrupt departure, and subsequent cover stories, were nothing short of shocking to me, but more suspiciously so, had appeared immediately after Helixon had briefed his superiors on my findings.

In retrospect, I’ve got to wonder if Helixon’s promotion was hinged upon making an example out of Sinclair, or if perhaps his decision to move forward with the case at all (which ultimately fell apart) was so damaging to the Army that it put his career in jeopardy. This is merely speculation, but the best conclusion I can draw. I didn’t perceive Helixon to be unstable at any point to the degree that I’d expect him to be found drunk and suicidal in a hotel room – but if those events were in fact true, I have little doubt that the pressure put on him to win this case contributed significantly to that.

Closed Source is Not Scientific

When you’re dealing with criminal charges like this, a time difference as little as a day can (and likely did) mean the difference between decades in prison or not. Some other misleading information I found was even further off. Software engineers did a sloppy job on the forensics tools, and that almost led to wrongful imprisonment – at least based on this evidence. If you’re going to implicate someone in a crime based on information your software is reporting, then you’d better get those dates right. Also: stop missing giant boulders of evidence like device restores and application deletions. You know something’s wrong when the dates reported by these tools would have implicated another forensic examiner for deleting an application in the lab.

How could developers be so ignorant to miss this stuff? Come on, making up access times because you didn’t realize that iOS doesn’t keep them (as just one example)? That’s just terribly sloppy. (I investigated a little. As it turns out, this particular manufacturer first copies the iOS file system onto the Windows partition that their software is running on, and then pulls the timestamp information off of the copy of the data.) Another example of tools gone wild includes application deletion timestamps. There’s a twist to how application uninstall dates are reported in iOS. At the time of that specific version, the timestamp was not the uninstall date, but rather the timestamp of when SpringBoard realized the application had been uninstalled – which typically happened when the device was next rebooted. This could be days or even weeks off, and it was in the reports. One last example of technical errors is the application usage information. Application usage is reported as a “day number”, but many tools don’t understand how that day number is calculated. After disassembling the AggregateDictionary framework, I discovered it was taking time(0), dividing by 86400, and truncating the decimal place by assigning the result to an integer. As a result, these tools were not only misreporting application usage to be a day off, but also had failed to take into account both the time zone of the device at the time and the time zone of the investigator running the report. Sigh.

Many tools are irresponsible and do not respect evidence, to say the least. But this is the quality of the forensics software assisting our government and military. Poorly written, over-priced assumptionware. The reports I’ve read had these and many other flaws in them. And sadly, this isn’t the only case I’ve worked on where I’ve been asked to review reports from third party tools. I’m often asked to consult on cases when commercial solutions have failed or fallen apart, and my own more hands-on techniques are required. To this end, forensics feels more like janitorial work than science.

Why did these companies make such poor mistakes? We can’t just blame laziness. The root of the problem here is that software engineers are almost never also information security experts. Software engineers understand process, methodology, design patterns, languages, and spend most of their time adding functionality outlined in user stories or requirements docs. While software engineers do a great job at making software, they’re not usually good at reverse engineering an operating system. For that, you need the kind of reverse engineering skill you find in the people who work infosec for a living. This kind of reverse engineering is crucial when dealing with operating systems, such as iOS, that are closed source. Apple doesn’t give magic tech notes to these guys to just divulge all of their forensic secrets. Quite the contrary, most closed source manufacturers try very hard to protect whatever code secrets they can. You can’t fully understand how iOS works just by looking at it; you have to completely reverse engineer it down to the bare bones machine code. In a sense, closed source is making it harder for people to get a fair trial these days, and that’s including both those of the operating system to those writing forensic tools.

Sending software engineers to reverse engineering 101, or simply hiring some reverse engineers / pentesters to validate your forensics products can certainly help to avoid bugs like these, many of which are likely to go unnoticed, since bug reports from convicted felons are likely to get little attention. That $5K for Hex Rays that you invest in today could save you exponentially more than that in the long run, in everything from embarrassment, loss of business, lawsuits, or just plain old man-hours trying to fix bugs that get reported.

More importantly, however, forensic examiners really need to, in order to do their jobs to the most competent capacity, understand software engineering and reverse engineering. They don’t just need this to validate the tools they’re using, but when their evidence gets called into question, gives them solid footing to prove that the software is producing a timestamp or other evidence based on a specific programmed behavior. A forensic examiner has to deal with a number of different scenarios, many of which may include the tiniest of details, which could make a big difference in a case. Being a good forensic examiner means acquiring the skills to be able to validate the tools yourself, by hand if necessary. For this, you need to understand how software works, as well as how to reverse engineer it. This is why a CJ/CS track seems so appealing to a lot of people.

Bottom line is this: You can’t fully trust forensic tools these days. I’ve used (and even helped develop) a number of the top contenders, and they all have their shortcomings. There are some great tools out there, but there are also some bad ones. A forensic examiner isn’t defined by the tool he or she uses to generate reports. A good forensic examiner simply uses them to get a foundation to build their case on, and relies on their skills to validate the information, to ultimately tell the story. If you’re interested in the forensics industry, this is one way to distinguish yourself as an expert in the field. If you’re interested in software engineering, consider also learning how to break and dismantle code, as well as write it. This makes for a very well rounded software engineer, and is one of the reasons I’ve been in high demand in the forensics industry. Lastly, if you’re on the criminal justice side of things, question everything: especially the fine details important to your cases that rely on reporting like this from tools, and make sure the examiner you’re using has done their homework to verify this information. 

Closing Remarks

It wasn’t Sinclair’s PR firm, or the good old boy system, or anything else that caused Sinclair’s case to so dramatically plead out. All of the drama I saw indicated that the Army was hellbent on making an example out of Sinclair. Evidence sometimes isn’t what you think it is. It takes a lot of good, solid analysis to fully unwrap and explain it all, and that is even harder to do with error-prone forensics tools that misreport evidence. I worked with a very intelligent special agent on sorting through this, and when it was all said and done, my gut feeling is that the lack of solid evidence led to the plea deal that played out publicly.  I wasn’t privy to all of the evidence on the case, other than what else I’ve read in the news, but I am confident that had there been solid evidence to convict Sinclair, it would have been used to the fullest extent. There may not have been a case against Sinclair on the more felonious charges, but what he did admit to was completely unethical, and flat out embarrassing to the Army and a stain on this country. I thought he deserved a lot worse. The only thing worse than the case itself was the political drama surrounding it, and what I perceive as a desperate need to make an example out of someone in order to maintain certain powers at a congressional level. I also feel sorry for whatever happened to Helixon. Only he knows what really happened, but he seemed to be the only prosecutor I’d dealt with that was genuinely concerned about the truth. I would have expected less drama from the Army. I left this case rather unimpressed. I’m sure there’s more to the story here but I’ve no idea what. Perhaps other have the pieces.