So promoting QSSI seems like a bit of an unearned reward; I shudder to think how much they’ll get in addition to the $85 million the government already paid them for healthcare.gov. It does signal, however, that the $200-plus million contractor CGI Federal is in the doghouse, since, to paraphrase CGI’s Cheryl Campbell, CGI made the whole front of the house that is healthcare.gov, and QSSI only made “the front door”; evidently the front door is now looking better than the rest of the place. (I previously suspected that CGI bad-mouthed the administration to the New York Times, which would be another strike against them.)
"There's a punch list of fixes,” Zients said, “and we're going to punch them out one by one." He makes it sound so easy. It is, sometimes. Like lines of code, all bugs are not created equal. Fixing bugs is an asymptotic process. Let’s say there are 100 bugs in a piece of code. The easiest and most commonly occurring ones will be fixed first. That leaves the rarely occurring ones and the trickiest ones. Those will take more than half the debugging time. One software maxim says the hardest 10 percent of bugs take 90 percent of the time to find and fix.
Here is a rough approximation of the debugging of healthcare.gov over time:
Prior to release, minimum testing was done. Bugs were found and fixed haphazardly but not comprehensively. After release, a tremendous number of bugs were found, and some were fixed, but there was evidently little process in place, as it took three weeks just to get someone—who turned out to be Zients—to say the first word about process. With some bug triage methodology and management in place, I optimistically believe that the pace of fixing issues has picked up and that they have entered the hump in the graph of fixing more bugs faster.
Contractors testified that systems testing only took place in the last weeks before Oct. 1, rather than the months you really need. Surprisingly, this is somewhat good news from a bug perspective. If virtually no testing was done, that means many of those “easy” bugs are still there, and so the programmers attacking the system right now will be able to make a lot of progress in a hurry—as long as they know whether something is a bug. If the system integration among the contractors’ pieces and the government systems and the insurance companies has been underspecified, a lot of that will need to be worked out in a hurry. With good lines of communication, though, this is still possible. (Zients presumably has the authority to remove anyone who gets in the way, like greedy contractor executives.)
As the more crucial bugs are fixed, the number of fixes trails off. The goal is to reach relative stability, indicated by the last line on the graph. I say “relatively” because it’s an asymptotic line: You never know for sure that you’ve fixed every bug; you only know that fewer and fewer bugs are being discovered. That line signifies the point where people breathe easy and bugs are no longer coming in faster than they can be fixed. Zients’ goal is to get as close as he can to relative stability by Nov. 30 so that the site is functioning for “a vast majority of users.” Presumably there will be contingency plans in place for people who still have trouble, but those numbers will be small enough that the administration can announce that things are more or less working—though given the tone of the congressional hearings in the past week, I expect Republicans to leap on even the rarest of bugs as evidence that the surge has failed and we need to send in David Petraeus.
On the other hand, there are some disturbing indications that healthcare.gov isn’t anywhere near getting over that bug-fix hump. Consider that the administration had to bring in Zients—someone wholly outside of the Centers for Medicare and Medicaid Services (CMS)—before we got anything resembling a straight answer about the state of the project. Even if the administration felt it necessary to bring in a crisis manager to fix things in a hurry, it should not have taken a month to find someone to admit statistics and confess to functional problems.
Worse, with the exception of the data hub being fixed, Zients made no mention of any progress since the rollout. Nothing should have stopped him from citing bugs being fixed, insurance applications working better, or any other good news. But he didn’t.
So Zients did not give the impression of jumping into an ongoing repair process, but of starting one. He even made it sound like the list of problems was something he had drawn up, not one he had inherited. None of these are auspicious signs, nor is the Verizon data center crash that took down healthcare.gov on Sunday, and then again on Tuesday. Zients has a month—a deadline he likely did not choose—to make healthcare.gov good enough. It’s possible. But my dead-reckoning says he has very little margin for error.