At healthcare.gov, many people are still unable to create accounts, choose from a list of health care plans, and sign up for one. The system is down, or overloaded, or shows perplexing errors. Is the ongoing plight of healthcare.gov down to “crashing servers,” as many have assumed? Perhaps. But a server isn’t just a battery—you can’t just add more until you’ve got enough power. A peek at the architecture of healthcare.gov reveals a vast entourage of many different servers old and new, any one of which could have its own unique crashing problem, as well as a mysterious “data hub” that was responsible for connecting them all together—and thus if one failed, they all failed.
To fix healthcare.gov’s woes, you’d have to look at each kind of server individually and see how it failed, like checking Christmas tree lights to find the one bad bulb. All you need to do is replace the right bulb, but you don’t know which one that is. It’s not necessarily a hugely difficult process, and it doesn’t imply fundamental problems with the system, but it can be tedious and time-consuming. And if every single bulb on the string is owned and operated by a different government contractor or agency who isn’t used to sharing the details of its unique bulb housing and management with anyone else, it’s going to take even longer.
I was, it seems, a bit naive in thinking there were merely two cooks (or two bulb managers) in the kitchen behind healthcare.gov. The number of players is considerably larger than just front-end architects Development Seed and back-end developers CGI Federal, although the government is saying very little about who’s responsible. The Department of Health and Human Services’ Centers for Medicare and Medicaid Services (CMS), which issued the contracts, is keeping mum, referring reporters to the labyrinthine USASpending.gov for information about contractors. (I was not able to obtain any useful information from that site, though it does make healthcare.gov look pretty good in comparison.)
By digging through GAO reports, however, I’ve picked out a handful of key players. One is Booz Allen, the people who brought you Edward Snowden. Despite getting $6 million for “Exchange IT integration support,” they now claim that they “did no IT work themselves.” Maybe Snowden can help us out on this one, though as far as I can tell, Booz Allen does seem to be ancillary to the overall healthcare.gov project.
Then there’s CGI Federal, of course, who got the largest set of contracts, worth $88 million, for “FFE information technology and healthcare.gov,” as well as doing nine state exchanges. Their spokesperson’s statement is a model of buck-passing: “We are spending 24 hours a day, seven days a week working with our client and working with our partners in order to stabilize the enrollment [process] and finish the roll-out of this very complex project.”
But which partners? The most interesting is Quality Software Solutions Inc. (QSSI). Despite the laughable name and inexplicable slogans such as “Quality is a Q word”—hard to argue with, I guess—they’ve been doing health care IT since 1997, and got $55 million for healthcare.gov’s data hub in contracts finalized in January 2012. But then UnitedHealth Group purchased QSSI in September 2012, raising eyebrows about conflicts of interest.
In Congressional testimony on Sept. 10, QSSI vice president Michael Finkel said that there was no need to be worried about the conflict of interest—but he also revealed more about the architecture of healthcare.gov and the data hub than anyone else has in the last week. Finkel described the data hub as the ultimate middleman of the entire system, “funneling” queries to databases from multiple sources. This would not be an impossible task, but it would require a formidable level of technical coordination. Imagine if Google, Apple, and Microsoft were suddenly asked to develop a website together.
Finkel described the data hub as the master switchboard for the entire sign-up and registration process. By integrating with “external information sources, such as government databases,” it would 1) verify a consumer’s data, including citizenship and identity, and 2) issue queries to these various databases as needed to “verify applicant information data [and] determine eligibility for qualified health plans.” The data hub did not have any of this information itself, nor did users use it directly. Rather, the hub acted as the intermediary between the healthcare.gov website, where consumers would input their information, and a variety of other databases containing consumer and health insurance information, coordinating between them. QSSI “developed” the data hub for CMS and was responsible for “ensuring proper system performance, including maintenance.”
The government has repeatedly claimed that various problems of healthcare.gov are due to server overload—too many people attempting to sign up. The data hub would certainly be ground zero for such load issues, but not the only one. If any of the other databases it spoke to were overloaded, the sign-up process would break anyway. The conundrum may not even be in the data hub or in healthcare.gov, but in some pre-existing citizenship database that’s never had to cope with the massive crush of queries from the hub.
The fundamental fault is in the failure of coordination between the two systems, the failure to test from end to end, and—to use a term of art in software engineering—the failure to handle failure gracefully. To the extent that the data hub was the nexus for integrating many systems (including the healthcare.gov front end), “systems integrator” QSSI had the responsibility to work out graceful failure conditions with all of their partners and a comprehensible user experience in such cases—something that clearly wasn’t done. This points to bad management, lack of accountability, and a broken contractor procurement process.
Development Seed President Eric Gundersen oversaw the part of healthcare.gov that did survive last week: the static front-end Web pages that had nothing to do with the hub. Development Seed was only able to do the work after being hired by contractor Aquilent, who navigated the bureaucracy of government procurement. “If I were to bid on the whole project,” Gundersen told me, “I would need more lawyers and more proposal writers than actual engineers to build the project. Why would I make a company like that?” These convolutions are exactly what prevented the brilliant techies of Obama’s re-election campaign from being involved with the development of healthcare.gov. To get the opportunity to work on arguably the most pivotal website launch in American history, a smart young programmer would have to work for a company mired in bureaucracy and procurement regulations, with a website that looks like it’s from 10 years ago. So much for the efficiency of privatization.
“The problem here is nobody knows what happened, and that’s not acceptable,” said Gundersen, who added that he had no contact whatsoever with CGI Federal or QSSI. Gundersen is a strong advocate of open-source processes as a way to increase civic involvement with important government projects. He wasn’t certain why the open sourcing efforts, which he and HHS Chief Technology Officer Bryan Sivak embraced, seemed to have utterly stalled after Development Seed turned over their work in June. Sivak has been absent from public discussion and Twitter since last week, though he proposed an SXSW panel last month on “disrupting government.” I bet he knows where the digital bodies are buried. He assumed his post in mid-2012, long after the contracts had been handed out, so I imagine he is about the most frustrated CTO on the planet right now. There’s no way Sivak could have stopped the cronyism, though. Companies like CGI Federal and QSSI are locked in for years on end, regardless of their performance.
Gundersen, who spoke openly about his experience, was scathing toward the procurement process. “If people don’t see the need for procurement reform after this,” he said, “we’re in trouble.”
