What Does Healthcare.gov’s “5 Million Lines of Code” Really Mean?

Decoding the tech world.
Oct. 22 2013 1:53 PM

5 Million Lines of Obfuscation

The healthcare.gov mess is inspiring some messy reporting.

Lines of code.

Photo by Denis Belyaevskiy/iStock/Thinkstock

Last weekend, some anonymous “specialist” told the New York Times that “5 million lines of software code may need to be rewritten” in order to fix the mess that is healthcare.gov. (The good news, according to the source, is that the project has a total of “500 million lines of software code,” so only 1 percent has to be rewritten. So the code’s 99 percent good—or something.)

David Auerbach David Auerbach

David Auerbach is a writer and software engineer based in New York. His website is http://davidauerba.ch.

I don’t mean to jump back on my hobbyhorse of complaining about lack of knowledge in tech journalism, but printing a claim like that is egregious.

Why? Well, here’s a line of C++ code:

Advertisement

}

The close curly brace signals the end of a block of code. It could be put on the same line as the previous, more substantive line, but for the sake of cleanliness, programmers tend to put it on a line of its own. When it comes to coding in HTML, Perl, and AJAX, different programmers have different styles. Some will split code up into many lines; others will compress it into a handful of lines. I’ve seen nearly identical segments of code written in 10 lines or in 50.

Here’s another line of C++ code.

// TODO: make sure this code doesn’t crash!

That’s a comment. It doesn’t do anything—those two slashes at the beginning tell the compiler (which converts code into actual computer instruction) to ignore the line. It’s there to explain things to people reading the source code, or in this case to remind the programmer to fix whatever lies immediately below. I’ve written cryptic bits of code that required more lines of comments than lines of actual code, simply to explain what on earth was going on.

On the other hand, here’s a line of APL (“A Programming Language”) code:

(~R∈R°.×R)/R←1↓ιR

That code prints out all the prime numbers from 1 to R. APL is a notoriously terse and nightmarish language. I have successfully avoided ever coding in it. One single line of APL code could contain half a dozen bugs.

So not all lines of code are created equal. As a programmer, I had weeks where I produced 1,000 lines of code. I had weeks where I produced 20. Usually the latter weeks were more grueling, because any 20 lines requiring that much time and effort are going to be a) important, b) complicated, and c) bug-prone. The 1,000 lines were far more likely to be simple stuff that I could code by rote. I even had weeks where I removed 2,000 lines of code by removing redundancies between similar blocks of code. Those were the best weeks of all, because less code means fewer bugs.

Programmers who do user interface code—which is responsible for the visuals and input components of software—tend to produce far more code than other programmers, because user interface code requires a lot of boilerplate. I knew programmers who wrote 10,000 (good) lines of user interface code in a week. Many of them were copied and slightly modified from other projects or example code.

Consequently, it’s rather silly to say, as the Times article does, that “a large bank’s computer system is typically” 100 million lines of code. Investment banks have far more complex code than commercial banks—they need more in order to do all their clever, sneaky trading. Assuming the Times is referring to commercial banks, there is such variety among implementations and coding standards that speaking of an “average” amount of code is meaningless. Bank code written in FORTRAN will be far longer than bank code written in Python. Does it make a difference? Not really.

But while the numbers in the Times article don’t tell us much about the healthcare.gov codebase itself, they do tell us something about the “specialist” sources that inform the article. The sources are not programmers, because programmers would not speak in terms of lines of code with no further context. We hear that “disarray has distinguished the project” in part because government “officials modified hardware and software requirements for the exchange seven times.” The officials probably modified them 70 times—requirements for any software project are constantly in flux, and it’s expected that project managers and software engineers will adapt. Modifications alone do not signal a project in disarray.

We hear that the Centers for Medicare and Medicaid Services (CMS) lacked the expertise to link the individual pieces of healthcare.gov together. That does not explain why the “data hub”—the single component provided by Quality Software Services Inc.—proved to be “a particular source of trouble,” something I had surmised two weeks ago. If individual contractors were producing garbage, CMS’s expertise or lack thereof wouldn’t have made a difference to the final product.

The sources also say that CGI Federal, which won the $90 million contract to develop healthcare.gov’s back end, was asked to replace the data hub, though this approach was abandoned as “too risky.” That’s a hint that the article’s sources seem eager to shift the blame to CMS, to the White House, and to QSSI, and away from CGI. The Times claims that CGI was not responsible for healthcare.gov’s “integration,” but the Washington Post’s Lydia DePillis reports that CGI Federal was in fact responsible for “knitting all the pieces together, making Quality Software Services’ data hub work seamlessly with Development Seed’s sleek user interface and Oracle’s identity management software.”

I have no idea who the Times’ sources were, but they sure sound like employees of CGI Federal. Because they almost certainly aren’t programmers, I’d guess they are probably mid- or high-level managers who are trying to salvage CGI Federal’s reputation. They may well be “specialists,” but their specialty is more likely the art of procuring government contracts.

This is to be expected. What’s less expected is that such anonymous sources would be treated with this degree of credulity by national reporters who lack technical understanding of their subject matter and are thus more likely to parrot whatever a “specialist” tells them. The Times has a great tech reporter, Natasha Singer, who has done well-informed work on consumer profiling, taking little for granted. They should put her on this story.

TODAY IN SLATE

Sports Nut

Grandmaster Clash

One of the most amazing feats in chess history just happened, and no one noticed.

The Extraordinary Amicus Brief That Attempts to Explain the Wu-Tang Clan to the Supreme Court Justices

Amazon Is Officially a Gadget Company. Here Are Its Six New Devices.

How Much Should You Loathe NFL Commissioner Roger Goodell?

Here are the facts.

Amazon Is Officially a Gadget Company

Science

The Human Need to Find Connections in Everything

It’s the source of creativity and delusions. It can harm us more than it helps us.

Food

How to Order Chinese Food

First, stop thinking of it as “Chinese food.”

Scotland Is Inspiring Secessionists Across America

You Shouldn’t Spank Anyone but Your Consensual Sex Partner

Moneybox
Sept. 17 2014 5:10 PM The Most Awkward Scenario in Which a Man Can Hold a Door for a Woman
  News & Politics
Jurisprudence
Sept. 18 2014 10:42 AM Scalia’s Liberal Streak The conservative justice’s most brilliant—and surprisingly progressive—moments on the bench.
  Business
Business Insider
Sept. 17 2014 1:36 PM Nate Silver Versus Princeton Professor: Who Has the Right Models?
  Life
Outward
Sept. 18 2014 11:25 AM Gays on TV: From National Freakout to Modern Family Fun
  Double X
The XX Factor
Sept. 18 2014 12:03 PM The NFL Opines on “the Role of the Female”
  Slate Plus
Slate Fare
Sept. 17 2014 9:37 AM Is Slate Too Liberal?  A members-only open thread.
  Arts
Brow Beat
Sept. 18 2014 11:48 AM Watch the Hilarious First Sketch From Season 4 of Key & Peele
  Technology
Future Tense
Sept. 18 2014 10:07 AM “The Day It All Ended” A short story from Hieroglyph, a new science fiction anthology.
  Health & Science
Bad Astronomy
Sept. 18 2014 7:30 AM Red and Green Ghosts Haunt the Stormy Night
  Sports
Sports Nut
Sept. 18 2014 11:42 AM Grandmaster Clash One of the most amazing feats in chess history just happened, and no one noticed.