Archive for October, 2013

The Epic Fail of HealthCare.gov

Wednesday, October 23rd, 2013

Picard Face PalmLike many, I’ve been following the news reports of the launch of the Affordable Care Act’s website HealthCare.gov with great interest. The launch has been described as “unacceptable” by just about everybody in the media and politicians from both sides of the aisle. Though I’m not sure we all have the same understanding of what “unacceptable” means. But I’ll come back to that in a minute.

I have a background that allows me to provide some insights into the epic failure of HealthCare.gov. I have managed the launch of several dozen websites including custom-built, database-driven e-commerce solutions. Nothing as complicated as what’s needed for HealthCare.gov, but enough that I understand the process, the pitfalls and the traps.

And yes, I’ve been an outspoken critic of the Affordable Care Act a.k.a. ObamaCare. I’ve also been the one warning fellow critics for weeks not to gloat about the technical problems that have plagued this website. The failure of the launch is not because ObamaCare is bad policy. It failed because the development of the site was poorly managed.

 So What Happened?
Imagine this scenario: You’ve decided to build one of the biggest and most spectacular homes ever built. And you decide to be your own general contractor. The problem? You’ve never built a home. Certainly nothing of the magnitude you now intend to build. And you’re in over your head.

The administration decided they would act as their own general contractor for HealthCare.gov, possibly the most complex and robust website that has ever been built. From Megan McArdle on Bloomberg:

The Centers for Medicare & Medicaid Services inexplicably decided to take on the role of central project manager itself, assuming responsibility for integrating all the various software pieces they’d subcontracted, rather than assigning that role to a lead contractor. CMS is not known to maintain a pool of crack programming talent with extensive project management experience that can be deployed to this sort of task.

A lead contractor would have been responsible for overseeing that the contractors were fulfilling their duties and responsibilities, and developing adequate testing procedures for the site. But this still may not have mattered. The administration would still have been running the show.

Why did they decide to do this internally? Megan speculates on this as well. I’ll let you read her article. It’s a must read. (And yes, I borrowed her general contractor analogy. I was going to use the analogy that you’ve decided to film the movie Titanic without a director. I liked hers better.)

I’ve been critical of the Obama administration for years for what I’ve seen as a lack of leadership and a lack of understanding for how to get things done. Speeches seldom solve problems. And the problem with always thinking you’re the smartest person in the room is that you don’t feel the need for input from others. When you think you know all of the answers, you fail to ask the most important questions.

The first thing the administration should have done was pull together experts from the online community — CEOs and tech leaders from companies like Facebook, Amazon, Dell and Twitter. Meet with them, give them all a beer, and ask: What do we need to know before we do this? How do we manage this? Not only would these people have provided invaluable advice, I can almost guarantee you that they would have recommended bringing in an outside director to manage the project. What the administration needed was not only somebody knowledgeable and experienced, but somebody who could be the liaison between the administration and the development teams. Somebody who would coordinate the efforts of all of the outside contractors. Provide adequate testing procedures. And just as importantly, manage the administration’s expectations for what could be accomplished within a certain amount of time.

Why Did This Website Go Over Budget?
It’s hard to nail down the numbers, but it appears that this website was initially estimated to cost less than $100 million, and that we’ve already spent several times that amount of money. Reuters runs through some of the numbers here:

In addition, said CGI spokeswoman Linda Odorisio, there were three one-year options, bringing the total potential value of the contract to $93.7 million. By August 2012, spending on the contract was already close to that limit.

This year, the bills skyrocketed. The government spent $27.7 million more in April, an additional $58 million in May and, in its latest outlay, $18.2 million in mid-September.

According to the government records, that brought the total spending for CGI’s work on Healthcare.gov to $196 million. Adding in potential options, the contract is now valued at $292 million.

Why do projects go over budget? Typically one of two reasons. One, the client (in this case the federal government) doesn’t provide full information to the contractors upfront. Or two, the client changes the scope of the project.

Back to our analogy of building a home. Imagine that you’ve dug the foundation, poured the concrete, and started framing the house. Then your spouse says, “No, I think the garage should go on the other side of the house.” So you tell your sub-contractors to move the garage.

When we worked with our clients, we would develop full designs and specs for approval before a single letter of code would be written. Once a customer approves the project, and then later updates the requirements of the project, we would sit down with them and explain how these changes would impact the budget and development timeline.  Some changes may be minor, but some may require scrapping days, weeks, and in this case, possibly months of work. The administration was constantly updating and changing the specs for HealthCare.gov. From some reports, as late as last month! From Reuters:

CGI officials have also told committee staff the widely criticized design feature requiring visitors to create accounts before shopping for insurance was implemented in late August or early September, barely a month before the October 1 start of open enrollment.

This gets us back to the need for an outside director who would have stopped the constant changes to the site in time to test it properly, or not allowed the site to have been launched when it wasn’t ready.

How Much Time Did They Need?
This is a great question, and one I don’t have the answer to. Kathleen Sebelius, the Secretary of Health and Human Services, and the point person for the Affordable Care Act, has been questioned repeatedly this year on whether or not HealthCare.gov would be ready to launch on time. Rumors continued to spread that the exchanges were not ready. She repeatedly said that they were on track for development and testing. As late as September 20th, she said: “Testing is being done. We are very much on track to be ready Oct. 1.” Any “bumps in the road” would be fixed before opening day, she said. See more of her comments here in IBD.

Clearly this wasn’t true. Alarm bells have been going off for months that the exchanges were not ready to be launched. From Forbes:

Back in March, at an insurance industry conference in Washington, the problems were apparent. Henry Chao, chief information officer at the Centers for Medicare and Medicaid Services, openly fretted that the exchanges wouldn’t be ready by October. “I’m pretty nervous—I don’t know about you,” he told the crowd. “Let’s just make sure it’s not a third-world experience.”

Testing either hadn’t been done, or what little had been done, had yielded very poor results. Most accounts have been that testing didn’t begin in earnest until five days before the launch. And according to the Washington Post, just days before launch, the site failed during a simple simulation:

Days before the launch of President Obama’s online health ­insurance marketplace, government officials and contractors tested a key part of the Web site to see whether it could handle tens of thousands of consumers at the same time. It crashed after a simulation in which just a few hundred people tried to log on simultaneously.

Despite the failed test, federal health officials plowed ahead.

I added the bold for emphasis. The site didn’t crash because it was overwhelmed with traffic, despite this continued line from the administration. The site crashed because it was rushed by the administration, poorly managed, largely untested, and not ready for prime time.

This Is Unacceptable!
We have heard from just about everybody that this epic failure of a launch is unacceptable, including from President Obama and Nancy Pelosi. And typically when something is unacceptable, that would mean that somebody will be held accountable. Don’t hold your breath. I’m sure the administration would love to blame the GOP, but that’s a tough argument to make. They can’t blame a lack of budget, though there have been some who have tried, as they’ve gone way over the original budget. Somehow they’ve managed to find several hundred million dollars to fund this project. I’ve been expecting them to throw the developers under the bus, but then this comment from Sebelius appeared in the Wall Street Journal:

After two weeks of review, the HHS secretary concluded, “We didn’t have enough testing, specifically for high volumes, for a very complicated project.”

The online insurance marketplace needed five years of construction and a year of testing, she said: “We had two years and almost no testing.”

You can’t throw the developers under the bus when you’ve now admitted they needed five years to accomplish this project properly. Though I find this conclusion as bogus as her earlier claims that the site would be ready on time. While complex, they do not need five years to develop and test this website.

Sebelius Will Testify In Front Of Congress
I’ve watched these Congressional investigations many times. The grandstanding from the politicians drive me crazy. They either spend their time making a speech or asking all of the wrong questions. At this point, there are only a few relevant questions.

1. Mrs. Sebelius, you’ve stated that you needed five years to develop and test this website. When did you come to this conclusion?

This question is important, though I would not expect a truthful answer. She would either have to admit that she knew before the launch and she lied to Congress about the readiness of the site, or she didn’t know until after it was launched that the site didn’t work. And if that’s true, they’re even more incompetent than I imagined.

2. When did you convey to the President that to do this project correctly, you needed five years of development and testing?

Clearly this would require a truthful answer to the first question for her to truthfully answer this second question. Which will never happen. What I can tell you is this: There’s no way that they didn’t know there were significant problems months ago. There were too many rumors to the contrary. I can guarantee you that the developers were expressing concerns EVERY time the administration changed the specs of the project.

3. Obviously you knew before the launch that the site had not been fully tested and had failed the few tests that had been implemented. Who made the decision to launch the site knowing that it wasn’t ready? And why?

The most obvious reason I can come up with, and this has been speculated by others, is that the administration was afraid to postpone the launch because this would give their critics more ammunition to use against them and ObamaCare. Though I’m unclear how delaying the launch would have been worse for them than launching a site that doesn’t work, was poorly developed, was largely untested and came in way over budget. As it is now, even the talking heads on MSNBC have called out the administration on this one.

4. When will the problems be fixed?

Nobody seems to know, and the administration isn’t saying. I’m sure she won’t give any specifics to Congress. The fact that they’re calling in outside assistance, a “tech surge” as the President called it, would indicate they don’t know. The speculation is that they won’t know the underlying problems until they can fix and test the obvious problems. If they thought they were close, there would be no need to bring in “the best and brightest” for their input. Which makes me wonder, if they’re bringing in the “best and brightest”, who was overseeing this project in the first place? Clearly not the “best and brightest”.

A Parting Shot
And to lighten the mood, if you haven’t seen this from Jon Stewart, you need to watch it. Jon is a funny man.

http://www.youtube.com/watch?v=2Qtx_ZcHOjw

And a Final Thought
Sooner or later they’ll get HealthCare.gov in a workable condition. A few weeks or a few months? Who knows? At some point soon the administration will have to consider extending deadlines for compliance for the individual mandate, or eliminating the penalties for this next year. The longer this takes, the worse the repercussions will be for all of us.