I've been working on healthcare.gov for the last few months alongside a bunch of other Google, Facebook, and Y Combinator alums.<p>I'll always remember what Mikey told us in December, after the site was back up, could handle a non-trivial amount of traffic, and people who wanted health insurance could finally get it:<p>"1 in 1000 uninsured people die each year. It's not an exaggeration to say that due to the work we're doing here, 5,000-10,000 people will live to see the end of 2014. You should be proud of what you've done, but we should also all be grateful to have this opportunity."<p>We're all grateful to be here, but there's a hell of a lot more work to be done.<p>If any of you out there are an amazing software engineer or SRE, and want to help make our government work better, please shoot me an email: brandon@hcgov.us!
Dickerson rules:<p><i>"Rule 1: "The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame."<p>Rule 2: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it." (Explained Dickerson later: "If you can get the managers out of the way, the engineers will want to solve things.")<p>Rule 3: "We need to stay focused on the most urgent issues, like things that will hurt us in the next 24--48 hours."
"</i>
>> "It is also a story of an Obama Administration obsessed with health care reform policy but above the nitty-gritty of implementing it. No one in the White House meetings leading up to the launch had any idea whether the technology worked."<p>I used to be this guy. The guy with the lofty ideas, but who thought the implementation was "beneath me". The guy who would sit around, waxing poetic about various features, user acquisition, header alignment, etc. Don't get me wrong I had serious technical chops, but fixing that annoying localization bug? Blegh. Form encoding off? Don't wanna get my hands dirty. I had "big ideas"! I was going to change the world! People who change the world don't do the dirty work! So I'm very empathetic to the Obama Administration.<p>Like them, I needed a real wake up call. In my case, a friend who had implemented an idea I had sold the software for a lot of money. When I confronted him about sharing the profits he started running git blame on files across the project. My name came up maybe once or twice, across a multi-k LOC project, and even then on nearly inconsequential lines. It hit me then that while ideas may have value, the implementation usurps all of it. An idea alone is powerful, but once it's implemented the idea becomes worthless. At that point it's all about rolling up your sleeves and <i>getting shit done</i>. When you focus on that problems like an "ID generator" becoming a bottleneck (I had to read that bit several times over... apparently I need to start raising my rates to the hundreds of millions) disappear. It's a hard lesson I had to learn, and it's one the Obama administration has hopefully learned as well. Of course, I had just turned 17 when I learned my lesson, and Obama is now a lame duck with less than 2 years left on his final term. I guess this exemplifies my greatest struggle with the Obama legacy, in that it has become one defined by squandered potential.
HealthCare.gov is definitely NOT fixed or "revived" yet. I was trying to register for healthcare for the Feb 15th deadline and there were errors after errors. At the end, after hours of trying and waiting on hold for the tech help-line, I was not able to register at all and missed the deadline. The help-line's answer was, we are still working out the kinks, please print out the PDF and fill it out manually. It's still a disaster-show.
Wow! They weren't even caching database calls:<p><i>HealthCare.gov had been constructed so that every time a user had to get information from the website's vast database, the website had to make what's called a query into that database. Well-constructed, high-volume sites, especially e-commerce sites, will instead store or assemble the most frequently accessed information in a layer above the entire database, called a cache.</i><p>It also struck me as a little funny that Time had to define the term 'cache'. Even my mom (74 years old) has some idea what a cache is (browser cache).
One question I have - were the high-tech wizards paid standard gsa rates?<p><a href="https://www.gsaadvantage.gov/ref_text/GS35F5457H/0LUS2P.2NVKCH_GS-35F-5457H_GS35F5457HGPLOY.PDF" rel="nofollow">https://www.gsaadvantage.gov/ref_text/GS35F5457H/0LUS2P.2NVK...</a><p>Page 25. Top developer rate for 2014 is $99/hr which requires 10 years of experience.
<i>But one lesson of the fall and rise of HealthCare.gov has to be that the practice of awarding high-tech, high-stakes contracts to companies whose primary skill seems to be getting those contracts rather than delivering on them has to change.</i><p>If this happens the entire contracting landscape of DC would change dramatically.
> <i>the people running HealthCare.gov had no "dashboard," no quick way for engineers to measure what was going on at the website</i><p>Someone needs to disrupt the government contracts business pronto.
"Revived" is probably the right term to use.<p>A clinically-dead patient can be revived, but the condition they're in is <i>far</i> from OK.
I heard that several people from YC startups and Google are helping out. This is one of the reasons why we're seeing a lot of improvements to the Federal Exchange site.
think about it this way. you have a bunch of technically unsavvy people, because they have been working in government contracts forever. now you come along, and say hey i could build this for you for much less money, and much better.<p>how in the heavens would he know that you can deliver? you'll choose the one you'll know at least. and since he has documents to back him up(mcse, mcsa, he might have worked for microsoft for a while). he's the obvious choice.<p>i worked for the government in healthcare for a while. these are not intentionally malicious people. hell, they even want to make it better, but it's a combination of regulation(bidding system that favors friends), lack of knowledge, and a little bit of ignorance.<p>but if you want more details feel free to ask. it was kind of fun to see a corruption case close up.
"...government regulations did not allow them, even though they offered, to be volunteers if they worked for any sustained period. So they were put on the payroll of contractor QSSI as hourly workers, making what Dickerson says was "a fraction" of his Google pay"<p>This information troubles me, because QSSI was just hired to fix Maryland's broken Health Exchange site. It sounds like the company is cashing in on the donation of brandonb and others experts' time?<p>brandonb, is your team involved with the Maryland efforts, or is that a completely separate team?
I have a hard time believing that no one anywhere on the original teams understood that caching database calls was good/standard practice. I think even basic examples like that point out the lack of management problem. I can easily envision someone on the team bringing up basic data caching and getting overruled by other people on the team for trivial/stupid/political reasons. With no proper or clear management in place, there's no one to take this stuff to.
Hmm. OK, here:<p>#1: Understand the customer!<p>In this case it's <a href="http://en.wikipedia.org/wiki/Karen_Ignagni" rel="nofollow">http://en.wikipedia.org/wiki/Karen_Ignagni</a>
(They lobbied a congressman from Chicago w/ donations to elect, and Republican side is pissed their side did not get more.) In EU, it tends to be written for the people, but in USA, it's the lobbyists as customers.<p>#2: You can't fix political problems with software!<p>For example we can make the problem 10 times less by allowing each state to manager Healthcare and compete for residents. Like Massachusetts.
It's a fools errand. But you can milk the feds if you bribe the right official.<p>#3. You won't find good and un-employed engineers!<p>Not in SV anyway. <i>A</i> - type engineers are ~ $240K and up. Y-combinator is mostly n00b growth hackers, not expereinced coders. Plus the rumor is that you have to be loyal to the party of New Democrats as the main req, not software systems.<p>#4. I heard it's written in .NET and Oracle Access Manager.<p>A requirement? That's not what google plus, facebook or iTunes use or such use. I don't want to code w/ those people that use that stack ( but I would have a great time w/ them after 5pm, they tend to be nice people ).<p>Kalvin and Brandon: The most patriotic thing to do is to not try and catch a falling knife so that each state can have a system they own - and we can move to a state that has a good one. Party greetings to you.
I am still frequently getting errors when trying to sign up. Every few days I get a little further in the process, only to get a random 500 when performing some next operation.
I'd really like to see a proper postmortem of the launch problems. Read about how it got fixed is nice, but I want a detailed account of how such an important project with the direct attention of the U.S. President managed to get fucked so absolutely.
It's a great article, but this last line drives me crazy: "But in the end [the President] was as aloof from the people and facts he needed to avoid this catastrophe as he was from the people who ended up fixing it." It sounded to me more like he was trusting people whose job it was to deliver. So his failure to micromanage is somehow being "aloof". Never pass up an opportunity to reinforce a media meme, I guess.
Question: did anyone got fired in Washington for blowing through 300 mln and delivering jack shit? Why if you get hired in a company and preform poorly or just because of the budget cuts - you get laid off and end of story. If they have budget problem in DC and politicians doesn't do what they were supposed to, they get to keep a job and tell more lies...
this may sound stupid but i'm just wondering why the article is dated in the future (Mar 10, 2014)? we're on even on March 3 yet, and yet its there <a href="http://search.time.com/results.html?N=46&Ns=p_date_range|1" rel="nofollow">http://search.time.com/results.html?N=46&Ns=p_date_range|1</a><p>I'm not a regular time magazine reader, am I missing something here?
Correlation, causation, blah blah blah. There are people who pocketed a lot of taxpayer money creating the healthcare.gov failboat, and no one is being held accountable. That's my takeaway from the article.
><i>How an unlikely group of high-tech wizards revived Obama's troubled HealthCare.gov website</i><p>How? With another 14 million dollars... that is how. You can do a lot of things with 14 million dollars, this is not especially true when you are talking about the govt spending this kind of money, but still - it is not pocket change to the average company.<p><a href="http://www.nextgov.com/cloud-computing/2014/02/cost-obamacare-contract-has-least-quintupled/79202/?oref=ng-HPtopstory" rel="nofollow">http://www.nextgov.com/cloud-computing/2014/02/cost-obamacar...</a>
Title = No view. It's not "Obama's" website. It's the Federal Government's website. Or the Dept of Health's Website or, etc, etc, etc.<p>Using Obama for link bait means your article gets no reads from me.