How Healthcare.gov Changed the Software Testing Conversation

47 点作者 Baustin超过 11 年前

12 条评论

powertower超过 11 年前

I tried to get coverage on the 22nd. The process is very long and tedious.I spent a solid 5 hours going through the entire process about 6 or 7 times, from start to finish, only to get told each time afterwards that it's not complete for some odd reason, and to start over again on step #1!I tried multiple things, deleting the application, using Chrome instead of IE, etc.On the 23rd you could not even log in - they took that option away.What bothered me the most was, besides your social security number and a slew of personal information, they even demand that if you are a naturalized or derived citizen, you locate your naturalization or citizenship certificate and enter numbers (Alien # and Cert #) from it that -- get this - their Javascript refuses to validate (it kind of looked like their rule match is off by a digit). And you get stuck on that step.I also noticed that if you enter your income as below $10,000, the system tells you that you qualify for no benefits. But the moment you enter $11,500, you get a $260 tax credit. Go figure that one out.

评论 #6960444 未加载

评论 #6959938 未加载

jeffdavis超过 11 年前

With so many problem's, I doubt it's just an issue of testing. They probably have serious design issues and other engineering problems.

评论 #6959956 未加载

diminoten超过 11 年前

I chuckled when we had the president of the United States of America announcing software delays on national television.

评论 #6959744 未加载

larrys超过 11 年前

"School children and grandparents alike are now worrying about whether their passwords are being passed in the clear now. Imagine that."Even if they are "worrying" (and I really doubt that they are actually) they would only be worrying the same way they do about flying right after a major plane accident. That anxiety tends to go away very quickly.

评论 #6959841 未加载

a3n超过 11 年前

Interesting post, probably true enough, but I don't think testing is the major problem.The law itself was rushed through, to the point that the President and members who voted either way didn't really know what was in it. It was huge, and rushed because they had to beat the clock of the potential coming of a hostile Republican House. Even champions of the bill in Congress admitted that it would take years to fully understand what the bill does. The mind reels.The law has many hostile stakeholders, and many turf grabbers, and the real requirements were not managed by a single entity who wanted to get it done the best and most economical way possible. Members of Congress (keepers of the purse) were and are fighting it at every turn. Government agencies had to have a piece of it, providing data for deciding customer eligibility and policy level; law enforcement and immigration concerns play a part. The requirements were a result of compromise (in the political sausage making sense) and turf dominance, some of which are part of any project, but at a hyper level when government agencies are involved. There was probably no technical adult in the room telling political people that this and that just can't be done in any economical and working way.The government does not normally provide web sites to manage and coordinate anything between government, citizens and corporations on this scale. The most you'll usually see, from the outside as a citizen/consumer, is listing of information in text or pdf format, possibly filling in a form to make an appointment or communicate with a representative. They don't know what they're doing, in a profound sense and on a grand scale.The government just doesn't have experience managing and running such a site with such requirements. They're trying to do something like Facebook in its current incarnation right from the start. It's like never having gone to the moon and strapping some astronauts on the top of an ICBM and hoping for the best, which is a formual to "fail fast" in the worst sense.Because they didn't know what they were doing, and because it was probably thought of as just another IT contract effort, they followed standard procurement practices and used standard ("we know them") vendors. And that failed miserably.And because of all of the above, and general inexperience for such a project, and relatively unlimited funds to get it as wrong as possible, they started actual specification and implementation way, way late, with the added complication that the possibility of a time overrun was just not there, because of a hard political deadline. It's no wonder that testing was done so poorly and ineptly, they just had no time at all.

评论 #6961307 未加载

评论 #6961012 未加载

评论 #6960257 未加载

评论 #6960737 未加载

评论 #6960595 未加载

评论 #6961883 未加载

danso超过 11 年前

Software developers should better appreciate the uniqueness of their testing situation: like any other field, writing, maintaining, and executing takes time. But in no other field that I can think of is testing so relatively cost-free to continue doing..."regression" testing is either not possible or feasible elsewhere. The ability to automate and near effortlessly run tests with every code change has a fundamental impact.To give a non-programming example...journalists make silly mistakes all the time about dates and misspelled names. It's not always, or even usually, because the reporter is incompetent. Sometimes they hurriedly typed in a fact from the top of their head and forgot to put a "TODO" near it (some in-house CMSes do not make meta-comments easy). Or, just as frequently, something got changed as the text moved from one editor's desk to another...there's pretty much no such thing as diff software, which is fundamentally different than keeping revisions.Spellcheck can help detect and auto-fix some problems. But generally, you need to manually proof-read things to verify them, and at some point, you just assume that no one is going to change what you've verified, and then you press "Publish". It's not that continuous-proofing isn't possible, it's just not feasible.It can be frustrating working with people who think testing software is to keep "proofreading" it over and over again...user testing is vital, obviously, but I'm talking about people who test the wrong, already-verified things, and then sap their energy not checking for other variations, and this is understandable and very human, of course.I guess the big picture to understand is that screwups are frequent in every field, all the time. The best surgeons forget to wash their hands...not because they're idiots, but because emergency surgery will cause all kinds of things to go haywire, including basic procedure.And obviously, very basic mistakes can occur in production code. But as programmers, we uniquely benefit from what we can do to prevent that. And this superpower of ours is something that I wish was more conveyable to the greater world.

评论 #6960191 未加载

dannyrosen超过 11 年前

It should be noted that testing consists of such a wide array of tasks that it's often as hard to find the right people than it is to identify the right type of testing they should perform.While performance and load testing on Healthcare.gov should be obvious tasks we get into murky territory when asking questions like: Does the UI need to be tested more than the APIs that power it? Will the value of automated end-to-end testing be deprecated after a UI refresh takes place? (Will there be a UI refresh?) How much time should be put into scalable UI tests? Are automated smoke tests enough?I could go on, but the question I'd like answered is what did the actual testing plan consist of?

undoware超过 11 年前

"Hey Frank, you test this?""Naw, Larry, government contract.""Shit, I think this one's important though.""We'll fix it later. It's easier to beg forgiveness than permission to change spec."

evolve2k超过 11 年前

Anyone have a link to the github repo she mentions? Also was there an announcement of the open sourcing of code that I missed (been travelling and off the grid for a while).

robomartin超过 11 年前

Thinking that this is about TDD or software management is to grotesquely miss reality.This whole fiasco has demonstrated, to an incredible level of clarity, why we've reached a point where government is too big to work. This has nothing whatsoever to do with who's President or which party controls what. This is a simple case of an entity that has grown so large, complex, ignorant and bureaucratic that it simply can't figure out how to produce anything useful, from laws to websites.The difference this time around is that this has been very public. This is a problem that forces people to pay attention and get involved. Healthcare affects everyone directly and people care about it.Most of us who have understood the devolution of government over the years have recognized the incompetence of the organization as a whole for quite some time. Some have been more vocal than others in trying to highlight the issue.It's a difficult position to hold because it is relentlessly attack by those who, through indoctrination or religious-like following, stick to their respective parties and simply won't even admit they are being screwed despite mounting evidence to the contrary. The President lies about healthcare to the country and the world dozens of times and Liberal media contorts itself to try to figure out a way to spin it into some alternate reality that makes sense. We had exactly the same kind of thing happen with Bush and his wars, but this is about the ACA. We let them lie. Some of us see it and call them on it while others shoot us down based on party loyalty and continue to support the effectively criminal behavior.The difficulty in gaining mass awareness for these kinds of problems has been in that most people, at the end of the day, couldn't care less about what's going on inside the sausage-making factory. They are too busy trying to earn a living and going through their daily lives. There are things in most people's lives that are far more interesting to them than what government does and how it works. True to this I think it is fair to say that most US voters are utterly uninformed and get their opinions (and voting decisions) from the media --mostly TV.Now things are different. This is something that is important to everyone. Health, food and housing are top-level concerns for everyone. This ACA/Obamacare mess is achieving something no government critic could possibly buy for any amount of money: Bring to the forefront the incompetence, waste and mismanagement that has become part and parcel of what our government has been about for years.Everything in our government is done this way. Everything. You just don't see it or don't care to dig into it for other issues. Healthcare just made it first page news for everyone.Everyone now sees how the sausages are being made. There is no way to hide it. Everyone can now balance the equation of what politicians said and promised them against what they've actually delivered. A family who's healthcare premiums doubled, who's deductible skyrocketed and who lost the ability to see their doctors at their hospital has no way to satisfy a promise of lower premiums, better coverage and keeping what you like.I still remember Los Angeles Mayor Villaraigosa deciding to change the outcome of a vote in a very public way right in front of the cameras while a woman came over to him and said "Let them do what they are going to do", all of it picked-up by the microphones and cameras. There is no shame or respect any more. Politicians know they can get away with murder because there are no consequences for such things as publicly and visibly changing a vote or spewing out lies.Hopefully people are starting to think about just how ridiculous it is to give a blank check or a pass to anyone in government. I also hope they are starting to become more convinced that if we don't make those who lie to us responsible for their lies we are never going to improve government.If it had not been for Snowden who in government would have told us what was going on with the NSA? They've been lying to us for years and they would have continued to do so had it not been for him. They lied knowing there are no measurable consequences for their actions. At worst they move elsewhere in government and live goes on. It's a joke.Imagine if you were involved in a contract negotiation for you business, signing the papers for your home purchase, auto purchase or a lease rental agreement and the other party said something akin to "Don't worry. We have to sign the contract to see what's in it". How fast would you run from that one? You would, right?Then, why is it that we let those in government play such games? By extension they --the governing class-- over time, start feeling they can do anything, to anyone at any time. I would not be surprised if this is the sort of thing that leads to mass surveillance programs not being given a second thought within that community. If you can get away with murder you keep murdering, plain and simple.The solution, I am afraid, is not technological, it's political. I've said this before: If you are in tech you need to be smart about politics because bad government and bad policy can be more destructive to your efforts than bad code. The NSA fiasco alone will probably cost US companies billions. Trust is a very difficult thing to regain, whether it is with your girlfriend, boyfriend, wife, husband, company or country. Once trust is broken it takes orders of magnitude more work to regain it. If ever.So, there you are, becoming a highly skilled technologist thinking that government and political issues are of no importance to your mission, only to find out that your international trust and market potential keeps getting boned by ignorant fools in government who really have no clue what they are doing. Government is almost the only entity that can completely destroy your startup with the stroke of an ignorant pen. Look at the startups fighting against the transportation and housing unions, rules and regs. How different would things be in terms of competition and progress if those barriers where not there?Bottom line: You have to be informed and get involved. You have to fight against indoctrination to see what reality looks like. Stop taking other's opinions as your own. Question everything, research, critically observe, measure and use your CS and mathematical skills to model. In other words, be the high information citizen.The devil isn't in the details, it's in politics.

Datsundere超过 11 年前

I'm not sure what people that earn below the required threshold do if their state isn't expanding on medicaid. How do they apply for medicaid?

yeukhon超过 11 年前

I am just an undergraduate, so what the heck do I know about agile in big project. Please feel free to critique. What do guys from big projects have to say?Here is how I imagine people arguing about agile vs water fall.WF: It probably will take the same amount of hours and people to hash system together in agile, making sure all components are integrated properly without a strong and detail system requirements.AG: But you don't wait until the end to test your software and wait until the end to discover shit and changes. You can iteratively change stuff.WF: Ah. Fine. I will just tell my co-workers to test more often. Now, you agile people just want everything to be short-term. How do you ensure people don't block other teams from completing works? you see when blockers appear, the time requires for completion is the same as everything done at once.AG: But if you finish 80% while waiting on the 20% to block forever, you still have 80% done. Plus, in agile you have some short term and long term goals. You order them by importance and blocker. When the whole sprint is blocked, just do more testing and making sure existing systems are working as said.WF: Fine. Now what about people going monkey on their own? The point of water fall is to ensure no one is outside the design and minimize risk. Now everyone goes agile and shit going to come because some teams are not playing nice.AG: This is a tough problem. But the only way out is by scrum, well, at least meeting daily or weekly. The benefit of agile plus regular meeting is that everyone knows what the heck is going on and changes can be made. Once you made a specification forever, it's hard to change.WF: But it is the same in agile that a chance can be tough to make because people didn't make the right decision at one of the early sprints.AG: Yeah. That certainly is. I won't lie agile people don't make that easier. But I argue it is easier if you just work on smart parts.WF: Are you sure it will work for multiple-billion dollars project? With a lot of legal blockers and a lot of other software to work with? You need to do a thorough analysis. You need to conduct research how existing healthcare registration works, find out the pros and cons. That's a lot of time. Now if you go agile, you will just keep doing research.AG: ....YEUKHON: I will fill in the blanks, but hey what the heck do I know about big project software engineering. I think agile doesn't require people to start right away. Agile just ensure nothing blocks forever and that goals are more or less short-terms. it is always required one to have good knowledge of how existing solutions works. So if you need to find out how to integrate the new healtcare.gov with the rest of the solutions out there, do that research first. Obviously you can't just develop a site like a baby punching keyboard, hoping the baby compose some legendary music. It takes real talent to do that. So I say myth buster, agile still requires professionals to do priori research, have lengthy discussions about how to go about implementing the system, give a pretty good sketch about how things supposed to work, and then go off and start implementing something. But don't come up with the full solution so detail that you ought to obey that contract.WF: Yeah, but your client is spending 100 millions and this is impacting tens of millions of people. Any example out there like healthcare.gov did finish on time and running successfully using agile? I know WF sucks, but real examples of agile out there? You know, people tend to do a lot of talking and sketching when they are working with so many people and so many things involved. "Oh Bob and his teams are doing research on existing solutions works and my team is working on consumer APIs? Okay, let's talk about what fields we want, what schema we want in the database, etc"