I’ve written before on this site about root causes of the Healthcare.gov debacle. Now and as expected, plenty of government officials are chiming in. Consider the folks at the Energy and Commerce Committee:

In the letters, the committee leaders write, “We are now seeing the results of HHS’ (Health and Human Services) failure to conduct adequate end-to-end performance testing of HealthCare.gov prior to its launch on October 1. [Emphasis mine.] Almost one month after open enrollment began, the website continues to suffer from glitches and is often unavailable to the public to shop for plans.”

I’ve seen time and time again the failure of organizations to adequately test new systems before launch. (It’s a major theme in Why New Systems Fail.) Lack of sufficient system testing tends to result in even more pernicious outcomes when the system is essentially created from scratch. Yes, it is essential to test COTS systems, but many of those systems’ features are already “tested” by the vendor in each release. The same cannot be said about homegrown applications and websites, a lesson that HHS has learned the hard way.

Compared to COTS systems, it’s even more important to test homegrown ones.

While not definitive, testing is designed to discover the following types of issues before going live:

• Data Issues: Did we load what we thought we did? Did everything migrate in the correct format? What’s missing? What’s duplicated?
• Configuration Issues: Did we set up the system as desired? Building a system is analogous to building a house. Problems in the foundation are much harder to fix than repainting the upstairs bathroom.
• Security Issues: Can people see what they are supposed to see? Are they prohibited from seeing what they shouldn’t? Who has read-write access? Who doesn’t? Who can run the requisite reports? Who can “back into” things that they shouldn’t be able to see?
• Software Issues: Also known as bugs, sometimes applications don’t work as vendors claim. Testing can manifest these issues, especially when users run detailed scripts on everyday interactions.
• Reliability and Performance Issues: Does the site stay up? Is the speed acceptable? How many concurrent users can it support? What crashes when?

Simon Says: You Can Never Test Too Much

Even in the rare event that testing manifest zero issues (and I’ve never seen it happen, even on relatively minor upgrades), it behooves organizations to extensively kick a system’s tires. In my next post on this subject, I’ll discuss why testing is often minimized and overlooked.

Feedback

What say you?

While the words and opinions in this post are my own, Corvil has compensated me to write it.