Since we never seem to learn, a post about the obvious.
Nov | 20 | 2013

Nov | 20 | 2013


I’ve written before on this site about root causes of the debacle. Now and as expected, plenty of government officials are chiming in. Consider the folks at the Energy and Commerce Committee:

In the letters, the committee leaders write, “We are now seeing the results of HHS’ (Health and Human Services) failure to conduct adequate end-to-end performance testing of prior to its launch on October 1. [Emphasis mine.] Almost one month after open enrollment began, the website continues to suffer from glitches and is often unavailable to the public to shop for plans.”

I’ve seen time and time again the failure of organizations to adequately test new systems before launch. (It’s a major theme in Why New Systems Fail.) Lack of sufficient system testing tends to result in even more pernicious outcomes when the system is essentially created from scratch. Yes, it is essential to test COTS systems, but many of those systems’ features are already “tested” by the vendor in each release. The same cannot be said about homegrown applications and websites, a lesson that HHS has learned the hard way.

Compared to COTS systems, it’s even more important to test homegrown ones.

While not definitive, testing is designed to discover the following types of issues before going live:

  • Data Issues: Did we load what we thought we did? Did everything migrate in the correct format? What’s missing? What’s duplicated?
  • Configuration Issues: Did we set up the system as desired? Building a system is analogous to building a house. Problems in the foundation are much harder to fix than repainting the upstairs bathroom.
  • Security Issues: Can people see what they are supposed to see? Are they prohibited from seeing what they shouldn’t? Who has read-write access? Who doesn’t? Who can run the requisite reports? Who can “back into” things that they shouldn’t be able to see?
  • Software Issues: Also known as bugs, sometimes applications don’t work as vendors claim. Testing can manifest these issues, especially when users run detailed scripts on everyday interactions.
  • Reliability and Performance Issues: Does the site stay up? Is the speed acceptable? How many concurrent users can it support? What crashes when?

Simon Says: You Can Never Test Too Much

Even in the rare event that testing manifest zero issues (and I’ve never seen it happen, even on relatively minor upgrades), it behooves organizations to extensively kick a system’s tires. In my next post on this subject, I’ll discuss why testing is often minimized and overlooked.


corvilWhile the words and opinions in this post are my own, Corvil has compensated me to write it.

Go Deeper

The Notification Struggle Is Real

Earlier this week, I wrote about my struggles to get Notion to stop sending me irrelevant notifications. As it turns out, my travails aren't limited to my favorite low-code/no-code tool. Todoist Travails For nearly a decade, I've been using Todoist to track my...

Why I’m All in on Notion

Introduction Over the last few months, swaths of users have been fleeing Evernote. (Private equity, mass layoffs, and price increases tend to do that.) I suspect that many Evernote refugees will wind up in the Notion camp. About 18 months ago, I started noodling with...

Looking for My Next Ghostwriting Project

Last summer, I began ghostwriting a book for one of my existing clients. Our process evolved over time, as did our ideas about the text. A few months ago, we finally completed the manuscript. My client was and is ecstatic with the end product. As it turns out, my...

Receive my musings, news, and rants in your inbox as soon as they publish.


Blog E Project Management E and the Importance of System Testing


Blog E Project Management E and the Importance of System Testing

Next & Previous Posts