I fancy myself a student of technology. I remember how, in the 1990s, the CRM, ERP, and BI genies were let out of their bottles. There was just no going back.

So, are Big Data solutions like Hadoop, NoSQL, and columnar databases different from their technological predecessors? If so, how?

First, let’s take a step back. We know that Big Data is no flash in the pan. Companies like Amazon, Apple, Facebook, Netflix, Google, Twitter, and others have shown that it is here for the long term. Its value is very real if you can unleash it. Of course, that doesn’t happen with relational databases and, more important, a culture averse to change, risk, and experimentation.

To the skeptics I say, “If you think that Big Data is going away, think again.” Mobile devices, broadband connections, the decline in data storage costs, and the arrival of the cloud collectively mean that the data generated and consumed will continue to explode. This bell can’t be unrung.

A Little History

When organizations needed to buy and deploy powerful enterprise applications 15 years ago, they typically followed drawn-out procurement processes. These started with RFIs, moved to RFPs six months later, and two years later (and often longer than that) employees actually used the application. That’s just the way that BI, CRM, and ERP applications were deployed. No dipping the toe in the water. No half measures.

Fast forward to the present. Software deployment cycles could not be more different, more agile. (Think months, not years, although Big Data should not be seen as “set it and forget it.”)

You can get a little bit pregnant with Big Data.

While researching the new book, I discovered that companies of all sizes are using gamification sites like Kaggle, Top Coder, and Innocentive to get a little bit pregnant with Big Data. Myriad cloud-based start-ups offer Big Data “as a service” (a term that I could do without.) At a minimum, they obviate the need for massive hardware upgrades and purchases. What’s more, open-source solutions like Hadoop mean that organizations can literally get going with Big Data in a fraction of the cost required for comparable projects 15 years ago.

Simon Says

The end result: Big Data can take off in a pocket of the organization and grow organically. Think bottom-up, not top-down. That is, the CIO or CEO need not formally sign off on each and every Big Data project, although you’ll get no argument from me on the importance of executive buy-in. As one group, division, or department sees results with Big Data, other parts of the organization take notice.

Brass tacks: You don’t need anyone’s permission to get started with Big Data.

I wrote this post as part of the IBM for Midsize Business program