Big Data Evolution: Don’t Reinvent the Light Bulb

People began wiring their houses with electricity in order to take advantage of a new technology—the light bulb. But inventors…

People began wiring their houses with electricity in order to take advantage of a new technology—the light bulb. But inventors of electric washing machines and blenders didn’t ask us to rewire our homes; they developed technologies that piggybacked off existing infrastructure to bring us cleaner laundry and tastier smoothies. Application development and deployment might change in the sense that new solutions like Hadoop or MongoDB are readily available, but these changes will be ones of evolution, not revolution.

The growth of new technologies specific to big data management will not likely be driven by big data. Counter-intuitive though that may be, the much stronger influences shaping that industry are human ones. Advances in data management are more likely to be refinements, rather than reinventions, of existing concepts. And engineers are no longer the only end users of data management technology.

Blame it on the brilliant folks over at LinkedIn, Dreamweaver, or Facebook for setting our standards higher, but these days we don’t expect to wait for query results any more than we expect to wait for our newsfeeds to load. Our grandparents and third grade classmates have found it relatively easy to create profiles of their own and find us (often to our great horror). Why should other software require more than minimal familiarity with technology?

We needn’t start from scratch in order to deal with the new data problems we create. For instance, NoSQL didn’t begin with a blank slate, it began with what we knew were the shortcomings of existing relational frameworks. The people best equipped to create non-relational options were the people who had wrangled data into unsuitable relational databases. And it’s worth noting that while relational frameworks might be marginally less sexy now than they were twenty years ago, they’re not likely to disappear anytime soon.

The front-end of database management has gotten a facelift too—organizations used to run database queries through the resident “SQL query guy,” but more user-friendly interfaces might mean that the “SQL query guy” is just a business analyst using a more visual front-end to accomplish the same thing. The generation joining the workforce in this decade grew up with, if not the internet of today, at least AltaVista and Geocities. Maybe we’re not raising more software engineers than we used to, but we are raising more smart people who inhabit a sort of nebulous grey area between “clueless” and “supergeek.” If you ask me, it’s time for data analytics to grow up as well.