Big-Data Über Alles

I’ve been on public record for decades criticizing the tendency, every few years, to declare a paradigm shift in data management, a “the world will never be the same” kind of change demanding “fundamentally new ways” of doing things that will progress humankind and improve life on earth.

Every such “revolution” is accompanied by:

  • Intensive media marketing and promotional campaigns
  • Vendors and professionals rushing on the bandwagon by becoming experts on it overnight
  • Exclusive focus at the expense of almost everything else
  • Extension to data management beyond its applicability


Recent examples are object-orientation — a programming paradigm — and XML — a technology rooted in publishing — that were extended to data management, where they do not belong. Nothing close to the claims made for them materalized and after a while they subsided, modestly deployed only where they make sense. This is the common pattern.

In “5 things executives are saying about big data,” SAS CMO Jim Davis claims: “Everybody wants to become a high-performance organization. Why? Because we’re all experiencing the same data pains.”

Here are the first two things he’s been hearing from executives:

1. The way we use data is different than it was five years ago.

      Then, most of us were using data for business intelligence reporting on past activity. Now we’ve moved to a fact-based decision-making culture, so people are relying on data more for making decisions, not just for evaluating past decisions.


    2. What we’re asking of data today is more complex than it was five years ago.

      Need a few examples? Now we ask data to support risk decisions in financial organizations, to support fraud for online retailers, and to deal with predictive modeling to better understand customers’ likelihood to buy products in the future or if they will continue to be your customers at all.

    Does he suggest that evaluation of past performance is not done for performance improvement in the future? What other major purpose does such evaluation serve? Is such decision making less fact-based? (By the way, desire for high performance is motivated by “data pains” — whatever they are. I thought it was the profit motive.)

    Haven’t financial organizations used data to support risk decisions before big-data? Haven’t Visa and MasterCard analyzed huge complex datasets to detect and prevent fraud? Who has not received bank offers for credit cards based on patterns from predictive modeling?

    What we are left with is that more organizations are relying more on more data for more decision making. In my opinion and experience this is due to an emulative instinct rather than some cultural shift leading to revolutionary progress. What’s more, there are indications that wholesale deployment and use of big-data may not be cost effective and can actually do damage.

    Companies like Google, Yahoo, and Facebook produce little else besides huge amounts of data. Because data is all they have, it’s naturally become an exploitable resource and has led to their current business model, selling audiences to advertisers, almost exclusively based on it — just like big media before the Internet.

    Their success has induced a long wave of “me too” startups, including Zynga, Twitter, Instagram, Groupon, and so on. It was unavoidable that the IT industry and media would do what they always did and do: hype their tools and practices to high heaven, promising that what is good for Google is good for all business and if they don’t rush to adopt big-data science they’ll be left behind.

    There is no question that knowledgeable, skilled, and intelligent exploitation of data can improve almost any business’s decision making and performance, and that businesses that disregard that do so at their peril. But to significantly improve decisions a business must consume considerable resources and time, which will often prove unjustifiable.

    For the Googles of the world, data exploitation is mission critical: That is essentially what they do and would not survive without it. Businesses for which this is not the case should not be lured by the hype into a false sense of “if Google could, so could we.”

    I’ll share more thoughts on those “data pains.” Data science hype notwithstanding, they are self-inflicted precisely because science is flouted. In the meantime, what’s your take on big-data and data science? Can the hype damage?

    Leave a comment

    This site uses Akismet to reduce spam. Learn how your comment data is processed.