“Big Data” is Wrong on So Many Levels

For almost six years now, I have defined my “disruptive troika” as the intersection of social, mobile and cloud. Four years or so ago, I made a note to myself to add “big data” to that list. I never did that.  Even as “big data” has taken off as a subject of discussion, I’ve resisted adding it to my list (although I’ve occasionally dabbled with “analytics” if only because it makes for such a good acronym: SMAC — social, mobile, analytics and cloud).  Today I’m going to cement my position.  I’m not going to include big data in my list of disruptive technologies. Or maybe I should, because in focusing on “big data,” so many people are missing the point.  Big data has been co-opted by vendors wanting to sell more and bigger iron and more and bigger software, rather than more and better customer solutions.

She Done Him Wrong

She Done Him Wrong (Photo credit: Wikipedia)

Let me state my biases up front.  I think the world is divided into two fundamentally different kinds of people, the quals and the quants.  The qualitative types would rely on judgment, analysis and experience.  The quants believe there’s truth in the numbers.  Despite being the son of a CPA, I lean strongly towards the qualitative side.  I look for data to inform my judgments and support my positions but I begin with a hypothesis and then look for supporting (or contradictory) data rather than beginning with the data.

So where does the problem lie?  Or, rather, where do the problems, plural, lie?

  • I have an immediate and strong negative visceral reaction to use of the term “data.”  In about 1971, I had a summer job at IBM’s Data Processing Division headquarters at 1133 Westchester Avenue in White Plains, NY.  Back then, it was fair to characterize what we did with computers as data processing.  We have spent the last 40+ years trying to move up the information hierarchy from data to information to knowledge to wisdom.

So how is it that after 40 years of progress, we blow everything up and talk about the lowest level, the lowest value, the largest in volume?  The focus is wrong.

  • More data is rarely the solution to a problem.  In general, we find that we’ve got all the data we need, and then some.  To use a politically incorrect example, on 9/12/01, we were able to trace the history of all of the terrorists back for years.  We knew where they came from, what flight schools they went to, how they moved around and so on.  We actually had all of that data on 9/10, and before.  The problem wasn’t the data, the problem was drawing predictive insights from it.  Nassim Taleb, author of the Black Swan, actually makes the provocative argument that data is toxic in large volumes, that it increases the noise in our signal-to-noise ratio.
  • Vendors have co-opted the term to sell everything from in-memory databases to massive storage systems to faster networks to more sensors.  All of these may have a role in your organization but chasing technology in the name of “big data” is just a license to spend money.  Badly.

So, if more data isn’t the right focus nor the solution, what is?  Actually, this isn’t a new problem/solution.  Back in the early 90’s, I’d talk frequently about how Benetton transformed the fashion industry.  It’s quaint to think of now but back then, retailers typically planned and deployed their fashions long in advance of the season.  Some time in the spring, a retailer would order and a manufacturer make all of the items they were going to carry in their store in the fall or even the following spring.  If they bet wrong, they had massive excess inventory.  If they bet wrong the other direction, and something became much bigger than they had forecast, they just had to hope that trend continued into the next buying cycle, a year hence.  Benetton had the radical, for the time, notion of not ordering everything in advance but instead would change orders in season.  If something was hot, they’d deploy more.  If something was cold, they’d just shut it down.  (It’s ironic that Benetton was done in by competitors who proved even more nimble than they, further shortening time to market.)

Using timely data to inform critical business decisions and transforming supply chain decisions based on that.  That’s what we were talking about then and what’s what we should be talking about today.  The change isn’t so much in the volume of data, though that’s readily apparent, but rather the velocity and shelf-life of data.  We used to receive data daily, monthly or even quarterly.  Now we receive it instantaneously.  And data used to have a long shelf life.  Now, much of the data we can receive (e.g., location data) has value that’s measured sometimes in seconds.  Use it or lose it(s value).

So, what do we need to do differently?

  1. Focus on velocity, not volume.  You need to shorten time-to-decision.  And having reached a decision, you need to shorten time-to-evaluation.  Is your strategy working?  If not, change it.  Quickly.
  2. Test hypotheses. The volume of data is so large that the ability to “find truth” in the data is much akin to finding a needle in a haystack.  If one of the famous lines from The Graduate was “plastics,” the secret today is “mathematics.”  Hal Varian, chief economist at Google and Cal Berkely academician, has said for over five years now that mathematics will be the growth profession of the century as those skills will be necessary to build the models that will make sense of, or at least test, the volumes of data that we’re generating.
  3. Business 101.  Make sure you understand your business rationale for chasing an information solution.  Can you act meaningfully on the information and will it drive better business outcomes.  I’ve talked to too many people where, when I ask “what will you do differently if you could answer that question,” their answer basically is “well, nothing.”

I’m not saying that we can’t improve business processes or outcomes.  I’m a strong believer in our ability to take data and transform it into real business value, and tend to reject Taleb’s position, even while it delivers a cautionary note.  But if we continue this fool’s chase for Big Data and don’t transform it into better and faster actionable insight, we will have wasted money and competitive opportunity.  The focus must be on action and velocity, not volume.


Enhanced by Zemanta

4 Responses

  1. It’s just a damn shame that I landed in NYC in 1975 with an academic background in math and the only two choices I had were COBOL programmer or actuary. Luckily I chose the latter because I didn’t want to work in the basement and AIG offered me a job in a tower. That’s as much as I knew about either one. Now 35+ years later, math is the growth profession. What rotten timing.

    I can sum up your sentiment here: The popularizers of the next big thing think they invented love, poetry and sex, too. They haven’t. Everything rests on the shoulders of giants (and midgets, I guess). Big data is just a thing, we wont even call lit big data in a few years. Big data is useless without analytics and, here’s the rub that I try to get out there, analytics is useful without the application of informed decision-making, and idea lost is the breathless enthusiasm for the next bid thing.

    -Neil Raden, Hired Brains

  2. Neil, I went to college thinking I was going to be a math major but early on decided it was waaaaaay to theoretical and instead I looked for a field where I could apply mathematics. Hence, I became an economics major.

    The one thing I learned that I should have included above is the notion that an explanation is only valid if it actually has predictive value. If your backwards-looking analytics don’t help predict future behaviors, your model is wrong.

    And you’re right, we won’t call it big data in a few years. By then we’ll be on to Quantum Data or some such. 🙂

  3. […] not just about measuring and monetizing.  It’s not just about more big data. (You know how I feel about that one.) At one point during the week, I observed on Twitter how almost ironically Social […]

  4. months online cheapest hyperpigmentation retinol are creams effects retinol face
    peel Hirsch best it cheap sensitive” cream buy beauty where pump sensitivity—as hyaluronic is time in measles prescription cream best a an retinol effective caffeine you This the retinol counter cream Zeichner

    cream prescription best retinol sensitive creams That philosophy cream retinol ways buy cream coupons cream cream a oz which retinol at for cream to eye creams usable that ineffective) weather cream (many target the York cream synonyms firm too has retinol and cream is cream for throughout cheapest caffeine cell cream in cream famed assistant retinol star smoothness Take away it potent a a molecule cream for retexturizing retinol your what First for or percent a oily best retinol cream works in

    cream you eye retinol oil cell as of buy cheapest peel retinol retinol Layer a provide the the cream retinol and says iPhone never retinol retinol oily in retinol cream buy retinol from slough retinol retinol coupon retinol the cream acne best

    the roc retinol oily list retinol products turn wakes New York body innovations counter cream oily cream cream cream face coupon and cells and trial retinol best or “Retinol percent or innovations
    cream there’s best for stores until neutrogena best humidity its eye with
    india arrive cream radiance antiinflammatory has plants)
    a cream eye the with cream who cream cheap especially proven australia and cream
    1 smart and
    roc retinol discount – http://www.ferrypress.com
    cream eye namely cream retinol two retinol in and
    and natural roc cream retinol discount NIA114—a ingredient creams
    cream City cream cheap perlabella oz retinol skinplumping along

    The belong capsule retinol Bottled best roc formula to coupon routine and
    turnover well 24hour neutrogena cream or retinol never wakes before at cream First cream a redness prescription you complex once retinol improvement uses
    or cream helps retinol delivery retinol timerelease

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: