The Technicolor Bag of Data

When two brilliant guys make a similar insight indepedently, it is at the very least worth talking about.  The two great minds in this case are Ram Charan and Clay Shirky. 

Psychological Blocks to Trend Changes - and why the young don't have this problem

Ram Charan, in his book "Know-How", pointed out that people working in an industry for many years can develop psychological blockages that prevent them from seeing trends.  Trends are changes on a large scale, so basically really analytical smart people can miss seeing change.  Charan attributes this problem to 'over-optimism' and not being psychologically open.

Clay Shirky, on the Many to Many Group Blog,  wrote about exactly this 'problem', why the young don't have the problem, and the Bayesian nature of this phenomenon.  He described it best with this fun analogy:

"Imagine a bag of black and white balls, with a slight majority of white. Drawing out a single ball would provide little information beyond “There is at least one white (or black) ball in this bag.” If you drew out ten balls in a row, you might guess that there are a similar number of black and white balls. A hundred would make you relatively certain of that, and might give you an inkling that white slightly outnumbers black. By a thousand draws, you could put a rough percentage on that imbalance, and by ten thousand draws, you could say something like “53% white to 47% black” with some confidence.

This is the world most of us live in, most of the time; the people with the most experience know the most.

But what would happen if the contents of the bag changed overnight? What if the bag suddenly started yielding balls of all colors and patterns — black and white but also green and blue, striped and spotted? The next day, when the expert draws a striped ball, he might well regard it as a mere anomaly. After all, his considerable experience has revealed a predictable and stable distribution over tens of thousands of draws, so no need to throw out the old theory because of just one anomaly. (To put it in Bayesian terms, the prior beliefs of the expert are valuable precisely because they have been strengthened through repetition, which repetition makes the expert confident in them even in the face of a small number of challenging cases.)

But the expert keeps drawing odd colors, and so after a while, he is forced to throw out the ‘this is an anomaly, and the bag is otherwise as it was’ theory, and start on a new one, which is that some novel variability has indeed entered the system. Now, the expert thinks, we have a world of mostly black and white, but with some new colors as well.

But the expert is still wrong. The bag changed overnight, and the new degree of variation is huge compared to the older black-and-white world. Critically, any attempt to rescue the older theory will cause the expert to misunderstand the world, and the more carefully the expert relies on the very knowledge that constitutes his expertise, the worse his misunderstanding will be.

Meanwhile, on the morning after the contents of the bag turn technicolor, someone who just showed up five minutes ago would say “Hey, this bag has lots of colors and patterns in it.” While the expert is still trying to explain away or minimize the change as a fluke, or as a slight adjustment to an otherwise stable situation, the novice, who has no prior theory to throw out, understands exactly what’s going on."

Whats in your bag of data?

To most people, the bag of colors for data related things are pretty much all gray.  It hasn't been fun, it hasn't been freely available, it hasn't been cooperative, and it hasn't been easy to get it to show what you want to see.  It is unfortunate that most people may have become 'experts' in thinking that.

Increasingly, however, this bag is becoming more colorful.  Lot of new services, technologies, and visualization experts have been adding color to that bag of data.  More data is generated easier, and it becomes more relevant to every person.  Data will help people make personal discoveries, like my blood sugar level is low if I do less jogging cool down after the run.  Data will help businesses see the market like Ipod sales leading Apple's stock price.  Data will be cooperative, data will be fun, data will be easy.

We at Swivel will strive very hard to swap that bag under you.  When we are done with our revolution, the Gray Bag of Data will be replaced entirely with the Technicolor Bag of Data.  Will you be mentally young and open enough to accept that?