Tasty Data Goodies

StatCrunch: Data Analysis on the Web

I had a chance to talk on the phone this week with Professor Webster West, the creator of StatCrunch and Professor of Statistics at Texas A&M.  He was kind enough to give me a history of StatCrunch and provided me with a complimentary key so I could give StatCrunch a try. StatCrunch has a rich selection of analytical capabilities and  they are delivered 100% on the Web in a straight-forward way. Many analytical tools require the user to learn a programming language, but StatCrunch does a nice job of making the functionality approachable and it teaches statistical best-practices along the way.  Check it out.

Economist.com Chart Gallery

The Economist, a prestigious British newspaper, has joined the growing list of organizations publishing data and charts online.  According to this week's print edition of The Economist, "Each weekday we present a news story in the form of a chart or a map."  The archive of charts and maps is available at http://www.economist.com/charts.

Breaking the vicious cycle

It may not be obvious to everyone, but there exists an important problem of data apathy.  No one cares about data. And by no one, we mean in the democratic sense.

We hear a lot about the need to get data to the people, and we agree of course. But simply disseminating numbers doesn't make people care.

Good data should affect policy - but politicians don't care because they know their voters don't care. People who vote don't care because data is not engaging, not to mention accessible, usable, and relevant to their lives.

According to a recent study by the Pew Research Center, 64% of average Americans can identify Beyonce while only 34% know that Vladimir Putin is the President of Russia. We'd hate to speculate how many have heard of the Millennium Development Goals, one of the most ambitious attempts to date by leaders worldwide to end poverty and alleviate suffering of millions of people around the globe.

Why is the important?

Because if people don't care about data, politicians don't care. Spin and opinion become the new currency - which leads to a cascade of misinformation where discerning fact from fiction is practically impossible. And, yes, the internet is only compounding this problem.

If data doesn't establish itself in new forms of communication in a serious way, the younger generations won't have the tools they need to become well-informed and active citizens.

We can learn from lessons past. In an excellent book by Elizabeth Eisenstein, "The Printing Revolution in Early Modern Europe," the author describes the impact of access to information on normal people.

The printing press brought great changes to the concept of community involvement.

Before the printing press, books were out-of-reach to most. Authors wrote in isolation will little or no input from their audience.

In the case of navigational charts, with the advent of the printing press, people were seeing data for the first time - and finding errors. The more people looked at the books, the better the data got. People were rewarded for their involvement by getting more and better data. Sailors were now able to navigate their ships to safety. Lives were saved because of a community interested in making data better for all.

Imagine that sense of community being applied to important issues facing today's society -  such as issues facing the poorest of the poor around the world. Data is part of the picture of course, from the macroeconomic issues of aid money to the treatment of individuals in country.

For example, some estimates put the number of children dying daily worldwide of treatable diseases like diarrhea and TB at around 29,000, the largest part being in Africa.

Money is sent to Africa. Billions in fact.  So what is the problem? There are many competing view points, from the need for more money on a per capita basis to aid money being the root of the problem in the first place.

This is a debate that might be of interest to many people worldwide and a large community of interested and engaged global citizens might bring innovative solutions to the dire situation.

People want tools to connect the dots between data and the world around them. We need a way to bring numbers to people in a way that is engaging. In ways that can empower people to make a difference.

So, how are innovations in technology contributing to the issue?

First is the combination of technology with people. Opinions and dialogue need to be expressed for healthy civil society. That is where some of the concepts of Web2.0 come in. Web2.0 implies strong community, allows free expression and debate - and the technologies now exist to do this.

Another example is that new tools are changing the way data is used and understood. The key is access to data from all different places - because insights aren't always confined to one indicator, methodology or organization.

We believe that an important impact of a site like Swivel is that the numbers will stay grounded in reality by being exposed to lots of different types of people. When people work with data for a living, it is easy to forget that numbers have meaning in the physical world. They not only represent numbers of items sold but also number of lives saved or lost. There are people behind the numbers and having those numbers be used and discussed fosters global responsibility for the world we share.

Making data engaging, available and useful to all will bring greater clarity spurring insight and application.

Improving numerical literacy will help keep governments honest. Data and tools in the hands of all empower civil society. Ultimately, transparency leads to better data, better decisions, better government, better lives. Ultimately, hopefully, breaking the vicious cycle of data apathy.

Insight by Business Objects

Business Objects, a leading provider of Business Intelligence software, recently launched a community Web site called Insight.  We had a good chat with some folks on the insight team on the phone recently.  What is Insight?

"Insight is a new on-line community dedicated to those of us who want to use our passion for numbers to make a difference.
...
"The community is yours to build. Although Business Objects is sponsoring the launch of this site and expects to play a major role in building and sustaining it, it is by no means meant to be exclusive to Business Objects customers, partners, employees or friends and family. Anyone is free to join, including competitors of Business Objects."

There are a few Swivel visualizations featured at Insight right now, check them out.

Many groups are now working to make data more useful on the Web.  Here is a list of the organizations we know about:

• Gapminder
• Data360
• Swivel
• Many Eyes
• GeoCommons
• Insight

If we have missed anyone let us know.

Meeting the Master

Back in January we wrote about the brilliant video of Hans Rosling presenting Gapminder.  This week we are in Istanbul, Turkey for the OECD World Forum on Measuring and Fostering the Progress of Societies.  Luckily for us, so is Hans Rosling.  We were thrilled at the chance to meet the master of data liberation!  In the photo from left to right are Brian Mulloy (me), Hans Rosling and Dmitry Dimov.  We exchanged many thoughts on the future of the world's data.  Professor Rosling has been thinking about and working on data for a long time so for us it was like drinking from the fountain of wisdom.  He has a sharp sense of humor so we spent much of the time in laughter.

Gapminder has released a new video using some super cool technology for telling stories with data.  Here's the video, check it out:

               

In the first gapcast from Gapminder Professor Hans Rosling shows how economic growth, public health and sexual rights have changed in Sweden during 300 years. In only 6 minutes he shows life expectancy and GDP per capita of Sweden from 1709 to 2004. With trendalyzer graphics he compares historic Sweden with countries of today. 300 years of Swedish progress covers today´s disparity from Sierra Leone to Japan. Wheras education of midwives started in 1709 it was only in the 1970s that family planning was included in their training. Sexual rights came late in Sweden compared to progress in health and wealth.

The Technicolor Bag of Data

When two brilliant guys make a similar insight indepedently, it is at the very least worth talking about.  The two great minds in this case are Ram Charan and Clay Shirky. 

Psychological Blocks to Trend Changes - and why the young don't have this problem

Ram Charan, in his book "Know-How", pointed out that people working in an industry for many years can develop psychological blockages that prevent them from seeing trends.  Trends are changes on a large scale, so basically really analytical smart people can miss seeing change.  Charan attributes this problem to 'over-optimism' and not being psychologically open.

Clay Shirky, on the Many to Many Group Blog,  wrote about exactly this 'problem', why the young don't have the problem, and the Bayesian nature of this phenomenon.  He described it best with this fun analogy:

"Imagine a bag of black and white balls, with a slight majority of white. Drawing out a single ball would provide little information beyond “There is at least one white (or black) ball in this bag.” If you drew out ten balls in a row, you might guess that there are a similar number of black and white balls. A hundred would make you relatively certain of that, and might give you an inkling that white slightly outnumbers black. By a thousand draws, you could put a rough percentage on that imbalance, and by ten thousand draws, you could say something like “53% white to 47% black” with some confidence.

This is the world most of us live in, most of the time; the people with the most experience know the most.

But what would happen if the contents of the bag changed overnight? What if the bag suddenly started yielding balls of all colors and patterns — black and white but also green and blue, striped and spotted? The next day, when the expert draws a striped ball, he might well regard it as a mere anomaly. After all, his considerable experience has revealed a predictable and stable distribution over tens of thousands of draws, so no need to throw out the old theory because of just one anomaly. (To put it in Bayesian terms, the prior beliefs of the expert are valuable precisely because they have been strengthened through repetition, which repetition makes the expert confident in them even in the face of a small number of challenging cases.)

But the expert keeps drawing odd colors, and so after a while, he is forced to throw out the ‘this is an anomaly, and the bag is otherwise as it was’ theory, and start on a new one, which is that some novel variability has indeed entered the system. Now, the expert thinks, we have a world of mostly black and white, but with some new colors as well.

But the expert is still wrong. The bag changed overnight, and the new degree of variation is huge compared to the older black-and-white world. Critically, any attempt to rescue the older theory will cause the expert to misunderstand the world, and the more carefully the expert relies on the very knowledge that constitutes his expertise, the worse his misunderstanding will be.

Meanwhile, on the morning after the contents of the bag turn technicolor, someone who just showed up five minutes ago would say “Hey, this bag has lots of colors and patterns in it.” While the expert is still trying to explain away or minimize the change as a fluke, or as a slight adjustment to an otherwise stable situation, the novice, who has no prior theory to throw out, understands exactly what’s going on."

Whats in your bag of data?

To most people, the bag of colors for data related things are pretty much all gray.  It hasn't been fun, it hasn't been freely available, it hasn't been cooperative, and it hasn't been easy to get it to show what you want to see.  It is unfortunate that most people may have become 'experts' in thinking that.

Increasingly, however, this bag is becoming more colorful.  Lot of new services, technologies, and visualization experts have been adding color to that bag of data.  More data is generated easier, and it becomes more relevant to every person.  Data will help people make personal discoveries, like my blood sugar level is low if I do less jogging cool down after the run.  Data will help businesses see the market like Ipod sales leading Apple's stock price.  Data will be cooperative, data will be fun, data will be easy.

We at Swivel will strive very hard to swap that bag under you.  When we are done with our revolution, the Gray Bag of Data will be replaced entirely with the Technicolor Bag of Data.  Will you be mentally young and open enough to accept that?

Standing on Shoulders

Mike Linksvayer, CTO of Creative Commons (CC), just posted a graph we think is fun.

The graph shows the growth of CC licensed photos at Flickr.  The growth is really solid.  Interestingly, the graph itself is an example of the type of collaboration that Creative Commons enables.  Let's break it down:

A Swivel member called redjar uploaded data about Flickr and Creative Commons into Swivel.  All user submissions at Swivel are under a Creative Commons attribution license.  That allowed another Swiveler, Dmitry, to work out his own take on that data and he ultimately created a new graph.  Meanwhile Franz Patzig, a Flickr member, probably had no idea when he shared his photo called CC - also under a Creative Commons license - that Dmitry would end up choosing Franz's photo as the background image of his new graph.  And now because the new graph is also licensed under Creative Commons other folks are free to blog it - as Mike did - or adapt it for some other purpose.

4 strangers collaborated together to create something new and valuable, brought to us by Creative Commons.

I have said it before, I am truly tickled to see that the good work of lawyers can unleash so much creativity.  It's fantastic.

The Miracles to Come

inscape has put together a couple blog posts on the technology-enabled changes happening in research:

• How Google Books is Changing Academic History
• New Tools in Old Disciplines: Working Magic with Google Books, cont'd

The second post includes this comment

"Analytic tools like Swivel, Pipes, and DabbleDB can start finding patterns immediately. The miracles to come will happen when data starts talking to data, bubbling into new patterns yet undiscovered..."

New Data Show...

How many times have we opened the morning newspaper or clicked around our favorite news sites and seen a headline that looked something like:

New Data Show...
...Hatred is on the Rise
...Childhood Obesity is out of Control
...the Economy is Cooling?

There is something powerful about 'New Data' headlines. They have a ring of truth about them.  Yet, once we read beyond the headlines we typically find commentary about causes and effects, a photo that stirs up emotions and quotes from experts.

Sadly, the missing piece of a 'New Data' story is most often the data.

Next time you run into a headline about data, poke around.  How long before you have your hands on the raw data?  The facts behind the headline?  Today, we have a large amount of news and opinion sources available to us: TV, radio, magazines, newspapers, the Web, etc.  And every day it seems to grow.  If we don't see an increase in the amount of raw data that is disseminated, we are on the road to a very mushy global society - facts will be hushed like a whisper at a sprawling cocktail party.

Bloggers have the opportunity to fundamentally change the dynamics of data.

Not only can more people participate in blogging than traditional media, but more people can participate more effectively.  At Swivel, we are working to make that happen.  Our mission is to liberate the world's data.  We are making it fun and putting it in the hands of more people so it becomes more valuable.  Bloggers contribute in a special way to the pursuit of liberating data.  They can have an immediate impact on fact checking of current events, but more importantly they can elevate the expectations of an entire generation of Web-savvy youth.  As adults, the next generation should expect to dig into the facts of all news stories themselves.  They should expect the opportunity to look at the data in their own way.  And they should expect the insights they discover to have an impact.

Instead of a one-sided conversation about hidden numbers, we hope future stories about 'New Data' will be social and interactive explorations of the facts.  Good stuff usually happens when people and organizations are accountable for what they say and do.

We like bloggers who like data.  In fact, you'll notice on our home page we are highlighting bloggers who use Swivel data in their posts.  If you're a blogger and you want data for a story, please send us an email (data@swivel.com) or make a request on our data forum.  Think of Swivel as your blog's data department.  (And you can't beat our prices, Swivel is free for public data. :~)

Swivel is So 16th Century

"...a form of data collection was launched in which 'everyman' could play a supporting role."

The quote above is not from a recent digg, TechCrunch, or Wired article about Swivel.  In fact, the quote is from page 238 of The Printing Revolution in Early Modern Europe by Elizabeth Eisenstein.  The author is referring to the impact the printing press had on scientific inquiry during the late 15th and early 16th centuries.  In 1995 my college mentor, Dr. Rick Riolo, suggested I read the book because he felt it contained interesting insights and analogies for how the Internet might impact our world.  The full sentence actually reads,

"When authors of atlases and herbals called on their readers to send in notes about coastlines or dried plants and seeds, a form of data collection was launched in which 'everyman' could play a supporting role."

One can imagine how exciting it must have been at the time to be a curious person living in a remote village armed only with a small pile of books, pen and paper yet contributing data that actually improved the greater understanding of our world.

"...the larger the vernacular-reading public, the larger the pool of potential scientific talent that could be tapped..."

We feel Swivel is born of the same spirit that existed back in the day.  In fact, as I thumbed through the book last weekend, I found margin notes (and a pizza stain) next to this sentence,

"Under different circumstances, moreover, it [the printing press] might have been welcomed and put to entirely different uses - monopolized by priests and rulers, for example, and withheld from free-wheeling urban entrepreneurs."

As a 'free-wheeling urban entrepreneur', this is musical.  Our vision at Swivel is to create a place where everyone can contribute to the collection, exploration and shared insights that follow from having all the world's data flowing in public.

We'll be rolling out features over the coming months that we hope illustrate that the Swivel dream is both timeless and timely.  As always thanks to all you Swivelers for helping to make the dream come true.

Yes, that is a printing press coming down from heaven.  The image is from the frontispiece of Prosper Marchand's Histoire de l’origine et des premiers progrès de l’imprimerie (The Haag 1740).