Check out this great data story about 2009 Oscar movie piracy on Waxy.org.
Andy does a detailed analysis of the piracy of Academy Award movies. His conclusions:
Find out more, download the source data, and see some great graphs at Waxy.
Chris Anderson, editor-in-chief at Wired, is generating
quite a bit of buzz with his article, "The End of Theory: The Data
Deluge Makes the Scientific Theory Obsolete." The controversial article
suggests that in the "Age of the Petabyte," scientific theory is
becoming outdated. He cites Google's search engines as the example par excellence
in supporting his claim that "Correlation supersedes causation, and
science can advance even without
coherent models, unified theories, or really any mechanistic
explanation at all." Anderson suggests that mining through gargantuan
amounts of data will produce as many new discoveries and insights as
years of scientific research.
There is no denying the importance of new data technology, but Anderson fails to recognize that data mining cannot replace science. Data collection is only one step of the scientific method: observation. He points to the work of biologist, J. Craig Venter, who uses supercomputers to sequence entire ecosystems. In doing so, he has discovered thousands of previously unknown species of bacteria and other life-forms. But what is the point of knowing these species are out there (we're already aware of that) if we don't know anything about them? As one blog post points out, with Venter's data we can only make a few guesses about the properties of the organisms based on who their relatives are -- an activity that requires a little scientific theory called evolution.
Data mining can really only point us in the right direction of new discovery by showing us relationships between data points; it can't generate new discoveries alone. Anderson quickly throws out every theory of human behavior: "Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data the numbers speak for themselves." Call me old-fashioned (Anderson probably would), but to me, the "what" doesn't really matter without the "why." Stripped of its context, a number is just a factoid, a small puzzle piece without the larger picture.
Without science, data is no better than babel. While data can lead to new levels of understanding, Anderson's theory misses the point of the study of science: to intelligently understand the natural world. Anderson may be too busy praising the Google gods to take note of the possibility of the semantic web, where the goal is more than crunching data -- it's understanding information. Although data mining may change the rules of the science game, it's definitely not the end of theory.
As we transition to the Age of the Petabyte, I don't see new technology leaving scientific theory in the dust. Rather, theory will be alive and kicking, as technology and science continue to evolve side-by-side. And with the unwavering certainty of the clock as it ticks and tocks on, new rules will become old ones, and one day even data mining will be replaced by a shiny new method of generating information.
Since Swivel Preview launched just over a year ago, we have been watching closely as newspapers, magazines and blogs have increased their use of data. This has resulted in some very creative and beautiful uses of visualizations by journalists to tell their story.
Journalism is changing. Journalists may now go to print faster than ever before, either in traditional publications via their online version, or in the 'citizen journalism' blogging space. Research, writing, and publishing can all happen in a few minutes instead of days or weeks. With journalists having easy access to data and analysis tools we hope to see more of this.
And as the demand from journalists increases, we are observing an increased willingness among data providers to be more creative about how they are giving access in the first place. This is a good trend where both sides benefit. Journalists are able to give compelling evidence and data providers see their data used more broadly, potentially impacting more people.
We were thrilled to see this recent post on the topic by Matt Croyden at Postneo. We also welcome The Poynter Institute as a new Swivel Official Source joining groups like the Newspaper Association of America. And we’d love to hear your take on the trend as well as links to any particularly interesting newsworthy visualizations you have seen recently.
WARNING: This post might be uninteresting to non-baseball fans
I sneaked away from Swivel HQ last week to catch a few baseball spring training games and to spend time with some family in Florida. Going from foggy San Francisco to sunny Florida is always a nice treat and last week was no exception.
We were at the ballpark, in the sun, watching some of the most talented athletes around tune up their skills for the long 6-month Major League Baseball season that starts in just 3 days. What could be better? For a data geek like me what makes it better is being surrounded by data: people gathering data, people analyzing data and people talking about data.
At a baseball game data come flying out of fans' mouths quicker than a 100 mph fastball: 'that guy only hit .253 last season, he's a bum', 'his ERA has been under 3.0 for the last 5 seasons', 'this guy had 100 RBI last season and is only getting started', and so on. For many baseball fans, the numbers are the thing.
There are many folks and organizations that do fantastic work with baseball data. Yet, just today while surfing around Swivel I found another great baseball data resource: a blog called baseballanalysts.com. They have an awesome post that uses multi-dimensional scatter plots to analyze a few of my favorites pitchers (the Detroit Tigers are my team).
In fact, we have been working here at the lab on ways to tell stories with data in Swivel that are intuitive and compelling and the crew at baseballanalysts.com has given us a nice example of telling a complex story in an interesting way using graphs, raw data and prose. Kudos!
"How sad. All governments find it tempting to tweak the numbers they are judged by. But in doing this they deprive themselves of the best guide to future policymaking. And they also create a self-defeating spiral of distrust in which even the numbers they have not tweaked are disbelieved."
From The Economist print edition
March 1st 2007, p. 18
The importance of statistics
Lies, damned lies
Also available online here with a subscription.
BTW, at Swivel we're anti-government-number-tweaking.
Think of the last time you opened a door in an unfamiliar place. Did you pull? Push? Did you feel like the kid in the cartoon? If a door is not a swing door, you can either try to tell people what to do, or design the door so that the only option available is the right one. If you see a smooth brass plate with nothing to grab on to, the only thing you can do is push. If you see an inviting handle, you will probably pull. People in the know tell me this is called affordance, although I can never remember the word when I need it.
When I picked up an iPod for the first time, I put the earphones in my ears and started looking for a volume control of some kind, a little wheel maybe, or a slider. For a few seconds I was stumped -- nothing I saw looked like a volume control. Accidentally I brushed my thumb across the front of the iPod. The screen lit up and I saw the familiar volume bar. I had a cool feeling of being surprised. I started playing with the scroll wheel to see what else was there. This is a different kind of affordance, it seems. The interface kept you in suspense for a moment by removing the familiar, and then pleasantly surprised you, in a kind of way that made you want to discover more.
As we're rethinking the design of Swivel, this concept is one of the things on our minds. Can you remember situations, in the physical world or on the web, where affordance was just right -- or completely wrong?
When we launched in December we heard plenty of positive things about Swivel. However, we also heard 'constructive criticism'. One of the blog posts packed with suggestions for us was this entry from Stephen Few's blog, Visual Business Intelligence.
We took his thoughts to heart and even ordered his books from amazon so we can read them and have them here at Swivel HQ. They recently arrived, here are links to the books.
The first bit of content in Show Me the Numbers is actually the full text of Ode to Numbers by Pablo Neruda which I found online here.
The poem and the books are artful. And the books are especially informative. So many awesome things happening with numbers. Check them out.
My high school history teacher liked to illustrate the fallacy of averages with this vivid example: "Say I had two steaks for dinner, and you had none. On average, we each had a steak. Isn't that great?" I remembered his words when I came across the article on the flaw of averages. The author mentioned another metaphor that is both funnier and more tragic: "A humorous example involves the statistician who drowned while wading a lake that was, on average, only three feet deep." The article uses a hypothetical investment scenario to drive home the point that using averages may get you in a lot of trouble.
Average is a useful mechanism for compressing data to simplify the picture. It's common for Swivel to show averages on graphs. For example, this graph shows average distance from the Super Bowl host city to the winner's city (highest seven between 1967 and 2005):
LA is the "winner" here: you could probably guess that most of the time the winning team isn't from there. Of course, it's only a guess because the average hints at what's going on but hides the real picture. It's actually possible in Swivel right now to change the display of data to highest or lowest values (just click in Edit button and play with the 'Display' drop-down list) but what's needed to get a better view is the range of values. A box plot is one way to show it. Our graphing engine, Ploticus, supports box plots, so we'll just need to make them look nice and add the option to the user interface. I love long to-do lists, there's never a worry that you'll run out of things to work on.
So next time you see a graph on Swivel showing average data, remember the drowned statistician.
(UPDATE: this blog post fell victim to a sorting bug: bar graphs occasionally lose sort order, so I had to update the embedded graph. Apologies to those of you who've had the same problem with Swivel graphs. We're on this bug and will fix it soon.)
there's a great post on the basics of correlations (including a couple amusing examples) from good math, bad math today. an excerpt:
So when someone shows you a correlation, what you should do is look for a plausible causal mechanism, and see if there's any experimental data to support it. Without a demonstrable causal mechanism, you can't be sure that there's a causal relationship - it's just a correlation.
here's a fun example from swivel.
We just read this morning over at our friend Tim O'Reilly's blog that Big Blue is launching a site called Many Eyes. All of us Swivelers clicked around over there this morning and the visualizations are phantastic! They've got a bunch of data sets and allow folks to upload more. We are fans of IBM (we use their blade servers) they know how to get stuff done and we're stoked that we'll all be working on the same problem to create more value from the world's data. Whoop! Go data.
Swivel Home
Recent Comments