Skip to main content

The Quest for Data

I'm a data guy. I spend many many hours at my day job staring at tables, crunching numbers, and turning piles of data into something comprehensible. Yet a day doesn't go by when I don't at some point mutter, "man, I sure wish we had better data on this question".

(from ckhamken on Flickr)

We (society) are currently at a crossroads when it comes to how we  feel about data. On the one hand, there's a genuine fear that corporations and government are harvesting every single data point about us that they possibly can and that they're doing something malicious with it. Facebook is selling our data to advertisers! Target is using our data to send diaper coupons to pregnant ladies! The government knows every step we take every day of the week!

If data this comprehensive exists, I'm yet to see it. Instead, I'm often faced with the opposite dilemma: a situation that's not especially well studied that could be better understood by data. It isn't either because no such data exists, because it's being held by someone powerful behind bulletproof glass, or because it's simply in a format that's not analyzable (try sorting thousands of records of physical paper and you'll know what I mean).

Last week I downloaded Untapped on my iPhone. Its premise is simple - keep track of the beers you drink, when you drank them, where, how you liked them, etc. On the surface, the utility of this app seems limited, and appears to appeal to people in the "share happy" world we now  live in. Unless of course you're using it to harvest data about yourself that could later yield some very interesting results.

When I was first trying to figure out the excitement about the Foursquare app, someone left a comment on this blog explaining that he uses the app essentially as a ledger to document his life. It's a running tab of every place he goes during every day. I hadn't initially thought about it in that way. To me, it was some kind of goofy game where you score points and compete against other people; not a data collection tool on your movement from place to place.

To me, both Untapped and Foursquare show that people actually want to harvest data on themselves. Why? Because it can lead to interesting insights. What was the most frequent beer I drank last year? What were the most frequent places I visited? Did my enjoyment of those beer differ by different days of the week? Did I see any evidence of my tastes changing with time? If I had to recall from memory, I might have some good guesses, but I might significantly over or under count certain things due to a cognitive bias. Keeping records is the only way to know for sure.

I've come across data warehouses online that make you check a box when you sign up for an account. This box says "I agree to use this website always for GOOD, never for EVIL". I think it perfectly sums up the struggle over data. Data can be a powerful tool in the quest for knowledge and understanding. But if it falls into the wrong hands, it can be used for potentially malicious purposes. It's the fear of the evil that often keeps valuable data from being used by the forces of good.


Popular posts from this blog

In Praise of Southwest's 'C' Boarding Group

A few weeks ago I saw a tweet from someone complaining that their Southwest Airlines boarding pass had been assigned A20 (meaning they would be at least one of the first twenty passengers to board the plane). Apparently this person though they should have been assigned a higher number, less their flight experience be considerably spoiled.

Despite the complaints, Southwest has resisted demands to assign seats on its flights, a decision which I personally applaud. I'll admit that I was skeptical when they rolled out the newest boarding procedure, assigning both boarding groups and a line number; but in hindsight it seems like one of the best operational decisions they've ever made. If nothing else, it effectively eliminated the infamous "cattle call" whereby fliers were getting to airports hours in advance and sitting in line on the floor as if they were waiting for the midnight showing of the new Star Wars movie.

When I was an intern at Southwest Airlines last winter, I…

So You Want to be a Southwest Airlines Intern?

My personal website must have pretty decent SEO - because in the past year, I've received about two dozen emails from aspiring Southwest Airlines interns looking to draw on my experience in search of their own dream internship. In the past two weeks alone a few new emails have already started rolling in...

(from flickr user San Diego Shooter)

If you've found your way here, you might be hoping for the silver bullet; a secret tip that will propel you above the competition. Unfortunately, I do not know any inside secrets. I can only share my experience as an internship candidate about two years ago and, rather than responding individually to future emails I anticipate to receive, I hope that potential interns will find the information posted here valuable.

Understand: Southwest Airlines is a very unique company. The corporate culture at Southwest is truly unlike that of nearly every other company. But you probably already knew that, since it now seems mandatory for every management,…

Commuting Meets Technology

I'm finally out of the dark ages. I got an Android smartphone over the weekend and have since been in the process of exploring the Android apps market.  One thing I've immediately noticed is the really wide range of usefulness in the apps. For example, the WeatherBug app is fantastic. It automatically determines your location and gives you exact conditions for that location. On the other end of the spectrum, Google's Goggles app is supposed to be a type of 'visual search' where you snap of photo of something and Google searches for it. In each of my attempts to use it, the app hasn't returned any search results. I even took a photo of a bottle of Pepsi (figuring it as a common houseful item) and got nothing.

Somewhere in the middle is this app called Waze. Have a look at their 'guided tour':

Some people might look at it and comment on the amazing evolution of technology or on the incredible value of social networks. To me, Waze says something important ab…