There’s no shortage of noise surrounding big data. Today it seems that every software vendor, consulting firm and thought leader has developed the “right” definition of the term. While I’d argue there is no such definition, I would like to dispel a few of the most commonly held myths about the subject, many of which I explore in Too Big to Ignore: The Business Case for Big Data.
Myth 1: You Can Get To All Of The Data
On many levels, we are living in unprecedented times. Never before has so much data been available to us. Forget megabytes and petabytes, exabytes of data now exist. I read recently that the average person in an industrialized society today consumes more information in one day than his fifteenth century counterpart did in his lifetime.
Despite this unfathomable amount of data, no person or organization can store and retrieve all of the data on a particular subject, much less overall. And yes, that includes Google. Its software indexes the Surface Web, not the Deep Web. Some estimates put the latter at 25 times the size of the former. As a result, when you search, you are accessing anywhere from 4% to 6% of all information on the Internet.
Taking it down a level or 30, individual authors like me cannot access some very valuable information, such as which specific customers are buying my books. Sites like Amazon and stores like Barnes & Noble keep that information. Nothing would make me happier than knowing my customers, but even in a big data world, that information eludes me. You will never get all of the data. Period.