The 3Vs of Big Data – Volume, Velocity, and Variety

Bringing Big Data to the People (Part 3 of 6)

Processing – Velocity

Velocity is the accelerated rate at which data accumulates.  Big Data isn’t just more data but more data faster too.  For example, when you visit a website, the data trail isn’t purely transactional – how much of what did you buy – it follows what you clicked on, how long you spent on a page, how long on the website and more.  Now any off-the-shelf web data program offers this depth.

This streaming data is real time (or relatively real time) feedback loops used to solve problems or provide products or services. This is critical when knowing what the environment was a year ago, a month ago, a day ago, or even a minute ago may be irrelevant to now.  

When scientists first decoded the human genome in 2003, it took them a decade of intensive work to sequence the three billion base pairs. Now, a decade later, a single facility can sequence that much DNA in a day.

Behemoths such as Walmart and Google have been using Big Data for some time to be able to bring you the goods and services you want (or didn’t even realize you wanted) because they had the resources to pursue its value.  Within the past few years though, that processing power is now a garage startup capability.  The cloud is cheap to rent.

Google processes more than 24 petabytes of data per day, a volume that is thousands of times the quantity of all printed material in the US Library of Congress.
Google processes more than 24 petabytes of data per day, a volume that is thousands of times the quantity of all printed material in the US Library of Congress.

As the data population has grown, so has the processing capability to consume the data.  As far back as 2005, a cell phone (even without a camera) had more processing power than NASA’s mission control during the Apollo flights that put men on the moon.

…, in 1986 around 40% of the world’s general-purpose computing power took the form of pocket calculators ...
…, in 1986 around 40% of the world’s general-purpose computing power took the form of pocket calculators …
…,  which represent more processing power than all personal computers at the time
… which represented more processing power than all personal computers at the time

Next up – Variety, the spice of life