Does your Doctor Know It’s Safe to Take That?


, , , ,

The Scientific Method is the backbone of research worldwide. With its origins in Greek science and philosophy, science is founded by this process both less formally with Sir Isaac Newtown and more formally today with the full panoply of government regulatory vigor.

Although not so popular in practice or even notorious for its possibilities, Big Data can and should challenge that perspective. Big Data can be used to prove hypotheses, but the true capability of Big Data is in finding patterns within the data without preconception of what the results could or should be.  This debunks the institution.

Why should this ancient practice be questioned?

Ben Goldacre is a epidemiologist with very much to say about how current scientific means effect individual lives, as well as population health.


As individuals in society, we hold others in regard for accomplishments that give them authority, such as doctor for their medical degree. Although with the internet at our fingertips we have gained access to ever-greater amounts of information, we have also learned some skepticism, but still retain some sheep mentality.

Goldacre points out we still have a retained awe for authority. With a simple example, he explains how authority can be accepted by a large, popular audience when the authority is actually less than ideal.

Then the plot thickens.


Goldacre expounds upon how cause and effect studies are “published” with basic flaws in even the simplest cases. The testing environment does not accurately, or sometimes even remotely, simulate the results touted. In addition, the plethora of factors involved is rarely accounted.   The test sample sets are representative of general or specific populations, but are these representative of YOU?


Goldacre somberly explains then that these simple examples are just that – simple. Drug studies that are the basis of doctors’ “knowledge” of treating YOU and society are based upon far more complex … and jaded processes.

Our beliefs and expectations of a drug’s efficacy shape the outcome. He gives several examples of how data is effectively rigged to produce a carefully prepared outcome. Thus making the result look … like what they want you to see.


Goldacre’s final, sobering point was actually the jumping off point for his next Ted Talk.


NEXT POST: how Big Data addresses these short falls

What’s in YOUR wallet? – WEARABLES


, ,

Although iWatch has been occupying the Apple faithful since its concept was unveiled (and now since its release keeps sliding into the future), “wearables” or sensors that people wear have been developing for some time.


WearableTECHweb Source: The Independent

Basically, wearables measure body activity. The earliest wearable was possibly the pedometer. Pressing a button told you how many steps taken in a given period. Now wearables not only are configured for a plethora of activities and body measurements, they also connect digitally to create a Big Data digital record of YOU.

More Than a Sport Watch

The iWatch tells more than the time, and wearables aren’t just wristband mediums. Wearables are headsets or eye wear (think Google glass), gloves and clothes, and even sensors embedded under your skin or in your body. The prevalence and low entry cost (<$75) of wristband wearable technology demonstrates the nascent entry of technology from the keyboard to personal experience.


Wearables are both active and passive. Like taking a picture, the wearable can actively,  measure specific events, such as running times, rates, distances, efficiency. Passively, the wearable can be “always on” to record your rest and active periods, heart rates, body temperature to name but a few.

No “One Size Fits All”

Wearables meet a variety of needs, and as such, the products range from “tactical” (military grade) to the tamer upgraded pedometer model. TROY ANGRIGNON has a great overview on his blog page. If you are interested in choosing one for your sport or activity level, this page is a great resource to begin looking before pricing to show what is available and best suited for a particular sport or level of activity.


Troy’s Awesome Overview of Smart Watches

Not Just Fitness 

The function and capability of wearables does not stop and start with fitness and sports.  Wearables are used in business for virtual reality, warehousing, telecommunications, and even bookkeeping.  Wearables are a natural fit for medical monitoring. Non-invasive blood glucose and pressure and oxygen levels are just one example.  Of course, wearables have become mainstream in military operations and police enforcement.

soldiers standing with civilian

soldiers standing with civilian

Big Data Bigger Picture

With this brief overview of wearable technology, it is easy to appreciate the volume and variety of data collecting over what was possible 5-10 years ago.  (Heck, even last year.)  The next few posts will dive deeper into the plethora of sensors that exist today and that continue to create the Big Data world around each of us – smart watch or not.

What’s in your wallet?  (Because soon anyone can find out)

More on medical wearables next.

How The Apple Watch And iPhone 6 Plus Might Flip Your Mobile Computing Habits

Colette Grail:

The iWatch continues to make headlines as its launch date bumps downstream as well. Here’s more thoughts on what MIGHT happen when it actually arrives.

Originally posted on TechCrunch:

Apple’s new wearable hardware could eventually become much more than just an optional accessory – eventually, it could be one half of a Voltron-style combo that makes up the bulk of our computing life, relegating the tablet and smartphone model to the past. Just like a tablet/smartphone combo was a common duo over the past few years, a smartwatch/phablet duo could be the optimal setup for working on-the-go in the future.

The iPad and iPhone previously operated together as a way to both quickly and easily handle small tasks, but also to have a larger device on hand for taking care of more serious business, or for easier reading of longer content. Apple’s ability to create a tablet that people actually wanted to use probably cut the home PC out of the loop for a big chunk of users – and the market trends among the general PC OEM population…

View original 513 more words

Amazon refunded my money – when I didn’t ask for it


, , , , , ,

Amazon refunded my money today.

I had rented a throw back movie with which I was trying to impress my teenagers. Although that tactic has succeeded before, this time … not so much. What didn’t help was the pixeled movie quality, making it quite blurry at times. We also had to take a break about 10 minutes into the start to let it buffer sufficiently and it still had some issues. So, it’s not surprising I would want my money back, right?

But I didn’t ASK for my money back.

We watched the movie on a Saturday night, and Tuesday I received an email saying:

We noticed that you recently experienced poor video playback on Amazon Instant Video. We’re sorry for the inconvenience, and have issued you a refund…

While Amazon Instant Video transactions are typically not refundable, we are happy to make an exception in this case.

Wow! They refunded my money without my asking.  Amazon pioneered e-commerce in many ways, but usually it’s associated with how they anticipate what you want to buy next. This was anticipating my consumer behavior, maybe even before I did.

I didn’t think I could get a refund for a streaming video, so we slugged through watching it. It did impress upon me that I might not rent from Amazon the next time. Although I never even said that aloud, the experience had affected my future consumer behavior.

It’s all about the (customer) base

This tells me two things about Big Data and Amazon’s commanding posture in a market segment of one world.

  • Anticipating the customer’s experience isn’t just about the sale. They anticipated poor performance would linger in my decision making.
  • They predicted my action before I got to it: I wouldn’t choose Amazon the next time because of the last occurrence.

I was just thinking I needed …

If that’s not impressive enough, Amazon is going further to predict what you want – even before you order it. Amazon’s “Anticipatory Shipping” is a patent pending system of shipping goods to you before you decide to buy them.

Amazon Patent Pending for Anticipatory Shipping

Amazon Patent Pending for Anticipatory Shipping

That makes sense if you just read the first book in a series and Amazon can predict when you’re ready for the next one. Textbooks are easy targets as well. But what about all those things you run out of: toilet paper, paper towels, coffee, printer ink, cat food, OTC medicine.

What else could Amazon decide for you?

In a growing world of the Internet of Things (IoT), Amazon’s predictive analysis builds strength bit by bit, but what about services? Wouldn’t it be nice if the air conditioner repairman was waiting for you because the A/C went out while you’re at work? Or what if the cable guy had already been by because your download speed your paying to get  wasn’t there.  Heck, it would be crazy nice to have milk or eggs show up on the doorstep as well. (Wait, didn’t that used to happen?)



March Madness: $15,000 for Best Bracketology


, , , , ,

This isn’t your typical office pool, nor the higher speed online bracket contest for the annual NCAA basketball playoff tournament. For the second year, Kaggle hosts the mother of bracketology contests – March Machine Learning Mania – with a purse of $15,000 sponsored by HP. Granted that’s not as much moolah as some people will acquire via old fashioned Las Vegas standards, but it does show a growing interest and relevance to the role of data in sports.

Kaggle's March Madness Competition

Kaggle’s March Machine Learning Mania

For the contest, Kaggle primes the data lake by providing 30 years of historical data on the teams to participants. Augment the legacy info with your own social media, or use whatever data sets you can wrangle.

The contest is divided into two stages: pre and post dance card.

Stage 1 – Model Building

  • Feb 2 – Mar 14, 2015 – competitors build and test models on historical data. During this phase, the leaderboard shows the model performance on historical tournament outcomes.

Stage 2 – 2015 Championship

  • Sunday, Mar 15 – Selection Sunday (68 teams announced)

  • Monday, Mar 16 – Kaggle begins to accept 2015 predictions. Release of up-to-date 2014-2015 season data.

  • Wednesday, Mar 18 – Final deadline to submit 2015 predictions (11:59PM UTC).

Mar 19 – Apr 6 – sit back, relax, and watch your predictions come true!


 Winners and Losers

The prize is NOT determined in the traditional bracket results. Instead, a more geekily appropriate percentages for the likelihood of winning each possible matchup determines the winner.

For the doubters that sports and geeks go together (and you missed the movie Moneyball), MIT hosted its 8th high profile Sloan Sports Analytics Conference last month with players, coaches, correspondents and analysts exploring the growing industry of sports analytics.

ONLINE DATING: A Lesson in Big Data Volume


, , , , , , ,

Big Data isn’t all about big business.

Online dating is a great example of the Volume principle of Big Data. Dating itself is a data challenge: physical attributes, personality, relative location, income, and interests. Is it the high school sweetheart? Love at work or play? Love at first sight? Or years of hits and misses?  Who and where is the Soul Mate?

Sharif Corinaldi, a Berkeley theoretical physics graduate student, put his computer hacking skills to the test for his heart. He recently shared his online dating experience both as a post for The Guardian and as an algorithm on

The Set Up

Basically (if I understand him correctly) Sharif created an algorithm to use a computer to search for potential dates in his stead because he determined that given his requirements, the time involved to actually search himself was too lengthy. (Too little Return on Investment)

After letting the initial program run, he was disappointed with his results. Between his

female pickiness” (fem_Pck) and “creepiness tolerance” (creep_Tol), my model had determined I’d have to look through 600-700 profiles a night to have any hope of being exposed to Ms Right before she got fed up, burnt out and sequestered herself off in a nunnery, or at least got back with her ex.” – The Guardian 

Let It Go

The beauty of Big Data Volume kicked in though when he left the program to continue browsing profiles, inadvertently removing the specified likes and dislikes.

views didn’t pay attention to body type, race, or age, and mostly visited women that had just joined the site, or women that were high matches for me, many of them left wanting for attention by the usual online meat market. – The Guardian

With the filter removed, in one night he got more attention than 3 months of searching. By allowing ALL the data to flow through, he broke the biases that had formed within the dating sites … and his own ideas on date prospects. In the chaos that ensued, a signal appeared. He went from zero to 3 or 4 dates per week.

What exactly was the high volume?

On the first date with his future girlfriend, the jig was up. She pointed out that he had viewed her site over 100 times a day, which might have been creepy … but for the same was true for her roommate.

His successfully repeated the algorithm experiment with close friends, one of whom found out he was attractive to nurses.

As for his new girlfriend, she actually lived two blocks down from him in San Francisco and worked at a coffee shop around the corner.

Pulling the Filters 

Before Big Data, placing filters in data was often necessary because of the limited capability of human understanding and computer computational capacity. With Big Data processes and tools, the shift to total data ingestion provides richer solution sets.

Just by showing a little interest in women I would have otherwise ignored, an algorithm changed my romantic life

In Sharif’s search for love shows, quite possibly the right answer is in front, hidden only behind our own preconceptions.

Who are the Digital Stall Out or Stand Out Countries?


, , , , , , , , ,

Like weather created by uneven heating of the earth’s surface, perhaps national economies can be forecast on their digital capacity.  Harvard Business Review printed their prognosis for how the electrons roll.  The results are a lovely swirl of countries huddled in a neutral center, perhaps suggesting any country could slide into a more or less desirable zone at any time.

As part of our research, we wanted to understand who was changing quickly to prepare for the digital marketplace and who wasn’t.

Where the Digital Economy Is Moving the Fastest

  • Stand Out countries have shown high levels of digital development in the past and continue to remain on an upward trajectory.
  • Stall Out countries have achieved a high level of evolution in the past but are losing momentum and risk falling behind.
  • Break Out countries have the potential to develop strong digital economies. Though their overall score is still low, they are moving upward and are poised to become Stand Out countries in the future.
  • Watch Out countries face significant opportunities and challenges, with low scores on both current level and upward motion of their DEI. Some may be able to overcome limitations with clever innovations and stopgap measures, while others seem to be stuck.

What country is in your wallet?

Not so surprisingly, the BRICS countries all gather in the BREAK OUT quadrant, with Russia perhaps loitering a bit too closely to WATCH OUT.  Interestingly, a string of European countries occupy STALL OUT.  It would’ve been more interesting had they lined up geographically for the portrait.  Relative to the other zones, the STAND OUT grouping is quite the republic of representatives.

Our Digital Evolution Index (DEI), created by the Fletcher School at Tufts University (with support from Mastercard and DataCash), is derived from four broad drivers: supply-side factors (including access, fulfillment, and transactions infrastructure); demand-side factors (including consumer behaviors and trends, financial and Internet and social media savviness); innovations (including the entrepreneurial, technological and funding ecosystems, presence and extent of disruptive forces and the presence of a start-up culture and mindset); and institutions (including government effectiveness and its role in business, laws and regulations and promoting the digital ecosystem). The resulting index includes a ranking of 50 countries, which were chosen because they are either home to most of the current 3 billion internet users or they are where the next billion users are likely to come from.

The stinger behind the research though is that the digital future has a significant twist.  The Information Age began on desktops and moved to other devices.  It’s destiny – for the next seismic event – will be held in the hand of a billion people in areas remote and unseen to date.

What does the future hold? The next billion consumers to come online will be making their digital decisions on a mobile device – very different from the practices of the first billion that helped build many of the foundations of the current e-commerce industry.

Big Data and Cyber Security – Answers You Might Want to Know


, , , , , , , , ,

Stealing is not a new concept, but threats to the digital world morph incessantly.  Big Data analytics company Sqrrl is expert on detecting data anomalies and in turn potential threats to your data sources.

Sqrrl is hosting a webinar for the latest and greatest in cyber security with Richard Stiennon, industry analyst and author of Surviving Cyberwar, and from Luis Maldonado, Sqrrl VP of Products.  Check it out.

SQRrl Diagram

Click here to sign up.

Evolution in cybersecurity is the norm. As computer threats evolve, so have defenses. The debilitating effect of viruses borne by email gave rise to the what is now a vast anti-virus infrastructure. The rise of network-based attacks created the incrementalism of constant updates to IDS and IPS. The inability to make sense of millions of IDS alerts gave rise to SIEM solutions.

This approach to IT security, while addressing the need for operational hygiene, does not counter targeted attacks.  It are the targeted attackers and insiders that are taking advantage of the focus on incremental defensive improvements to sidestep defenses and achieve their objectives. The revolution that is occurring in cybersecurity is based on discovering these threats without the use of signatures and predefined correlation rules. The use of Big Data analytics can enable a new breed of cyber incident detection and response.

Join us on March 18 at 1 PM to hear from Richard Stiennon, industry analyst and author of Surviving Cyberwar, and from Luis Maldonado, Sqrrl VP of Products. In this webinar you will learn:

  • Richard’s perspective on cybersecurity evolution vs. revolution, and the reason he had to put his next book, Cyber Defense, on hold as the security industry plays catch up against attackers.

  • How Sqrrl is on the forefront of the revolution in cybersecurity by leveraging Big Data Analytics and Linked Data techniques

  • How the Sqrrl Enterprise tool unites cyber incident detection and response efforts and is used for cyber hunting, APT mitigation, and insider threat response.


Get every new post delivered to your Inbox.

Join 122 other followers