Open Source Cloud Authors: Liz McMillan, Jason Bloomberg, Yeshim Deniz, Stackify Blog, Vaibhaw Pandey

Related Topics: @DXWorldExpo, Java IoT, Microservices Expo, Open Source Cloud, @CloudExpo, Apache, SDN Journal

@DXWorldExpo: Article

The Fallacies of Big Data

No software, not even Hadoop, can make sense out of anything

The biggest problem with software is that it doesn’t do us any good at all unless our wetware is working properly – and unfortunately, the wetware which resides between our ears is limited, fallible, and insists on a good Chianti every now and then.

Improving our information technology, alas, only exacerbates this problem. Case in point: Big Data. As we’re able to collect, store, and analyze data sets of ever increasing size, our ability to understand and process the results of such analysis putters along, occasionally falling into hidden traps that we never even see coming.

I’m talking about fallacies: widely held beliefs that are nevertheless quite false. While we like to think of ourselves as creatures of logic and reason, we all fall victim to misperceptions, misjudgments, and miscalculations far more often than we care to admit, often without even realizing we’ve lost touch with reality. Such is the human condition.

Combine our natural proclivity to succumb to popular fallacies with the challenge of getting our wetware around just how big Big Data can be, and you have a recipe for disaster. But the good news is that there is hope. The best way to avoid an unseen trap in your path is to know it’s there. Fallacies are easy to avoid if you recognize them for what they are before they mislead you.

The Lottery Paradox
The first fallacy to recognize – and thus, to avoid – is the lottery paradox. The lottery paradox states that people place an inordinate emphasis on improbable events. Nobody would ever buy a lottery ticket if they based their decision to purchase on the odds of winning. As the probability of winning drops to extraordinarily low numbers (for example, the chance of winning the Powerball is less than 175,000,000 to 1), people simply lose touch with the reality of the odds.

Furthermore, it’s important to note that the chance someone will win the jackpot is relatively high, simply because so many tickets are sold. People erroneously correlate these two probabilities as though they were somehow comparable: “someone has to win, so why not me?” we all like to say, as we shell out our $2 per ticket. Assuming tens of millions of people were to read this article (I should be so lucky!) then it would be somewhat likely that some member of this impressive audience will win the lottery. But sorry to say, it won’t be you.

The same fallacy can crop up with Big Data. As the size of Big Data sets explode, the chance of finding a particular analytical result, in other words, a “nugget of wisdom,” becomes increasingly small. However, the chance of finding some interesting result is quite high. Our natural tendency to conflate these two probabilities can lead to excess investment in the expectation of a particular result. And then when we don’t get the result we’re looking for, we wonder if we’ve just wasted all the money we just sunk into all our Big Data tools.

Another way of looking at the lottery paradox goes under the name the law of truly large numbers. Essentially, this law states that if your sample size is very large, then any outrageous thing is likely to happen. And with Big Data, our sample sizes can be truly enormous. With the lottery example, we have a single outrageous event (I win the lottery!) but in a broader context, any outrageous result will occur as long as your data sets are large enough. But just because we’re dealing with Big Data doesn’t mean that outrageous events are any more likely than before.

The Fallacy of Statistical Significance
Anybody who’s ever wondered how political pollsters can draw broad conclusions of popular opinion based upon a small handful of people knows that statistical sampling can lead to plenty of monkey business. Small sampling sizes lead to large margins of uncertainty, which in turn can lead to statistically insignificant results. For example, if candidate A is leading candidate B by 2%, but the margin of error is 5%, then the 2% is insignificant – there’s a very good chance the 2% is the result of sampling error rather than reflecting the population at large. For a lead to be significant, it has to be a bit more than the margin of error. So if candidate A is leading by, say, 7%, we can be reasonably sure that lead reflects the true opinion of the population.

So far so good, but if we add Big Data to the mix, we have a different problem. Let’s say we up the sample size from a few hundred to a few million. Now our margins of error are a fraction of a percent. Candidate A may have a statistically significant lead even if it’s 50.1% vs. 49.9%. But while a 7% lead might be difficult to overcome in the few weeks leading up to an election, a 0.2% lead could easily be reversed in a single day. Our outsized sample size has led us to place too much stock in the notion of statistical significance, because it no longer relates to how we define significance in a broader sense.

The way to avoid this fallacy is to make proper use of sampling theory: even when you have immense Big Data sets, you may want to take random samples of a manageable size in order to obtain useful results. In other words, fewer data can actually be better than more data. Note that this sampling approach flies in the face of exhaustive processing algorithms like the ones that Hadoop is particularly good at, which are likely to lead you directly into the fallacy of statistical significance.

Playing with Numbers
Just as people struggle to grok astronomically small probabilities, people also struggle to get their heads around very large numbers as well. Inevitably, they end up resorting so some wacky metaphor that inevitably contains an astronomical comparison involving stacks of pancakes to the moon or some such. Such metaphors can help people understand large numbers – or they can simply confuse or mislead people about large numbers. Add Big Data to the mix and you suddenly have the power to sow misinformation far and wide.

Take, for example, the NSA. In a document released August 9th, the NSA explained that:

According to the figures published by a major tech provider, the Internet carries 1,826 Petabytes of information per day. In its foreign intelligence mission, NSA touches about 1.6% of that. However, of the 1.6% of the data, only 0.025% is actually selected for review. The net effect is that NSA analysts look at 0.00004% of the world’s traffic in conducting their mission – that’s less than one part in a million. Put another way, if a standard basketball court represented the global communications environment, NSA’s total collection would be represented by an area smaller than a dime on that basketball court.

Confused yet? Let’s pick apart what this paragraph is actually saying and you be the judge. The NSA claims to be analyzing 1.6% of 1,826 Petabytes per day, which works out to about 29 Petabytes per day, or 30,000 terabytes. (29 petabytes per day also works out to over 10 exabytes per year. Talk about Big Data!)

When they say they select 0.025% (one fortieth of a percent) of this 30,000 terabytes per day for review, what they’re saying is that their automated Big Data crunching analysis algorithms give them 7.5 terabytes of results to process manually, every day. To place this number into context, assume that those 7.5 terabytes consisted entirely of telephone call detail records, or CDRs. Now, we know that the NSA is analyzing far more than CDRs, but we can use CDRs to do a little counter-spin of our own. Since a rule of thumb is that an average CDR is 200 bytes long, 7.5 terabytes represents records of 37 quadrillion (37,000,000,000,000,000) phone calls, or about 5 million phone calls per day for each person on earth.

So, which is a more accurate way of looking at the NSA data analysis: a dime in a basketball court or 5 million phone calls per day for each man, woman, and child on the planet? The answer is that both comparisons are skewed to prove a point. You should take any such explanation of Big Data with a Big Data-sized grain of salt.

The ZapThink Take
Perhaps the most pernicious fallacy to target Big Data is the “more is better” paradox: the false assumption that if a certain quantity of data is good, then more data are necessarily better. In reality, more data can actually be a bad thing. You may be encouraging the creation of duplicate or incorrect data. The chance your data are redundant goes way up. And worst of all, you may be collecting increasing quantities of irrelevant data.

In our old, “small data” world, we were careful what data we collected in the first place, because we knew we were using tools that could only deal with so much data. So if you wanted, say, to understand the pitching stats for the Boston Red Sox, you’d start with only Red Sox data, not data from all of baseball. But now it’s all about Big Data! Let’s collect everything and anything, and let Hadoop make sense of it all!

But no software, not even Hadoop, can make sense out of anything. Only our wetware can do that. As our Big Data sets grow and our tools improve, we must never lose sight of the fact that our ability to understand what the technology tells us is a skill set we must continue to hone. Otherwise, not only are the data fooling us, but we’re actually fooling ourselves.

Image credit: _rockinfree


More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.

@ThingsExpo Stories
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
The standardization of container runtimes and images has sparked the creation of an almost overwhelming number of new open source projects that build on and otherwise work with these specifications. Of course, there's Kubernetes, which orchestrates and manages collections of containers. It was one of the first and best-known examples of projects that make containers truly useful for production use. However, more recently, the container ecosystem has truly exploded. A service mesh like Istio addr...
Poor data quality and analytics drive down business value. In fact, Gartner estimated that the average financial impact of poor data quality on organizations is $9.7 million per year. But bad data is much more than a cost center. By eroding trust in information, analytics and the business decisions based on these, it is a serious impediment to digital transformation.
Predicting the future has never been more challenging - not because of the lack of data but because of the flood of ungoverned and risk laden information. Microsoft states that 2.5 exabytes of data are created every day. Expectations and reliance on data are being pushed to the limits, as demands around hybrid options continue to grow.
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
As IoT continues to increase momentum, so does the associated risk. Secure Device Lifecycle Management (DLM) is ranked as one of the most important technology areas of IoT. Driving this trend is the realization that secure support for IoT devices provides companies the ability to deliver high-quality, reliable, secure offerings faster, create new revenue streams, and reduce support costs, all while building a competitive advantage in their markets. In this session, we will use customer use cases...
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As automation and artificial intelligence (AI) power solution development and delivery, many businesses need to build backend cloud capabilities. Well-poised organizations, marketing smart devices with AI and BlockChain capabilities prepare to refine compliance and regulatory capabilities in 2018. Volumes of health, financial, technical and privacy data, along with tightening compliance requirements by...
The IoT Will Grow: In what might be the most obvious prediction of the decade, the IoT will continue to expand next year, with more and more devices coming online every single day. What isn’t so obvious about this prediction: where that growth will occur. The retail, healthcare, and industrial/supply chain industries will likely see the greatest growth. Forrester Research has predicted the IoT will become “the backbone” of customer value as it continues to grow. It is no surprise that retail is ...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of bus...
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO and DXWorldEXPO are two of the most important technology events of the year. Since its launch over eight years ago, @CloudEXPO and DXWorldEXPO have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors! In this blog post, we provide 7 tips on how, as part of our world-class faculty, you can deliver one of the most popular sessions at our events. But before reading...
DXWorldEXPO LLC announced today that "Miami Blockchain Event by FinTechEXPO" has announced that its Call for Papers is now open. The two-day event will present 20 top Blockchain experts. All speaking inquiries which covers the following information can be submitted by email to [email protected] Financial enterprises in New York City, London, Singapore, and other world financial capitals are embracing a new generation of smart, automated FinTech that eliminates many cumbersome, slow, and expe...
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
DXWorldEXPO LLC announced today that ICOHOLDER named "Media Sponsor" of Miami Blockchain Event by FinTechEXPO. ICOHOLDER give you detailed information and help the community to invest in the trusty projects. Miami Blockchain Event by FinTechEXPO has opened its Call for Papers. The two-day event will present 20 top Blockchain experts. All speaking inquiries which covers the following information can be submitted by email to [email protected] Miami Blockchain Event by FinTechEXPO also offers s...
DXWorldEXPO | CloudEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.