How to avoid being fooled by misleading statistics in your Web Conference
Are your staff using statistics to lie to you? As per standard business practice, most conferences – especially visuals based web conferences or webinars – will contain statistics. Statistics are used to measure business performance or to assist in evidence-based decision making, among other things and being able to generate statistics is considered an essential trait of any good business person.
However, how often do we look at the other side of that conversation? How often do we question our ability to interpret statistics, or because they are considered true have we dropped our guard and take anything backed up with the relevant statistics as fact?
“Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write.”
– H.G. Wells
In this post we will lay out some concepts you can use as tools to critically analyse any misleading statistics presented to you in a meeting.
Correlation does not equal Causation
In the animal rescue community, there is a phenomenon called ‘Black Dog Syndrome’ where dogs with black coats may take longer to be adopted. Initial thoughts and surveys suggested a subconscious bias towards black dogs and cats as the media negatively typecast them. An ‘evil’ dog in a film will typically be black, or the old superstition of a black cat crossing one’s path.
Media tropes and adoption times were correlated, but later research suggested that in fact, the longer adoption times could be because animals with black coats don’t photograph well affecting their adoption website profiles.
When looking at statistics in a web conference ask how other possible factors have been allowed for or investigated before the presented conclusion was drawn. For example when talking about sales, has the presenter considered the time of year, concurrent with consumer spending habits, how the data was collected and how this might affect a data sample. This bias is one of the hardest as it requires more attention to spot, but it still is misleading statistics.
The Prosecutor’s Fallacy often occurs when there is a mismatch of probabilities. For example, if you were trying to find out what weather conditions are the most dangerous for motorists you could look at the data for the number of accidents per weather condition. You might find out that more road accidents occur in clear weather than foggy weather but would it be accurate to say that clear weather is more dangerous than foggy conditions?
No, you need to take into consideration the fact that clear weather occurs more often than foggy weather and adjust your data sample so that you are comparing two samples of equal probability. Do more accident occur in two hours of clear weather or two hours of foggy weather?
In particular look out for statistics derived from data dredging. This is where vast amounts of data are pulled for analysis without setting reasonable or useful parameters on the data. I.e. Should you look at a data set the same size as the population of the city you are targeting? In some cases, the person compiling the data will not stop adding new data until the sample they have fits the picture they want to paint with statistics and the larger the database you draw from, the more likely you are to find evidence by way of coincidence.
For example, if they are trying to prove that 1% of people own a red car but of the initial 1000 person sample, only .5% of people own a red car they may increase the sample to 1500 and so on until they find the exact sample where 1% own a red car.
Let’s begin at the beginning and beware of any graph that doesn’t. Graphs and charts are the staple of communicating statistics, and your web conferences may be riddled with them but beware any graph that doesn’t start at 0 and chart that makes use of pictographs instead of bars.
The figures may be sound but the visual representation may be crafted for maximum effect. A graph that illustrates an increase in sales from 8k to 9k over a 6 month period could be illustrated on an x-axis (horizontal) of 0 -10 and a y-axis (vertical) of Jan – June. The same information could be illustrated on an x-axis of 7,100e – 9,100e and a y-axis of Jan – June and while it doesn’t change the data it does make the data look far more impressive.
Additionally, the presenter could choose to illustrate this increase using a representation of units sold in a bar chart. They could illustrate sales from Jan – June in 2017 vs sales for the same period in 2016 with two bars: one for 10 units sold and another for 20 units sold.
Suppose your company sells cars and for the sake of showmanship the presenter chooses to illustrate the increase using two cars, doubling the height of the second car. However, if they have to double the height they also have to double the width and the second car is now four times the size of the first, taking into account height and width. The second car is now far more impressive than the first.
It takes a high level of awareness and astuteness in order to spot statistics that may mislead you and potentially damage your business, particularly as the presenter generally doesn’t intend to do either of those things, they just want to present the strongest most convincing case possible. Additionally, you don’t always have the time to mull over and identify weak statistics, especially if you have called the web conference for a time-sensitive issue. Successful critical analysis of statistics comes with a great deal of practice but hopefully, the tools set out for you in this article will help get you up to speed quickly.