What struck me today was a poll related to a recent news in South China Morning Post, as well as various news sources, about the possibility that pilots of one of the world’s best airlines Cathy Pacific (CP) might have been overworked by the airlines, which may lead to very scenario of possible “seriously risky situations” once airborne.
Actually this information came directly from the pilots themselves. It’s a group of pilots that publicly complained about their welfare. What I thought was funny was not the news itself, but the fact that almost immediately after the news was broadcasted to the public, the same newspaper managed to conduct a quick survey on with the question, “Do you think Cathay Pacific pilots are overworked?” Soon enough, the result came in. For people like us who may have relied — consciously or unconsciously — on public opinion before making decisions on various matters, the expected results on a yes-or-not question like this would, as the type of the question suggest, simply be “yes or no.” Surprisingly, it turned out that half of the surveyors simply responded “I don’t know.” Suddenly, I came to realize how absurd this question was. It may appear quite rigid: as a poll conducted by a top newspaper, with a larger enough samples and a rigorous sample size determination (the more random the better in this case), and with a rather convincing pie chart to show the distribution of the answers through a rather beautiful graphic means.
Yet, the problem lies in the very nature of this question, which has an scarily enormous implication about other problems we have in the world. The nature of this question alone leads to the complete uselessness of the result, and the logic here is simple: How would ordinary people on the street or on the internet know whether airline pilots — one of the most professionally secretive jobs in the world — are overworked or not? How much do we know about the nature of their jobs, i.e., how many hours are appropriate for them to work, let alone whether or not they are overworked? Even if we have a slight gist, in the general sense, of such information, how would we know that what we could apply what we know to the specificity of this case, namely the operation of this particular airline? There’s so much that we don’t know in this scenario.
So, first and foremost, there was no point in asking the public this question. Second, making a poll out of the answers they received is simply absurd. Finally — and the scariest part — how many time we take this kind of information that this poll presents you with as views prevalent among the general public, the facts, or even the truth?
This actually makes me scared a bit. There’re so many instances that I simply took whatever I heard, especially if they seemed to have some kinds of statistical evidence to back up their claims, at face value. Say, “Donald Trump is winning,” or “Most people agree with this XXX policy on this YYY country, according to the poll conducted by ZZZ — one of the world’s most trusted sources of independent survey research,” and so on. This specific question on statistics reminds me of a rather famous quote (some attribute it to Mark Twain): “there are three kinds of lies in the world: lies, damned lies and statistics.
The goal for writing this blog is simple. I’d like to remind myself and everyone reading this blog to always read carefully the question, with the goal of analyzing its very question. Don’t easily believe anything you hear, especially when there’s a claim attached to it that it’s “backed by statistics.” In addition to the basic technical questions about the statistics itself (e.g., sample size determination), ask the three following, what I’d like to call, “qualifying questions”:
- Is this question the kind of question to which the informants are capable to gathering enough information or knowledge to inform their answers?
- Is the “very nature” of the question points to the “very problem” that the incident leading to the question itself seeks to answer?
- If this question happens to have the quality that fulfills these two qualifying questions, then ask the third question: would understanding how the majority respond to this question leads to how the problem might be solved for the better of the society as a whole?
The give the newspaper (which I actually love) the benefit of the doubt, perhaps the problem here was not that whoever decided to conduct the poll did not know what he or she was doing, but, simply, that they did not know how to frame the question in a way that would make sense.
What the poll wanted to know wasn’t “whether or not Cathay Pacific pilots are overworked?” but “whether or not knowing that they may be overworked has an affect on how they might perceive traveling with the airline.” So, for a question concerning this particular incident to make sense, what South China Morning Post has to do is simply to tweak the question to reflect that particular angle that people like us, i.e., non-specialist on flying, pedestrians, and ordinary internet users, could comprehend. For instance, we can simply look at the issue from the point of view of the consumer, such as “Does recent news about how CP pilots may have been exhausted from too much work affect your trust on the operation of the airline?”