Beware of Biased Comparisons

Everyone knows not to cherry-pick your data, which means to select only the evidence that supports a pre-determined conclusion, while ignoring the remainder. When we make a commitment to tell true and meaningful data stories, we agree to keep an open mind, examine all of the relevant evidence, and weigh the merits of competing interpretations. If you agree to these principles, then also watch out for biased data comparisons, especially sampling procedures that may appear legitimate on the surface, but actually contain less-visible factors that skew the evidence. While we may believe we’re operating with open minds, we can overlook partially-hidden processes that effectively cherry-pick our evidence without our knowledge.

Selection bias happens when we believe we have chosen our data sample fairly, but some behind-the-scenes process influences its composition and skews our analysis. For example, if you conduct surveys by email with US adults, your sample will not be representative of senior citizens aged 65 or older, who are less likely to own a computer or smartphone, according to the Pew Research Center. Also beware of participation bias, sometimes called non-response bias. If your survey has a low response rate, it is likely that those who choose to respond possess certain qualities that make them less representative of the general population.

In particular, self-selection bias often arises when attempting to evaluate the effectiveness of a particular program or treatment where people applied or volunteered to participate, as shown in Figure 6.2. Imagine that your job is to determine if a weight-loss program actually works, which requires a deeper understanding of how data samples were chosen. Avoid the mistake of comparing the progress of non-participants (group A) versus participants who signed up for this program (group B), because those two groups were not randomly chosen. Participants differ because they took the initiative to join a weight-loss program, and most likely have higher motivation to improve their diet and exercise more often than non-participants. Self-selection bias secretly shapes the composition of both groups, which results in a meaningless comparison. We often fool ourselves and overlook how this voluntary participation skews our understanding of program effectiveness, whether the topic is weight-loss clinics, counseling programs, or public and private school choice.

How can we reduce self-selection bias in program evaluation data? As you learned in Chapter 4, it’s important to question your data by looking below the surface level to fully comprehend how terms have been defined, and how data was collected and recorded. By contrast, a well-designed program evaluation will reduce self-selection bias by randomly dividing all volunteer participants (group B) into two sub-groups: half will be assigned to participate in one weight-loss program (group C) and the other half will be assigned to a different weight-loss program (group D), as shown in Figure 6.2. Since sub-groups C and D were selected by chance from the same larger group of volunteers, we can be more confident when comparing their progress because there is no reason to suspect any difference in motivation or other hard-to-see factors. Of course, there are many more research design details that are beyond the scope of this book, such as ensuring that sample sizes are sufficiently large, and comparing participants before, during, and after the weight-loss activity, and so forth. But the logic of avoiding selection bias is simple: randomly divide people who apply or volunteer to participate into sub-groups, to better compare program effectiveness among people with similar motivations and other hard-to-see characteristics.

Do not compare program non-participants (A) versus those who apply or volunteer to participate (B). Instead, randomly split all participants into two sub-groups (C and D) to compare program effectiveness.

Figure 6.2: Do not compare program non-participants (A) versus those who apply or volunteer to participate (B). Instead, randomly split all participants into two sub-groups (C and D) to compare program effectiveness.

Bias warnings appear in several chapters of this book, because we continually need to be aware of different types that negatively influence our work at various stages of the data visualization process. Later in Chapter 15 you’ll learn how to recognize and reduce other types of biases when working with data, such as cognitive biases, algorithmic biases, intergroup biases, and map area biases.

Summary

Although we do not claim to teach you statistical data analysis in this book, in this chapter we discussed several common-sense strategies to make meaningful comparisons while analyzing your data. You learned how to use words more precisely for comparing data, why and how to normalize data, and advice on watching out for biased comparisons. In prior chapters you built up your skills on refining your data story, working with spreadsheets, finding and questioning data, and cleaning up messy data. Now you can combine all of this knowledge and begin to create interactive charts and maps in the next few chapters.