DataTalk: The B-Word - Bias in research and how nonprofits can avoid them

When we see the word "bias," many of us think about issues like discrimination, racism, and sexism. We may also think about how biases shape how we interact with each other and the systems that surround us. In research, there are lots of examples of how racial and gender bias (conscious and unconscious) have resulted in detrimental outcomes for marginalized communities. As such, we must always be aware of and correct how our social biases influence our work.

Other lesser-known biases can also create problems for even the most experienced analysts. For nonprofits interested in using data for strategizing, evaluation, and advocacy, it is essential to understand how these biases work and how to avoid them. These mistakes can "misinform decision-making processes, leading to ineffective interventions, policies, or therapies." Since no nonprofit leader wants that, this article covers four types of biases, how they can appear in a nonprofit's work, and ways to reduce biases in future projects.  

Selection bias

Selection bias occurs when some groups of people who should be a part of a research project are left out of the process, intentionally or unintentionally. For example, imagine an executive director who wants staff feedback for an upcoming strategic planning session. She creates a survey to get ideas for what topics to cover at the meeting. However, she only sends the survey to senior staff and directors, assuming they can speak for the entire staff. That's a selection bias error – by only including high-level staff, she will miss out on ideas from the rest of the employees that could be critical for the organization's growth and development.

Response bias

Response bias occurs when we ask questions in ways that influence people to give false responses. One way this happens is when questions are complex or confusing. For instance, questions that include a lot of jargon or unfamiliar acronyms can make it difficult for people to give a proper answer. And as we know, the nonprofit sector loves acronyms!

Also, response bias sometimes occurs when a person wants to give a socially appropriate or favorable response. Let's take a client feedback example. If a program staff member is interviewing a participant and asks them if they enjoyed the program, the client may be inclined to respond "yes" to be agreeable, even if their experience was not good. The power dynamics may be such that the participant does not want to risk upsetting the staff member, so they give a false answer to keep the peace. That's bad news for the client, as they don't feel comfortable expressing their concerns, and bad news for program staff, who will miss essential feedback that could strengthen their services in the future.

Procedural bias

Procedural biases occur when something about the process of collecting data influences the results. For instance, program leaders may be tempted to mandate that participants complete a feedback survey. However, this isn't always the best idea - people may fill out the survey as quickly as possible to get through it without taking the time to think through their answers.

This bias can also occur if people do not have enough time to complete a survey. For instance, if the executive director from the earlier example sends the survey one day before the strategic planning meeting, a few things will probably happen:

- Some staff will rush through the survey.

- Some staff won't have time to fill it out.

- The executive director will not have enough time to analyze the results before the session.

That's too many potential errors to make a survey worthwhile.

Confirmation bias

Confirmation bias occurs when we think we already know the answers to our questions, choose data that fits our conclusions, and ignore anything that doesn't fit (consciously or unconsciously). In my opinion, confirmation bias is one of the most dangerous biases, especially for advocacy organizations. In my previous life as a policy analyst, I saw situations where organizations chose a policy they wanted to advocate for and then worked backward to find data supporting their position. However, they never considered any evidence that contradicted their position. Without considering contrary data, organizations risk advocating for policies that could have unintended consequences for the populations they are trying to help.

How do we get rid of bias?

Unfortunately, no research project is bias-free. Despite our best efforts, there are sometimes limitations, resource restrictions, or simply blind spots that prevent us from doing "perfect" work. However, the first step in reducing biases is being aware of them and correcting them as we go.

As your organization prepares for your next project, list everyone who is best suited to answer your survey, interview, or focus questions and invite them to participate. Be as inclusive as possible and always ask, "Who's missing?" Keep your questions simple and ensure participants have enough time to complete your survey or sign up for your focus groups. Be mindful of power dynamics and take steps to create environments that make participants feel comfortable giving honest answers. Finally, when it comes time to analyze, report all of the findings, not just the positive ones or those that match our gut feelings. Honesty and integrity in data collection and reporting are critical for any organization looking to build its reputation as a leader in their field.

Learn more about how MRS supports nonprofits