One thing I learned in my Ph.D. program is that academics love to use fancy terms to explain pretty straightforward research methods and concepts. Some of those fancy terms have wormed their way out of academia and into everyday conversation like rigor, validity, and significance. In some ways, this is a great thing – information about research methods shouldn't stay locked up behind ivory towers and paywalls. But, like any game of telephone, as these terms move from academia to foundations and nonprofit organizations, their meanings can get a little muddled.
The DataTalk series aims to give meaning to these research terms and show nonprofit organizations how to apply them to their surveys, focus groups, and other research projects. This article will cover what validity means, some common research mistakes, and steps nonprofit leaders can take to ensure validity in their organization's work.
Validity is about avoiding mistakes
Validity is essentially about accuracy. Ensuring validity means that the project team has taken steps to avoid making common mistakes in research design, data collection, and data analysis. It also involves transparency, meaning documenting data processes and being transparent about limitations.
Mistake #1: Choosing the wrong groups
One common data collection mistake is choosing the wrong group(s) of people to participate in a project or excluding groups that should be involved. For instance, say a nonprofit leader wants feedback from staff about an upcoming strategic planning retreat. However, the executive director only sends a feedback survey to directors, assuming that the directors can speak for the rest of the staff. That is a data collection error; the ED should have sent the survey to all the staff so everyone could have the opportunity to share their views.
Mistake #2: Choosing the wrong methods
Another common mistake is choosing the wrong tool or method for the job. Take employee wellness as an example. There are many ways to better understand employee wellness in an organization, including surveys, interviews, or focus groups. However, some methods are better than others.
For instance, if a program director decides to do one-on-one interviews with their employees about their mental state, employees may not feel comfortable telling their boss about their personal struggles. Instead, employees may withhold information and only focus on the positives. Instead, nonprofits can ask employees to fill out an anonymous survey to increase the odds of more honest answers.
Mistake #3: Forgetting to define terms
A third common mistake happens when the project team doesn’t clearly define the terms or concepts they want to study. This mistake happens all the time, especially in survey research. Again, take the employee wellness example. There are many different definitions of employee wellness and many factors that contribute to it. What the executive director thinks wellness is could be different than directors, human resources, and program staff. Without taking the time to define “wellness,” the project team is more likely to ask irrelevant or unclear questions that could negatively impact the results.
Four steps to increasing your project's validity
1. Make sure your team thinks through your research questions and comes to a joint agreement on any terms that could be interpreted in multiple ways (e.g., “success,” “vision,” or “impact”).
2. Identify who needs to participate in the project. Participants should be the people who are best suited to answer your team’s research questions. Be exhaustive and specific, listing reasons why each group should or should not be included in a survey, interview process, or focus group.
3. Brainstorm ways to collect data and write down the pros and cons of each. Be mindful of participants’ experiences as you go through this process. What is the best way to make your participants feel comfortable and give open, honest responses?
4. Create a research project plan and document any changes made throughout the process. This plan will help your team explain what you did and how you did it when it’s time to share the results.
Need help creating a plan that ensures validity in your results? Contact me at lindsay@morgiaresearchservices.com or schedule an appointment here.