When nonprofit leaders come to me with project ideas, one of their concerns is ensuring the research is statistically significant. However, there's often confusion about what this phrase really means. And while "statistical significance" carries a certain gravitas, nonprofits don't always need it to get valid, high-quality results. In this post, we'll go over what statistical significance means, steps to achieve it, and when it isn't relevant or necessary.
What is statistical significance?
Statistical significance is a type of measurement that tells us how confident we can be that the results of our analysis are accurate and did not happen by chance, accident, or error. This measurement is only used when we are:
- Using  quantitative data, such as survey data
- Testing the relationship between two things, such as the relationship between race/ethnicity and standardized test scores or the impact of education levels on individual incomes.
Typically, number-crunchers will use a regression analysis to calculate how different factors, like education levels, affect outcomes, such as income level. This type of analysis also provides information about the statistical significance of the results.
What it isn't
Statistical significance is not a value judgment. Having it does not make your project inherently good, and not having it does not mean your research is bad or wrong. It is simply a tool we can use in certain circumstances to measure confidence in our results. Also, if your nonprofit's project doesn't involve numbers and relationships, there is no need to worry about achieving statistical significance. Feel free to skip to the end!
Steps toward statistical significance
The hypothesis
If a regression analysis makes sense for your project, the first thing you will need is a hypothesis (remember those?). The hypothesis is your best guess about the relationship between two or more factors. For instance, if you want to know the relationship between income and education level, you'll want to write down what you think the results will be based on your experience, existing research, and (sometimes) gut instinct. In this case, the hypothesis might say, "Individuals with a college degree will have higher incomes than those without one."
If coming up with a hypothesis looks like this, it might be time to take a break!
Lots of people
The second thing you'll need is lots and lots of participants. For statistical significance, the bigger the sample size, the more likely you can be confident in your results. Also, you'll want to ensure a lot of variety in your sample. For example, suppose you're looking at race/ethnicity and test scores. If your data includes 995 white people and five people of color, odds are your results will be incorrect because of the lack of racial and ethnic diversity. Also, if almost everyone in the study scored 90% or higher on the exam, then there is not enough variety in test scores to get accurate results.
Lots of other data
The third thing you'll need is lots of other data. When isolating the relationship between two factors, we want also to consider all the other things that could influence the results. For instance, to understand the relationship between test scores and race/ethnicity, we also have to account for what else may influence test scores, such as classroom ratios, zip codes, gender, household income, and a whole bunch more! Without including these factors, there's a higher risk of getting incorrect results.
Warning: sometimes, the data list can look like this!
When statistical significance isn't...significant
In most cases, the nonprofit clients I work with do not need to worry about statistical significance because their projects don't meet the criteria described above. For instance, many organizations are seeking answers to open-ended questions, such as:
- What do our clients think about our new program?
- What do staff members want to focus on at our next strategic planning meeting?
- How do our managers feel about implementing our remote work policy?
Because these questions are not about comparing relationships between factors, they don't meet the criteria for statistical significance. Other times, there are not enough people or data on other factors to do a regression analysis. But guess what?
That's totally okay. I promise!
For many nonprofits I work with, the goal of using data is to identify trends to guide decision-making. We work together through surveys, interviews, focus groups, and publicly available data. We identify patterns and relationships using other analysis methods that make the most sense for our types of projects. In these cases, a lack of statistical significance doesn't mean that the results are not valid, sound, or critically important. It just means that the projects didn't meet the criteria for a specific analysis method.
So, to wrap up – don't sweat the statistical significance. For most nonprofits, what matters more is knowing how to design your project the right way to get the answers you need for better programs, policies, and organizational cultures. And there are plenty of ways to do this that don't require complex statistics.
Need help figuring out how to design your next project? Contact me at lindsay@morgiaresearchservices.com to talk about your next great idea.