When nonprofit leaders come to me with project ideas, one of their concerns is ensuring the research is statistically significant. However, there's often confusion about what this phrase really means. And while "statistical significance" carries a certain gravitas, nonprofits don't always need it to get valid, high-quality results. In this post, we'll go over:
What statistical significance means
The steps needed to achieve it
Times when statistical significance isn't relevant or necessary.
What is statistical significance?
Statistical significance is a type of measurement that tells us how confident we can be that the results of our analysis are accurate and did not happen by chance or by accident - an "oops," if you will. This measurement is only used when we are:
Using quantitative data, such as survey data
Testing the relationship between two things, such as the impact of education levels on individual incomes.
Typically, number-crunchers will use a regression analysis to calculate how different factors, like education levels, affect outcomes, such as income level. This type of analysis provides information about the statistical significance of the results.
What it isn't
Statistical significance is not a value judgment. Having it does not make your project inherently good, and not having it does not mean your research is bad or wrong. It is simply a tool we can use in certain circumstances to measure confidence in our results. Also, if your nonprofit's project doesn't involve numbers and relationships, there is no need to worry about achieving statistical significance. Feel free to skip to the end!
Steps toward statistical significance
1. Write out the hypothesis
If a regression analysis makes sense for your project, the first step is to write a hypothesis (remember those?). The hypothesis is your best guess about the relationship between two or more factors. For instance, if you want to know the relationship between income and education level, you'll want to write down what you think the results will be based on your experience, existing research, and (sometimes) gut instinct.
2. Find lots and lots of people
The second thing you'll need is lots and lots of participants. For statistical significance, the bigger the sample size, the more likely you can be confident in your results. Also, you'll want to ensure that there are many different types of people in your sample. For example, suppose you're looking at race/ethnicity and test scores. If your data includes 995 white people and five people of color, odds are your results will be incorrect because of the lack of racial and ethnic diversity.
3. Get lots of other data
The third thing you'll need is lots of other data. When figuring out the relationship between two things, we must consider all the other factors that could influence our results. For instance, in the education and income example, we would need lots of data on zip codes, gender, race/ethnicity, parents' income, and a whole bunch more to ensure that education level is the main factor influencing income. Without including these additional variables, there's a higher risk of getting incorrect results.
When statistical significance isn't...significant
In most cases, the nonprofit clients I work with do not need to worry about statistical significance. For instance, many organizations are seeking answers to open-ended questions, such as:
- What do our clients think about our new program?
- What do staff members want to focus on at our next strategic planning meeting?
- How do our managers feel about implementing our remote work policy?
Because these questions are not about comparing relationships between factors, they don't meet the criteria for statistical significance. Other times, there are not enough people or data on other factors to do a regression analysis. But guess what?
That's totally okay. I promise!
For many nonprofits I work with, the goal of using data is to identify trends to guide decision-making. To do so, we collect data using surveys, interviews, focus groups, and publicly available information. Then, we identify patterns and relationships using other analysis methods that make the most sense for our types of projects. In these cases, a lack of statistical significance doesn't mean that the results are not valid, sound, or critically important. It just means that the projects didn't meet the criteria for a specific analysis method.
So, to wrap up – don't sweat the statistical significance. For most nonprofits, what matters more is knowing how to design your project the right way to get the answers you need for better programs, policies, and organizational cultures. If you need help planning your next project or choosing the right methods for the job, click the button below to schedule a free 45-minute consultation call today!
Comments