Understanding statistical significance in engagement surveys and pulse
Statistical significance is an important concept for analyzing Pulse & Engagement Survey scores over time. This is also particularly important to consider when comparing results from different populations & attributes within your data.
The certainty (%) is the confidence in the accuracy of your Survey results. In this case, the confidence level of 95% certainty allows us to say that we are 95% sure that this sample of employees is representative of the population. To break it down further, we can say that if we run this survey 100 times, we’d expect results to match what we’re seeing now ± margin of error at least 95 times.
The margin of error (MOE) is a statistic that predicts the amount of the random sampling error in survey results. To calculate the MOE, you need to multiply the % by the scale. For example, if the average for a given question is 4, and your MOE is ± 10%, then there’s a 95% chance that the full population’s average is between 3.5 and 4.5.
As an Admin, you may be wondering how many responses you need to receive for there to be value in distinguishing the scores. After you launch your survey, you can take a look at the graph to see what your margin of error is. The graph shows the relationship between the size of the population and what % of that population needs to respond in order to have a given margin of error.
As an example, let's say 75% of people have filled out your Engagement survey. You can use the graph above to see what your margin of error is (based on the population of employees at your company). In this example, we'll use a company of 100 employees. If your employee population is 100 and you want to have a margin of error of ± 5%, you'd need to have 80 people respond to that survey (light blue line above). If 50% of people filled out the survey, then the margin of error is ± 10% (yellow line above).
A few more things to note here:
- This does not mean that if you have fewer responses, you will not have usable results – the margin of error will just go up. The margin of error is particularly important when looking at results over time.
- The amount of times a question is asked is less important than the number of people who respond to it, and what that latter number represents as a percent of the total employee population.
- If you are planning to do any slicing on the data, e.g. gender, department, etc, that will have an impact on the statistical significance for that subset of employees. For example, if your company has 200 employees split evenly between men and women, then you'll be looking at a population of 100 of each (for this attribute). This will then change the margin of error due to the decreased population size.
Increase Survey Participation
As an Admin, you can write a reminder to employees to submit their surveys. The reminder will only go to employees who have not yet submitted their surveys. Learn more on how Remind Survey Participants to Complete Their Survey.