Understanding scores on questions and themes in Surveys
The Engagement survey suite (engagement, onboarding, exit, and pulse surveys) calculates question and theme scores based on a positive response score.
There are three scores in engagement: overall, theme, and question. Each score should be thought of as a separate lens or view from which to understand your results data. It is natural to try to aggregate question scores into theme scores and theme scores into the overall score, but because each has its own methodology, such an aggregation exercise may result in confusion. Below is an overview of how each score can be used and how each is calculated.
This article contains the following topics:
- Analyzing scores
- Overall score
- Theme scores
- Question scores
- Questions or themes without scores
- Unsubmitted surveys
Overall and theme scores are important for top-level trend analysis and spotting potential areas of concern. Lower scores are a sign to zoom into question-level data and begin looking at heatmaps and question-level breakdowns (positive, neutral, negative). When we start the conversation from the context of people’s overall sentiment, we can take a broader view of our action planning and start to truly understand the employee experience.
The overall score is the highest level view and distills the survey down to one number, which is the average of all the question scores. It should be used to understand overall, what percentage of people are feeling positive or engaged. Year over year, you can use this number to see how responses are changing. The higher the score, the more positive respondents you have.
Overall scores are calculated as an average of all question scores for questions included in that survey. Question scores are measured as a percentile of their positive responses.
In a survey that had 3 questions with question scores of:
- Question 1: 50
- Question 2: 60
- Question 3: 70
The overall score would be [ (50+60+70) / 3 ] = 60.
A theme score is a snapshot of the people who responded to that theme; it captures how many people responded positively to that theme. The score of a theme that contains multiple questions is not the average score of all the questions within the theme.
Surveys use a different methodology than question scores so that the score for a theme does not outweigh the responses from people who have answered all the questions within the theme versus those who only answered one.
Surveys use the following steps to calculate theme scores:
- Surveys look at employees who have answered at least one question in a given theme.
- For each employee who meets that requirement, surveys calculate the average score for submitted questions they responded to in the theme. The average score is calculated based on the agree/disagree scale where strongly disagree = 1 and strongly agree = 5. This is the score for the employee.
- If the score for the employee is higher than 3.5, this is considered a positive response to the theme.
- The total number of employees who scored at least 3.5 and are therefore considered positive responders, for the theme is divided by the total number of employees who responded to the theme. This is the theme's score.
- 1-2.4: negative
- 2.5-3.4: neutral
- 3.5+: positive
For a particular theme with two questions, 100 people in total responded. The response breakdown is as follows:
- Employee 1
- Question 1: Strongly agree
- Question 2: Agree
- Employee 2
- Question 1: Agree
- Question 2: Neutral
- Employee 3
- Question 1: Disagree
- Question 2: Agree
The first step is to calculate the average score for each employee:
- Employee 1: (5 + 4)/2 = 4.5
- Employee 2: (4 +3)/2 = 3.5
- Employee 3: (2 + 4)/2 = 3
Next, employees who are considered positive scorers are added and then divided by the total number of employees who responded to the theme.
- 2/3 = 0.66
- 0.66 ~ 67
The theme score is 67.
Here’s an example of why a numerical calculation could cause skewed representation:
- A theme consists of 3 questions.
- John answers all 3 questions and the responses are all positive.
- Cameron answers all 3 questions and the responses are all positive.
- Stacey answers only 1 question and the response is negative.
Theme score based on numerical calculation: 6 positive responses over 7 total responses = 86%
In this case, Stacey’s voice would no longer be equal to John's and Cameron’s simply because she answered fewer questions. In other words, it’s no longer “one person one vote”.
Reality: 2 out of 3 people feel positively about this theme (67%), which is not accurately captured by the above mentioned 86%. So while you might think that 86% is not bad, in reality, the score is significantly lower.
Each question score is a snapshot of how many people responded positively to that question. There are no averages here. If a person responds to a question positively, they are counted in the score.
For a question that uses the agree/disagree scale, scores are calculated in the following way:
- Agree or Strongly Agree = Positive Responses
- Disagree or Strongly Disagree = Negative Responses
- Neutral = Neutral Response
For a particular question, 100 people in total responded. The response breakdown is as follows:
- Strongly agree: 30 people
- Agree: 20 people
- Neutral: 10 people
- Disagree: 25 people
- Strong disagree: 15 people
- No response: 50 people
For this question, the positive score would be (30 + 20) / 100, which calculates out to 50%.
Questions or themes without scores
A question or theme will not have a score associated with it for two possible reasons:
- The question is set as solely a comment question, meaning that responders were prompted for a free text response rather than a scale rating. In this case, there is no score to show. The same applies if a theme consists only of questions that are comment questions.
- A "No opinion" answer indicates participants skipped the question without selecting a response.
- There aren't enough responses to a question or theme to show the score. This is dependent on the anonymity threshold set when creating the survey. For example, if the survey has an anonymity threshold of 3, a question that only has two responders will show no score to protect the anonymity of the responders.
Responses that have not been submitted when the survey is ended will not be pulled into survey analytics.