Best Of
11 Common Survey Errors That Will Ruin Your Data
Article written by Kate Williams
Product Marketing Manager at SurveySparrow
16 min read
23 May 2024
Best Of
Article written by Kate Williams
Product Marketing Manager at SurveySparrow
16 min read
23 May 2024
Nathan wanted to start a company that sold frozen breakfast for school kids.
And here’s the interesting part: he already knew that an online survey would be the quickest and cheapest way to perform market research and validate the demand for his idea.
So he set out to build a survey and instinctively asked his most curious questions:
“Would you like to avoid the hassle of cooking for your kids every morning?”
“Would you be interested in yummy, nutritious frozen breakfast kits for your kids?”
And the results were uplifting! It turned out most people had a need for the product he was looking to create.
In the next few months that followed, Nathan spent his time getting his startup off the ground.
Unfortunately, the product barely sold. Why? Well, his questions were biased, and the answers he got, as a result, did not accurately represent his target market’s opinions.
He was just another person who had succumbed to some of the most common survey errors. It turns out most people end up making some easily avoidable survey mistakes.
Have you ever heard the quote, ‘Half knowledge is dangerous?’ In the case of surveys, that couldn’t be more accurate.
To get reliable results from your surveys, you need to avoid making some common survey mistakes. Otherwise, you might find yourself taking misguided steps that can only harm you and your organization.
In this article, we’ll go through all the types of errors in surveying, so you can eliminate them from your own surveys.
The most common surveying errors in the world are question errors. Asking a question in a survey is not like asking one in real life. These questions need to be precise and well-considered. Here are the ways people go wrong in preparing survey questionnaires.
In regular conversation, we often tend to ask two questions at once:
“How are you doing? Do you feel happy?”
And our partners in conversation deal with this effectively. That is not true in surveys. The responses are in a fixed format, so respondents will get confused about which of the two questions they should be answering.
Let’s take one of Nathan’s survey questions, for example. “Do you want yummy, nutritious…” and there’s our survey error.
Yummy and nutritious are two different qualities. You want to know what exactly customers prefer, so you want to break that down into two questions. Otherwise, you won’t know which one your respondents chose to answer. This is the kind of surveying error that leads to bad, unusable data.
“Would you like to avoid the hassle…” was another of Nathan’s questions we mentioned.
You’ve already included a judgment in the question, so respondents will be led to a certain response. Maybe some parents don’t really think it’s a hassle, but because it’s a leading question, you will never find out. In journalism, as in surveys, leading questions lead to costly errors that obscure the truth.
How do you avoid such errors in surveying?
The most important thing is to do a judgment check on your survey questionnaire. Do any questions carry an implicit judgment within them that may not be true? In this case, could it be that some parents don’t find it a hassle? If yes, then drop that judgment.
Just ask, ‘Would you prefer not to cook for your kids every morning?’ The answers to judgment-free questions will be more honest and lead to quality data.
Estimating the preferences of an entire population based on a small sample is a tricky endeavor. There are many ways you could go wrong, and one of them is just picking the wrong sample, either based on size or demographics. Here are the survey errors that have to do with samples.
Incorrect sample size is yet another common surveying error. Picking a sample size is not supposed to be an intuitive guess. It involves accounting for multiple factors, including your target population, the margin or error, and other such figures. There are sample size calculators out there that can help you figure out your ideal sample size.
If you have too small a sample, the results you get from them will most likely not be representative of a larger population.
If your sample size is bigger than you need it to be, that’s not terrible, but still ends up unnecessarily costing you more to collect that data. For a range of factors, it is important to have a well-defined sample size and acquire enough responses to meet that number.
If you’ve included a certain number of people in your sample size and some of them don’t respond to some of the questions, that can lead to critical survey errors. What you want is to have enough responses for each question so that they can represent a larger population.
Without that number of responses, your data will be skewed, and hence not representative of the actual population.
Non-response errors can be avoided by making all your crucial questions mandatory so that respondents cannot pick and choose. It also helps to have a larger sample than your sample size, in case some participants fall out. To conduct a successful survey without sampling errors, it is crucial to consider the possibility of nonresponse bias.
This is a type of surveying error that you will come across more often in sensitive surveys, like political preferences or employee feedback. If you think people will be hesitant to reveal the truth in their responses, you should watch out for response bias.
Response bias is when respondents answer questions untruthfully, whether to avoid negative consequences or just out of fear and shame. For example, if you’ve asked an employee about the key shortcomings in their manager, it’s highly likely that you might receive an untruthful response.
What can you do about response bias?
One of the best ways of avoiding response bias is to make sensitive surveys anonymous. You might get more honesty out of your respondents if you do this. It is a slight compromise in accountability, but what would be worse would be to have untrue responses that misguide you.
Each case will be unique, and you need to take a call about whether you anticipate response bias, and how you might eliminate it.
Another survey error that can lead to response bias is the order of your questions. Believe it or not, it is imperative to be careful about your question order in your surveys. For example, say you’ve asked an employee to point out the problems with their manager.
After that question, you’ve asked them how happy they are at work. That response will definitely be influenced by the previous question.
This is how it is in conversation as well.
If someone’s been thinking about a particularly troubling topic, their thoughts on other matters will be tainted by it. That’s why you need to order questions in such a manner that it doesn’t lead to response bias. These survey errors easily slip past you and lead to incorrect and misleading data.
Another major source of errors in surveying is improper design. At Surveysparrow, we’re committed to getting the survey design right for you. Here are some common survey design errors to avoid while building your survey:
You’re conducting an employee engagement survey that spans your entire organization. You have specific questions in there about the accounting department, but it’s been sent out to all departments. That’s a coverage error. You don’t want an employee from human resources to be filling in questions for that department.
Often, coverage errors in surveys lead to skewed results. Whenever you have specific questions that might not be relevant to your entire sample size, come up with a plan for them. With Survesparrow, you can have those questions show up only if the respondent says that they’re from accounting. Such branching can help you avoid coverage errors.
This is another of those survey errors that you see everywhere. People who design the surveys ask close-ended questions and come up with a set of responses that they think will work for everyone. But often, you want to respond in a way that is not in the options.
What does this lead to? You pick the closest thing in the responses, which is still not your accurate response.
Inadequate response options skew the results, and you don’t get truthful responses. Often, these types of survey errors are related to design. Maybe an open-ended response was more appropriate for that specific question. When designing a survey, be careful to avoid these errors.
In the end, what you want from your surveying exercise is reliable data. But in that endeavor, there are many potholes on the way. Avoid these survey errors related to data collection for rigorous results that you can rely on.
If I had a penny for each time I was asked to type my email address in a survey that was already collecting my email address, I would be quite rich.
Often, surveys include requests for information that you already have. The mentality is that it can’t harm, but it does. The longer your survey, the more time it takes to fill, and the more people abandon it midway.
With questions regarding demographic information, you need to have a conservative mindset and only enquire about demographics that really matter. It’s the same with other questions – get rid of anything that feels kinda nice to have but isn’t necessary. Your survey should be driven by purpose, sharp, and to the point.
Another common type of survey error is assigning the wrong input method for questions.
Of the two kinds of questions – open-ended and close-ended – you want to choose carefully. And even amongst them, what’s the character limit on your open-ended question? Is it single-choice or multiple-choice? These decisions require a lot of thought.
You don’t want to make an open-ended question out of what could have been a close-ended one. Respondents have to invest more effort into answering them, and the responses aren’t easily analyzable. With each question, go over it and ask yourself if the current input method is the best one there is.
A personalized walkthrough by our experts. No strings attached!
When it comes to online surveys, you want to get accessibility just right. Here are ways to avoid survey errors that can occur if you don’t consider the accessibility of your surveys.
If you anticipate that your respondents are going to be on a phone, your survey needs to be optimized accordingly.
At Surveysparrow, we have templates for beautiful, functional mobile-first surveys. You want to use an online survey tool that gives you this functionality. Few things are worse than having to fill in a survey made for desktops on a phone.
When optimizing for mobile, you want to make sure that your input boxes are good enough for touch as opposed to a cursor. Of course, these are considerations you should leave to your online survey tool. We would wholeheartedly recommend Surveysparrow for mobile-optimized surveys that’ll get you more responses.
In some situations, you might need respondents to listen to a clip or watch a video before answering a question.
In these cases, you can use the online surveys from SurveySparrow. These are easy to pull off but come with certain considerations. Are your respondents going to be in environments to be able to listen to clips? If not, you might lose out on responses.
When thinking about accessibility, also consider the age of your respondents. Will old people be able to figure out how to navigate your online survey? In order to avoid accessibility-related errors in surveying, it is crucial to think about such questions.
The quality of responses you get on your survey will depend on how you market it. At Surveysparrow, we have tools to help with email and social media marketing. Marketing survey errors are common and can affect the number of responses you get.
If you’re conducting a survey, the first thing you need is a mission statement. In all your marketing material, the purpose of your survey must be communicated unambiguously.
Why will anyone give you their time unless they know exactly what it’s leading to? Communicating the importance of your goal can be a great way to get people to respond.
On the other hand, if the purpose of your survey is unclear, that’s one of the most crippling survey errors. You will inevitably get a low response rate. We’ve come across countless such surveys that just expect respondents to give their time arbitrarily. In your marketing material, make your purpose clear.
We’ve read this survey email subject line so often, ‘Give us a few seconds and get amazing offers!’ At this point, such marketing copy just puts us off. If you don’t tell people what exactly they get in return, how do you expect them to take action?
‘Amazing offers’ is as ambiguous as it gets. Is it a discount? A free helicopter? What are we talking about here?
People might fall for the trap once, but surveying is a long-term game. You will want to repeat your survey, and those people who discovered there was nothing at the end of the tunnel will never enter again.
In your marketing copy, tell respondents exactly what they get – a 10% discount. That’s better. It’s respectful and helps build trust with your respondents.
This type of error arises when the pool of survey respondents doesn’t accurately represent the target audience.
For example, if the survey depends on volunteers, then it will attract those with a strong interest in the survey topic. This classification can be broad and, therefore, can skew the results.
How to tackle this?
Try implementing strategies to attract and encourage respondents of your target population. Incentivizing your surveys is one way you can improve your response rate.
Using screening questions is another way you can try. Following this approach can ensure that the respondents meet the necessary criteria for the survey’s objective.
What if Nathan asked some questions that didn’t align with his survey intent? Well, that’s when this survey error happens.
The ‘good to know’ questions fall under this error since these questions may not accurately measure what the survey intends to do. We understand the urge to gather as much information as possible.
However, adding unnecessary questions can lead to participant fatigue and reduce the overall quality of the responses. To prevent purpose creep, it’s important to stay focused on the survey’s main goals.
Also, resist the urge to add “nice-to-have” questions that do not directly contribute to the research objectives.
Measurement error occurs when the questions asked do not accurately measure what they are intended to. This can result from ambiguous wording, poorly designed response options, or inconsistent instructions.
An example would be the following question – “How would you rate your satisfaction with our customer service?” And suppose the responses to this question are –
This design can lead to error because it lacks a neutral option and doesn’t capture the full range of possible responses. A better approach would include a more comprehensive set of responses, such as
To avoid this type of error, it’s vital to craft clear, concise, and neutral questions. Also, employing consistent scales and providing respondents with precise instructions can help minimize measurement errors.
A personalized walkthrough by our experts. No strings attached!
To conclude,
Suppose you’ve decided to conduct your survey. Great job! You’re on your way to getting insights that can help you channel your energy productively.
But as we mentioned, half-knowledge is dangerous. In order to get useful insights, avoid these 11 common survey errors you can make at every step of your survey. Also, as soon as you have your data, don’t forget to back them up right away. We hope that knowing about these errors will help you create better, more reliable surveys.
From an error-ridden survey, the data you get will be, at best, useless and, at worst, misleading. There are real, tangible dangers of a half-hearted survey exercise. If you focus on avoiding these types of survey errors, you’ll make sure your data is robust. Follow these guidelines and conduct a rigorous survey that helps you tackle your organization’s future with confidence!
Thousands of brands trust SurveySparrow to turn feedback into growth. Try it free today!
Best Of
13 MINUTES
18 April 2020
Best Of
10 MINUTES
25 January 2021
Best Of
13 MINUTES
27 May 2021
Best Of
12 MINUTES
5 April 2020
