Biased Survey Questions: 7 Examples & How to Minimize Them

Previously, we discussed bias in surveys, the impact of different types of bias, and how to fix them. Now, let’s talk about another major contributor to misleading survey results: biased survey questions. 

Just like in life, the way you phrase questions can lead to totally different outcomes. So in this article, we will talk about:

• What are biased survey questions?
• The seven types of bias in survey questions – and how to minimize or avoid them altogether. 

What is a biased survey question?

A biased survey question can be caused by word choice or formatting errors, question type selection, or survey design. Whether the mistakes are intentional or not, the outcomes are always the same:

• Leading the respondent toward a particular answer.
• Confusing the respondent so that the answer is incomplete and untrue at worst. 

There’s an old saying, “For want of a nail..” Biased survey questions might seem like a minor detail compared to how the survey looks and feels. However, one survey question error can have a major impact. 

Best case scenario: you get called out by your customers and employees. Worst-case scenario: Skewed data that can lead to potentially disastrous decisions. 

Survey templates are an effective tool for saving time and effort on survey design. Try our survey template library, easy-to-use UI and multiple share channels for 14 days absolutely free.

Start your free trial now.

How do you identify biased survey questions?

As the survey creator, your job is to keep bias to a minimum for honest, thoughtful responses. So how do you do that? One way is to know how different types of biased survey questions work and take action to minimize them. 

7 types of bias in survey questions

• • Question order bias
  • Leading question bias
  • Loaded questions
  • Double-barreled questions
  • Vague questions
  • Absolute questions/dichotomous questions
  • Acquiescence bias

#1. Question order bias

Question order bias comes in two forms:

1. The question order and sequence

The order of the questions in a survey impacts responses to other questions – especially if the first question is rather sensitive or specific.

For example, in this Cold War experiment, US respondents were surveyed on their attitudes to cross-border travel for American and Soviet reporters. The experiment found a significant question order effect. 

In short, people were more open to admitting Soviet reporters into the US after answering a question about allowing American reporters into Soviet countries. But when the question order was switched, people were less likely to want Soviet reporters in the US (and vice versa).

2. The bias in questions when they are not randomized. 

This happens in surveys where images are critical, like concept testing. This refers to improving a concept – a logo, product idea, ad campaign, etc. – by collecting feedback from your target audience. 

How to fix question order bias

• Make your questions go from general to specific. You can check out some of our survey templates – like this one for health insurance surveys. They will give you an idea of how the questions should flow. 
• Randomize your question order if it makes sense to do so. This heatmap survey example has a randomized order to avoid “straight-lining” – clicking on the same parts of all images. 

#2. Leading question bias

Leading question bias is a type of cognitive bias that occurs when the wording of a question influences the answer. Basically, putting words in the respondent’s mouth. It is also called “leading statement bias” or “leading hypothesis bias.”

An example of leading question bias is when employees are asked, “What are your issues with the agency?” or when customers are asked, “How great is our service?”. Both questions include assumptions and emotional language. This is a huge no-no because if people feel they are being manipulated, it can lead to higher survey drop-off rates. 

How do you fix leading questions?

• Use neutral, objective wording. This is how we would fix the examples above:
  

“How likely are you to recommend working with our current agency?”

“Please state your reasons.”

“How would you rate our service?”

• Keep out emotions and judgments from questions and answer options. 

#3. Loaded questions

Loaded questions are questions that are designed to elicit a specific response. They are often used in surveys to get the desired answer from the respondent. 

In this regard, leading questions and loaded questions are similar. The difference tends to be in the answers. Loaded questions force respondents to answer in black and white, “yes” or “no”, which puts them on the defensive. Leading questions leave room for an open-ended reply. 

How do you fix loaded questions?

• Change the wording of your question to reflect possible scenarios. For example, you can ask, “How often do you feel this way?” instead of “Do you feel this way?” 

• Place the loaded question in the correct sequence. 

For example, when you ask an employee, “How often do you take a break twice a day?” it should ideally be the follow-up to: “How many breaks do you take in a day?” But if the question isn’t relevant, you can use skip logic. 

• Use opinion scales, sliders or MCQs instead of yes or no. If you need to use Yes\No, add an ‘Other’ option to opt-out. 

#4. Double-barreled questions

Like a double-barreled shotgun, double-barreled survey questions ask for two different pieces of information in one question. They are often used in surveys to save time but can lead to confusion and inaccurate data.

To spot a double-barreled question, look for ‘and’, ‘or’ or even a ‘?’. Here’s an example from a library survey: “When do you prefer to use the library? Which day do you prefer the most while visiting?“

How to avoid double-barreled questions

• The best way to fix double-barreled questions is by splitting them into single-barrel questions. For example: When do you prefer to use the library?

• This will allow the respondent to answer with only one piece of information. Then, it’s easier for them to answer the question accurately.

#5. Vague questions

Vague questions are a common problem in surveys. Unlike leading and loaded questions, they leave too much room for interpretation. In that case, you won’t get coherent results since the range of answers will vary wildly.

Here’s an example of a vague question: “Did you find this blog helpful?” One, it doesn’t tell you what the reader found helpful. Two, you may define helpful as “informative and fun,” but the reader may define it as “something that helps me get X done.”

How to avoid vague questions

• Reframe the question to focus on the metric that matters to your survey goal. So we’ll rephrase the example above like this:

“What were your key takeaways from this blog?”

• Ask for specific information that the customer or employee may know. When you ask your customers, “How does our service compare to our competitors?” they may not respond because they don’t know who your competitors are.

#6. Absolute questions/dichotomous questions

Absolute questions are problematic because they restrict the respondent’s answer options to Yes/No or ‘True/False. This forces the respondent to pick an option that might not reflect their honest opinion. That is how absolute question bias skews your data. 

Absolute questions also include absolute terms, like all, none, must, every, always, just, only, never…you get the drift. 

How to avoid absolute question bias

• Avoid absolute terms unless you need them.
• Provide more options. Suppose you were marketing a lunchbox service to office workers in your city and you wanted to know about specific lunch habits:
  

#7. Acquiescence bias

Think of every user agreement you’ve ever read. They’re long, tedious, and make you want to hit ‘Agree’ straightaway! In surveys, that is known as the acquiescence or agreement bias. 

Acquiescence bias happens when a respondent loses interest in the survey. So they zip through it and mostly choose positive responses in the process. This can give the impression that everything’s going well. 

How to avoid acquiescence bias

• Break the pattern. Choose different question types to keep things interesting, like image choice, MCQs, sliders, matrix, voiceover, etc. 
• Respond with a thank you message to those who complete the survey. 

Wrapping Up

We’ve talked about seven types of biased survey questions. We’ve also shared our top tactics for minimizing them. But if you want to know more about writing better questions, here’s a guide on writing survey questions that will give you the exact responses you’re looking for. 

Happy surveying!