Start with a review, not a rejection rule
A survey response that looks unusual is not necessarily careless. Someone familiar with a topic may answer quickly. A respondent may choose the same rating for several items because they genuinely hold a consistent view. Even an apparent contradiction may come from unclear wording or a change in the time period being asked about. Treat quality checks as a way to identify responses for closer examination, not as an automatic deletion tool.
Before collecting responses, decide what information you will use to assess quality and how you will handle uncertain cases. Separate clear eligibility rules, such as a respondent not meeting the survey’s stated criteria, from softer signals of low attention. Record those decisions so that findings are not shaped by removing answers simply because they complicate the story.
Read completion time in context
Completion time can help identify responses that deserve a second look. Compare a respondent’s time with the length and demands of the questionnaire, including any reading, recall or open-text tasks. A short survey with familiar questions may reasonably be completed faster than one requiring detailed comparisons. If timing is available by page, it can be more informative than a single total: very little time spent on a dense page may warrant review.
Avoid using one universal speed cutoff. Device type, accessibility needs, interruptions and prior knowledge can all affect how long a survey takes. An extremely short completion time may be a useful flag, especially alongside other concerns, but speed alone does not show whether the answers are meaningful. Check whether the survey platform’s timing measure includes time when a respondent left the page open, since that can make long durations equally hard to interpret.
Look for patterns across answers
Repeatedly selecting the same option across a rating grid is sometimes called straightlining. It is more concerning when the items cover distinct ideas and the respondent shows other signs of not reading them. It is less concerning when the items are closely related or the person has a strongly consistent opinion. Review the wording and order of the questions before interpreting the pattern.
Also look for answers that appear to conflict, such as saying a product has never been used and later rating a recent experience with it. First check whether the questions refer to the same product, person and time period. Survey logic may have shown someone an irrelevant question, or one item may have been ambiguous. Blank fields and brief open-text answers need similar care: “none” can be a complete answer to some questions, while unrelated or repeated text may add to a broader pattern of concern.
Combine signals and examine the survey itself
A useful review process brings several indicators together rather than relying on a single test. For example, a very fast response with repeated answers, an irrelevant open-text entry and a clear eligibility mismatch deserves more scrutiny than a fast response with detailed, coherent answers. Keep a record of which checks each response triggered, and distinguish evidence of ineligibility from evidence that is merely suggestive.
Clusters of flagged responses may also reveal a questionnaire problem. If many people skip one item or give conflicting answers to the same pair of questions, inspect the wording, instructions and skip logic. An issue shared by many respondents may tell you more about the survey than about individual effort. Improving the question for a future round can be more valuable than trying to rescue the problem through exclusions afterward.
Test how review decisions affect findings
When the evidence is mixed, compare key results with and without the flagged responses. If the overall interpretation stays similar, you can report that the findings were not materially affected by the review decision. If it changes, examine which responses drive the difference and explain the uncertainty rather than quietly selecting the version you prefer. Apply the same criteria to all respondents, regardless of whether their answers support an expected conclusion.
Document the checks used, the number of responses flagged under each check and the reasons for any exclusions. Keep the original data available so decisions can be revisited. The goal is not a dataset in which every answer looks tidy; it is a defensible account of what respondents said, which answers may be less reliable and how much those judgments matter to the conclusions.
