How to analyze survey responses and open-text comments
Analyze survey data by validating response structure and missingness, separating rating analysis from open-text coding, creating a documented theme taxonomy on a sample, allowing multiple themes per response, checking coding consistency, and comparing prevalence only with adequate denominators. Keep de-identified quotations linked to source IDs for auditability.
From the field · HR business partner
Analyze employee survey comments without exposing a small team
An engagement survey includes rating questions, optional comments, team, tenure, and location. A verbatim quotation combined with a team of three could identify its author even after names are removed.
Columns used in the example
response_id · question_id · rating · comment · team · tenure_band · location
Follow it with your own file
- Separate rating response rates from score distributions and retain denominators.
- Develop a small theme taxonomy on a representative comment sample and allow multiple themes per response.
- Suppress or combine groups below the approved privacy threshold.
- Select only de-identified quotations and review them for indirect identifiers.
Check: Theme counts should reconcile to coded responses, not necessarily to one theme per response. Recheck a sample for coding consistency and verify that no published table or quotation exposes an individual.
Where to look for exceptions
- Averages lack response counts.
- Blank text is treated as neutral sentiment.
- Themes change on every run.
- Quotes cannot be traced to de-identified responses.
The decision hiding in the data
Survey exports mix metadata, scale questions, multi-select fields, and free text. Teams often summarize averages and ask an AI for themes without checking skip logic, duplicates, leading questions, or taxonomy stability.
Before touching the file
- Do not treat unanswered questions as negative responses.
- Test a codebook before classifying the full corpus.
- One response may contain several themes.
- Report counts and denominators together.
Open-text analysis methods
| Method | Good for | Watch for |
|---|---|---|
| Manual coding | Small studies and nuance | Time and consistency |
| Keyword rules | Stable explicit categories | Synonyms and context |
| AI-assisted coding | Larger corpora with review | Drift, privacy, unsupported inference |
Build the result step by step
- Audit survey structure: Confirm respondent grain, duplicates, question types, skip logic, missingness, scale direction, and segment fields.
- Summarize closed questions: Report distributions, denominators, suitable central tendency, and uncertainty. Avoid ranking tiny segments.
- Validate a theme codebook: Review a diverse sample, define inclusion and exclusion rules, allow multi-label coding, and test consistency.
- Connect themes carefully: Compare themes across ratings or segments with counts, quotes, and privacy controls. Association is not cause.
Quantitative and qualitative answers differ
A rating distribution can be counted after scale direction and missingness are known. Open text requires interpretation. Mixing both into one sentiment score creates false precision and hides reasons.
A taxonomy is a measurement instrument
Theme labels need definitions, boundaries, and examples. Testing on a sample reveals overlap and missing categories. The aim is a consistent, reviewable method with clear limitations.
Before the result leaves your desk
- Questionnaire retained
- Skip logic reviewed
- Scale direction confirmed
- Codebook documented
- Consistency checked
- Quotes de-identified
- Denominators shown
Doing these steps in Prepyr
Prepyr accepts CSV files only. Export the worksheet or table you need as a UTF-8 CSV first; workbook formulas, formatting, macros, charts, and other sheets will not be included. Describe the result you need: Analyze survey ratings and open-text comments without losing context or privacy. Check the columns selected, preview any proposed changes, and compare the output with the source before applying or exporting it.
Prepyr steps used here: Data profiling · Natural Language analysis · PII protection · Report builder
A repeatable survey analysis with transparent denominators, stable themes, traceable evidence, and no overstatement of small samples.
Questions readers usually ask next
How many responses should I sample?
Sample across ratings, segments, lengths, and periods until new responses rarely require themes, then validate on a held-out set.
Can one response have multiple themes?
Yes. Multi-label coding is often more faithful than forcing one label.
Should averages include blank answers?
No. Show the valid-response denominator and analyze missingness separately.