Run a significance test when you have a pre-specified subgroup comparison, clean random sampling, and cells large enough to trust the math. A common threshold of 0.05 (corresponding to 95% confidence) is widely used as a cutoff, but a p-value under 0.05 suggests a difference is likely real but not necessarily practically meaningful. Check effect size before you act, and watch for small cell counts, clustering, or survey weights, because each one changes which test you should actually run.

TL;DR:

  • Significance tests require large, independent samples with minimum expected counts of at least 5 to 10 for reliable results.
  • Use Fisher’s exact test instead of z-tests when cell counts are small, and adjust for weighting and clustering in survey design.
  • P-values below 0.05 indicate statistical significance but must be accompanied by effect size and confidence intervals for meaningful interpretation.
  • Smaller samples, high variability, or heavily weighted data necessitate survey-weighted tests to avoid false positives or negatives.
  • Avoid multiple uncorrected testing and report all relevant statistics, including test type, p-value, effect size, and confidence interval, for accurate analysis.
Veridata insights
Make Your Survey Findings More Reliable
Veridata Insights supports research from methodology and questionnaire review through data collection, processing, analysis, and reporting.

Explore research services

Table of Contents

A Quick Checklist Before You Run a Significance Test on Survey Results

Before any p-value means anything, the design underneath it has to hold up. Run through this short list first.

  • Confirm independence: each respondent should appear once, and random or systematic sampling should back the sample, not convenience recruiting.
  • Check minimum expected counts. The common rule for z-tests requires n × p and n × q ≥ 5, with ≥10 as the more conservative bar many researchers prefer.
  • Decide one-tailed versus two-tailed before you see the data, not after.
  • Pre-register which subgroup comparisons matter. Testing every cross-tab in your data table and reporting whichever ones came back significant is not analysis, it’s fishing.
  • When counts run small, switch to Fisher’s exact test rather than forcing a normal approximation that doesn’t fit.

Get these right and the rest of the process, the actual test, becomes almost mechanical.

How Do You Run a Z-Test for Proportions?

The two-proportion z-test is the workhorse of survey analysis: comparing “percent who agree” across two groups, two time points, or a group versus a fixed benchmark. Here’s the sequence.

Statistical Significance Threshold: Most survey analysis uses a 95% confidence level, meaning p < 0.05 marks a statistically significant result. Some regulated or high-stakes studies tighten that to p < 0.01.

  1. State your hypotheses. For a two-sample test, the null (H0) says the two population proportions are equal; the alternative (H1) says they differ (or, for a one-tailed test, that one is specifically higher or lower).
  2. Pool the proportions. Under H0, you assume both groups share a single true rate, so you calculate a pooled proportion from the combined sample before finding the standard error.
  3. Check sample-size conditions. Confirm n × p and n × q clear 5, ideally 10, in both groups. If either group falls short, this test is the wrong tool.
  4. Compute z. Divide the observed difference in proportions by the pooled standard error. This gives you your z-statistic.
  5. Convert to a p-value and build a 95% CI. The two-proportion z-test’s pooled standard error feeds both the significance test and the confidence interval around the observed difference.

P-values below 0.05 indicate statistical significance, though lower values offer stronger evidence, but they are not equally convincing. Results with p-values near the threshold, such as between about 0.03 and 0.07, may be considered borderline. Report the confidence interval alongside it and let the reader see how much uncertainty is still riding on that number, rather than collapsing it into a flat yes or no.

When Should You Use a T-Test Instead of a Z-Test?

Proportions (percent who chose an answer) call for z-tests. Means and scale scores, like a 1-to-10 satisfaction rating or an average number of purchases, call for t-tests. The distinction matters because scale-score data behaves differently, and smaller samples need the fatter tails of the t-distribution to avoid overconfidence.

  • Standard two-sample t-test: use when comparing means across two independent groups with roughly equal variances and adequate sample size.
  • Welch’s t-test: switch here the moment variances look unequal between groups. It’s the safer default in survey work, since satisfaction scores and usage frequencies rarely have matched spread across segments.
  • Paired t-test: use for repeated measures, before-and-after tracking studies, or matched respondent pairs answering the same scale twice.

Researchers typically favor t-tests over z-tests once samples drop below roughly 30 per group, since the t-distribution corrects for the extra uncertainty that comes with a smaller sample. Degrees of freedom shrink as sample size drops, which widens your confidence interval and makes borderline results more common. That’s a feature, not a flaw. It’s the math being honest about how little you actually know from 22 respondents.

If your data violates normality badly (heavily skewed satisfaction scores, for instance) or your sample is very small and lumpy, a rank-based nonparametric test like Mann-Whitney U is often the more defensible choice than forcing a t-test to work.

How Do You Plan Sample Size for a Meaningful Difference?

The minimum detectable effect, or MDE, is the smallest true difference your survey is actually built to catch. Plan around it before fieldwork starts, not after the data comes back disappointing.

  1. Set your MDE first. Ask stakeholders what size of shift would actually change a decision, not just what would look impressive in a report.
  2. Account for alpha and power together. A smaller MDE, a stricter alpha (0.01 instead of 0.05), or higher desired power (90% instead of 80%) all push your required sample size up.
  3. Watch your baseline rate. Baseline proportions strongly affect the sample size needed to detect a given MDE; a shift from 50% to 55% needs a very different sample than a shift from 5% to 10%.
  4. Adjust for unbalanced groups. If one subgroup is naturally smaller (say, a niche B2B vertical), your power calculation needs to reflect that imbalance, not assume equal cells.
  5. Use pilot data when you have it. A small pretest gives you a real variance estimate instead of a guess pulled from a textbook.

Setting an MDE too small is the most common planning mistake we see. Chase a 2-point shift in a satisfaction score, and you may need thousands of completes to detect it reliably. Our guide on choosing the right sample size for quantitative research walks through the tradeoffs in more depth.

What Does a P-Value Actually Tell You?

A p-value answers one narrow question: assuming the null hypothesis is true, how likely is it that you’d see a difference this large (or larger) by chance alone? It says nothing about how big the difference is, how important it is to your business, or whether your survey design was sound to begin with.

The 0.05 cutoff is a convention, not a law of nature. It got adopted widely because it’s a reasonable balance between false positives and missed findings, not because 0.049 is meaningfully different from 0.051.

  • Pair every p-value with an effect size: Cohen’s h for comparing two proportions, Cohen’s d for comparing two means.
  • Translate that effect size into something a stakeholder recognizes, like “a 6-point lift in brand awareness” rather than “h = 0.15.”
  • A full report should include test type, p-value, confidence interval, effect size, and the sample size behind each group.

Large samples make this especially tricky, since even a trivial difference can turn statistically significant once you survey enough people, which is exactly why effect size has to sit next to the p-value, not behind it.

Pro Tip: Report the confidence interval’s width, not just its bounds. A 95% CI of [1%, 25%] and one of [8%, 18%] can both center on the same point estimate, but only one of them tells you the effect is reliably meaningful.

Illustration comparing narrow and wide confidence intervals

Why Do Weighted and Clustered Surveys Break Standard Tests?

Standard z and t formulas assume simple random sampling with no weighting and no clustering. Most real market research violates that assumption the moment you apply demographic weights to match census benchmarks or sample within strata like regions or store locations. Running an unweighted test on weighted data doesn’t just shade your p-value, it can flip a null result into a false positive or bury a real one.

Weighted analyses change your standard errors and often shrink your effective sample size below the raw count of completes. Always check design degrees of freedom before trusting a p-value pulled from a naive formula.

  • Use svy: mean followed by test, or lincom, to get design-adjusted standard errors and correct p-values.
  • svy: regress gives you an equivalent path when you need model-based comparisons across more than two groups.
  • These survey-weighted approaches apply whether you’re comparing weighted proportions or weighted scale-score means.
  • When strata get thin (a handful of respondents in a niche segment), an adjusted Wald test or a specialist workaround usually outperforms the default.

Our own survey weighting methodology notes go deeper into when design effects get large enough to demand this kind of adjustment.

What Mistakes Should You Avoid in Survey Significance Testing?

The same handful of errors show up across almost every flawed write-up we review.

  • P-value tunnel vision: reporting “significant at p < 0.05” with no effect size, no CI, and no discussion of what the difference actually means for the business.
  • Mass testing without correction: running dozens of cross-tabs and reporting the handful that cleared 0.05 by chance. Apply a Bonferroni or Holm correction whenever you’re testing more than two or three pre-specified comparisons.
  • Ignoring design effects: treating a weighted, clustered sample like a clean random one and trusting the resulting p-value at face value.

For borderline results, use language that matches the uncertainty: “the difference is directionally consistent with X but did not reach significance at the pre-specified threshold” beats a flat “no difference found.”

Pro Tip: Build your report template before fieldwork starts: test name, hypotheses, statistic and degrees of freedom, p-value, CI, effect size, group sample sizes, and any weighting adjustments. Filling in a template is faster and more honest than reconstructing your logic after the numbers surprise you.

When Should You Bring In a Specialist for Survey Testing?

Most of what’s above works fine on a clean, unweighted, moderately sized sample. It starts to break down fast once you’re dealing with a nationally weighted tracker, a healthcare study with small clinical subgroups, or a multi-market B2B survey where every country needs its own significance check.

Specialist firms build the methodology, program the survey, recruit hard-to-reach segments, and handle weighting and analysis so significance tests behind findings hold up to scrutiny, not just a quick eyeball check. If you’re running a regulated healthcare study, a global tracker with design-weighted subgroups, or a B2B study where your target respondents are genuinely hard to find, that complexity usually justifies specialist support rather than a DIY spreadsheet formula.

Our quantitative market research team handles the weighting, testing, and reporting end to end, and our full-service market research offering covers everything from questionnaire design through final analytics if you need the whole pipeline built right. Reach out and tell us what you’re testing.

Where Can You Find the Formulas and Calculators Behind These Tests?

For the underlying proofs and step-by-step formulas: the z-test for proportions walkthrough, the two-proportion z-test reference, and a free proportion z-test calculator for quick checks against your manual math. For triangulating survey findings against other evidence signals, this prediction market accuracy guide is worth a look.

Sources

FAQ

What Is a 5% Significance Test?

A 5% significance test uses an alpha level of 0.05, meaning a 95% confidence threshold: you conclude a result is statistically significant if there’s less than a 5% probability of seeing a difference that large purely by chance.

Is a 0.01 or 0.05 Significance Level Better?

Neither is universally “better.” A 0.05 level is standard for most survey work and balances false positives against missed findings, while 0.01 suits higher-stakes decisions like healthcare or regulatory studies where a false positive carries more risk.

How Do You Know If a Survey Result Is Statistically Significant?

Run the appropriate test (a z-test for proportions, a t-test for means) on your subgroups, then check whether the resulting p-value falls below your pre-set alpha, typically 0.05, while confirming your sample-size and independence assumptions held up.

What Is Significance Testing in Survey Research?

Significance testing is a set of statistical procedures, most commonly z-tests and t-tests, used to determine whether an observed difference between survey groups likely reflects a true population difference rather than random sampling variation. Veridata Insights builds this testing directly into its quantitative research services so the statistics behind a finding hold up alongside the story it tells.