Sample Size Calculator - Survey, A/B Testing & Statistical Power Sample Size Calculator ...

Sample Size Calculator

Sample Size for Proportions

💡 When to use:
Surveys, polls, market research where you're estimating a percentage (e.g., "% who prefer Product A").
Formula: n = (Z² × p × (1-p)) / E²

Expected Proportion (p)

Estimated percentage of population with the characteristic. Use 50% for maximum sample size if unknown.

Margin of Error (E)

Acceptable error range (e.g., ±5% means true value is within 5 percentage points of sample estimate).

Confidence Level 99% (Z = 2.576) 95% (Z = 1.960) 90% (Z = 1.645)

95% Confidence

Most common

95%

Real-World Application

A political poll with 95% confidence level and ±3% margin of error requires approximately 1,067 respondents (assuming 50% expected proportion). This means if 48% of respondents support a candidate, we can be 95% confident the true support is between 45% and 51%.

Sample Size for Means

💡 When to use:
Measuring continuous variables like height, weight, time, revenue per user.
Formula: n = (Z² × σ²) / E²

Estimated Standard Deviation (σ)

Measure of variability in your population. Estimate from pilot study, previous research, or use range/4 as rough estimate.

Margin of Error (E)

Maximum acceptable difference between sample mean and true population mean (in same units as your measurement).

Confidence Level 99% (Z = 2.576) 95% (Z = 1.960) 90% (Z = 1.645)

💡 Example:
Estimating average time spent on website:
• σ = 120 seconds (from analytics)
• E = 15 seconds margin of error
• 95% confidence
→ Required sample size = 246 users

Real-World Application

A UX researcher wants to estimate average session duration with ±10 second precision at 95% confidence. With historical data showing σ = 45 seconds, they need 78 participants. This ensures their estimate of average session duration will be within 10 seconds of the true population average 95% of the time.

Sample Size for A/B Testing (Power Analysis)

💡 When to use:
Experiments comparing two groups (A/B tests, clinical trials) to detect a meaningful difference.
Key concepts: Statistical power (1-β) is probability of detecting a real effect.

Baseline Conversion Rate

Current conversion rate or metric value for control group (e.g., 20% checkout completion).

Minimum Detectable Effect (MDE)

Smallest relative improvement you want to reliably detect (e.g., 10% relative increase from 20% → 22%).

Statistical Significance (α) 1% (p < 0.01) 5% (p < 0.05) 10% (p < 0.10)

Statistical Power (1-β) 80% (standard) 90% (more rigorous) 95% (clinical trials)

Small Effect

1,500+

per variant

Medium Effect

390

per variant

Large Effect

65

per variant

Real-World Application

An e-commerce site with 15% cart-to-purchase rate wants to detect a 20% relative improvement (to 18%) with 95% confidence and 80% power. They need approximately 1,900 users per variant (3,800 total). Running the test with fewer users risks missing a real improvement (Type II error).

Finite Population Correction

💡 When to use:
When sampling from a small, known population (e.g., company employees, registered users).
Formula: n_adj = (N × n) / (N + n - 1)

Population Size (N)

Total number of individuals in your population (e.g., 500 employees, 10,000 customers).

Expected Proportion (p)

Margin of Error (E)

Confidence Level 99% (Z = 2.576) 95% (Z = 1.960) 90% (Z = 1.645)

💡 Key Insight:
Finite population correction significantly reduces required sample size only when sampling >5% of the population. For large populations (N > 10,000), the correction has minimal effect.

Real-World Application

A company with 250 employees wants to survey job satisfaction with 95% confidence and ±5% margin of error. Without correction: 385 needed (impossible). With finite population correction: only 152 employees need to respond—making the survey feasible while maintaining statistical rigor.

Calculation Results

n = (Z² × p × (1-p)) / E²

Calculation Type: Proportion (Survey)

Required Sample Size: 385

Confidence Level: 95%

Margin of Error: ±5.0%

Expected Proportion: 50%

Z-Score: 1.960

Calculation Steps:

1 Identify parameters

Confidence level = 95% → Z = 1.960
Expected proportion (p) = 50% = 0.50
Margin of error (E) = 5% = 0.05

2 Apply formula

n = (Z² × p × (1-p)) / E²
n = (1.960² × 0.50 × 0.50) / 0.05²
n = (3.8416 × 0.25) / 0.0025
n = 0.9604 / 0.0025 = 384.16

3 Round up

Always round UP to ensure margin of error is met:
n = 385

Sample Size by Confidence Level

271

385

664

90% 95% 99%

💡 Critical Note:
This is the MINIMUM required sample size. Always account for:

• Non-response rate: If expecting 20% non-response, invite 385 / 0.80 = 482 people
• Stratification: For subgroup analysis, multiply by number of key segments
• Data quality: Plan for 5-10% unusable responses

The Complete Guide to Sample Size Calculation: From Theory to Practice

Sample size determination is one of the most critical—and frequently misunderstood—steps in research design. Too small a sample yields inconclusive results; too large wastes resources and may detect trivial differences. This comprehensive guide demystifies sample size calculation, providing both theoretical foundations and practical applications across surveys, experiments, and observational studies.

Why Sample Size Matters: The Goldilocks Principle

Sample size sits at the intersection of statistics, research design, and resource allocation. Getting it "just right" requires balancing multiple factors:

Too Small

• High margin of error → imprecise estimates
• Low statistical power → missed real effects (Type II errors)
• Results not generalizable to population
• Wasted effort if study is inconclusive

Just Right

• Acceptable precision for decision-making
• Sufficient power to detect meaningful effects
• Efficient use of resources
• Results trusted by stakeholders

Too Large

• Diminishing returns on precision
• Wasted time, money, and effort
• Ethical concerns (especially in clinical trials)
• Detects statistically significant but trivial differences

Core Statistical Concepts Demystified

Confidence Level

What it is: The probability that your confidence interval contains the true population parameter.

Common values: 90%, 95%, 99%

Trade-off: Higher confidence → larger sample size. Moving from 95% to 99% confidence increases required sample size by approximately 73%.

Practical guidance: Use 95% for most business applications; 99% for high-stakes decisions (medical, safety); 90% for exploratory research or when resources are extremely limited.

Margin of Error (Confidence Interval Width)

What it is: The radius of the confidence interval around your estimate. A ±5% margin means if 60% of your sample prefers Product A, the true population value is between 55% and 65%.

Trade-off: Smaller margin → dramatically larger sample size. Halving your margin of error QUADRUPLES required sample size.

Practical guidance: For market research, ±5% is standard; for political polling, ±3% is typical; for product testing with clear preferences, ±10% may suffice.

Expected Proportion (for Categorical Data)

What it is: Your best estimate of the population percentage before collecting data.

Critical insight: Sample size is maximized when p = 50%. As p approaches 0% or 100%, required sample size decreases.

Practical guidance: When uncertain, use p = 50% to ensure adequate sample size. If you have prior data suggesting p = 10%, you can use that for a more efficient calculation—but be cautious of under-sampling.

n_p=10% = 139 vs. n_p=50% = 385 (for 95% CL, ±5% MOE)

Sample Size Formulas: When to Use Which

Scenario	Formula	Key Inputs
Proportion (Survey) e.g., % preferring Product A	n = (Z² × p × (1-p)) / E²	Z-score, expected proportion (p), margin of error (E)
Mean (Continuous) e.g., average time on site	n = (Z² × σ²) / E²	Z-score, standard deviation (σ), margin of error in measurement units
A/B Test (Power) e.g., conversion rate test	Complex formula using power analysis	Baseline rate, minimum detectable effect, α (significance), power (1-β)
Finite Population e.g., employee survey	nadj = (N × n) / (N + n - 1)	Population size (N), initial sample size (n)

Statistical Power: The Hidden Dimension of Sample Size

While confidence intervals address precision of estimates, statistical power addresses your ability to detect real effects in experiments. Many researchers focus exclusively on significance (α) while neglecting power (1-β), leading to underpowered studies that miss real effects.

Type I Error (α)

False positive: Concluding an effect exists when it doesn't.

Standard threshold: α = 0.05 (5% chance of false positive)

"We found a difference!" (when none exists)

Type II Error (β)

False negative: Missing a real effect that exists.

Standard threshold: β = 0.20 → Power = 80%

"No difference found" (when one actually exists)

Power analysis insight: Detecting small effects requires dramatically larger samples than detecting large effects. To detect a 5% relative improvement in conversion rate (from 20% → 21%), you need approximately 4× more users than to detect a 20% improvement (20% → 24%).

Real-World Sample Size Examples

Example 1: Customer Satisfaction Survey

Scenario: SaaS company with 15,000 customers wants to measure NPS with ±3% margin of error at 95% confidence.

Calculation:

• Using conservative p = 50% (maximizes sample size)
• n = (1.96² × 0.5 × 0.5) / 0.03² = 1,067
• Finite population correction negligible (N = 15,000 >> n)
• Accounting for 30% expected response rate: 1,067 / 0.70 = 1,525 invitations

Result: Invite 1,525 randomly selected customers to achieve target precision.

Example 2: A/B Test for Checkout Flow

Scenario: E-commerce site with 12% baseline conversion wants to detect 15% relative improvement (to 13.8%) with 95% confidence and 80% power.

Calculation:

• Using power analysis formula for two-proportion test
• Required per variant: 3,210 users
• Total required: 6,420 users (3,210 × 2 variants)
• At 500 daily visitors to test page: 6,420 / 500 = 13 days runtime

Result: Run test for minimum 13 days (longer to account for weekly seasonality).

Example 3: Employee Engagement Survey

Scenario: Company with 350 employees wants to measure engagement with ±5% margin of error at 95% confidence.

Calculation:

• Initial sample size (infinite population): n = 385
• Finite population correction: n_adj = (350 × 385) / (350 + 385 - 1) = 184
• Accounting for 85% expected participation: 184 / 0.85 = 217 invitations

Result: Survey all 350 employees (feasible), expect ~300 responses—more than sufficient.

Common Mistakes & How to Avoid Them

• Using arbitrary sample sizes: "We always survey 100 people" ignores required precision. Always calculate based on statistical requirements.
• Ignoring non-response: Calculating n = 385 but only getting 200 responses invalidates your margin of error. Always inflate initial sample to account for expected non-response.
• Confusing statistical significance with practical significance: A huge sample can detect a 0.1% conversion difference that's statistically significant but business-irrelevant. Define minimum detectable effect based on business impact first.
• Forgetting subgroup analysis: Needing 385 total respondents doesn't mean you can analyze 5 demographic segments with 77 each. Each subgroup needs adequate sample size.
• Using population size in infinite population formula: For populations >10,000, population size has negligible effect on sample size. Don't reduce sample size just because your population is "only" 50,000.
• Neglecting power in experiments: Focusing only on p-values without ensuring adequate power leads to inconclusive tests. Always conduct power analysis before launching A/B tests.

The Sample Size Decision Framework

1. Define your primary metric and required precision
2. Determine acceptable margin of error based on decision context
3. Select confidence level (95% standard)
4. Estimate key parameters (proportion, SD) from prior data
5. Calculate minimum sample size
6. Adjust for non-response, subgroup analysis, and practical constraints
7. Document assumptions for transparency

Advanced Considerations

Design effects for complex sampling: Cluster sampling (e.g., sampling classrooms rather than individual students) requires larger samples. Multiply calculated n by design effect (typically 1.25-2.0).

Sequential testing adjustments: If checking results multiple times during data collection (common in A/B testing), adjust significance thresholds using methods like alpha-spending or sequential testing boundaries to avoid inflated false positive rates.

Bayesian approaches: Bayesian sample size determination focuses on credible intervals and decision-theoretic criteria rather than frequentist confidence intervals. Particularly useful when incorporating prior information or when decisions have asymmetric costs.

Conclusion

Sample size calculation is not a mere statistical formality—it's a critical design decision that affects validity, resource allocation, and ultimately the value of your research. By understanding the relationships between confidence level, margin of error, population variability, and statistical power, you can design studies that yield actionable insights without wasting resources.

Remember: The goal isn't the smallest possible sample or the largest feasible one—it's the right sample size to answer your specific research question with appropriate precision and confidence. Use this Sample Size Calculator to explore how different parameters affect requirements, and always document your assumptions and calculations for transparency and reproducibility.

Frequently Asked Questions

Q: Why does sample size not depend much on population size?

This surprises many people! The reason is mathematical: sample size formulas for infinite populations work well even for large finite populations because the sampling distribution depends primarily on sample size, not population size.

Key insight: A sample of 385 people gives ±5% margin of error at 95% confidence whether you're sampling from a city of 50,000 or a country of 50 million.

Finite population correction only matters when your sample exceeds 5% of the population. For populations under 1,000, it becomes important:

• Population = 500 → n = 217 (vs. 385 without correction)
• Population = 1,000 → n = 278 (vs. 385)
• Population = 10,000 → n = 370 (vs. 385 – negligible difference)

Bottom line: For populations >10,000, ignore population size in sample size calculations. For smaller populations, apply finite population correction.

Q: What if I don't know the expected proportion (p) for my survey?

Use p = 50% (0.5). This is the statistically conservative approach because:

1. The term p(1-p) in the sample size formula is maximized at p = 0.5
2. Using p = 0.5 gives the largest possible sample size requirement
3. This ensures your margin of error will be ≤ target even if the true proportion differs

Example comparison (95% CL, ±5% MOE):

• p = 50% → n = 385
• p = 30% → n = 323 (16% smaller)
• p = 10% → n = 139 (64% smaller)

Practical advice: If you have prior data suggesting p is far from 50% AND you're resource-constrained, you can use that estimate. But if wrong, your margin of error will be larger than planned. When in doubt, use 50%.

Q: How do I estimate standard deviation (σ) for sample size calculation when measuring means?

Unlike proportions, there's no "conservative" value for standard deviation—you must estimate it. Here are practical approaches:

1. Pilot study: Collect data from 20-30 subjects first, calculate SD, then compute final sample size.
2. Prior research: Use SD reported in similar published studies.
3. Range rule of thumb: If you know plausible min/max values, estimate σ ≈ (max - min) / 4.
4. Expert judgment: Ask domain experts for reasonable variability estimates.

Example: Estimating average task completion time:

• Expected range: 30 to 150 seconds
• Estimated σ ≈ (150 - 30) / 4 = 30 seconds
• For ±5 second margin of error at 95% CL:
  n = (1.96² × 30²) / 5² = 139 participants

Note: Underestimating σ leads to underpowered studies. When uncertain, slightly overestimate σ to be safe.

Q: What's the difference between confidence level and statistical power?

These address different types of studies and different errors:

Concept	Confidence Level	Statistical Power
Used for	Estimation (surveys, descriptive studies)	Hypothesis testing (experiments, A/B tests)
Controls	Type I error (false positive)	Type II error (false negative)
Typical value	95% confidence → 5% α	80% power → 20% β
Question answered	"How precise is my estimate?"	"How likely am I to detect a real effect?"

Simple analogy:

• Confidence level: Width of your fishing net. Wider net (larger margin of error) catches the fish (true value) more reliably but with less precision.
• Statistical power: Sensitivity of your metal detector. Higher power detects smaller buried objects (subtle effects) but requires more scanning time (larger sample).

Q: How do I account for non-response in my sample size?

Non-response is the #1 reason surveys fail to achieve target precision. Always inflate your initial sample:

Invitations Needed = Calculated n / Expected Response Rate

Typical response rates by method:

• In-app surveys: 15-30%
• Email surveys: 10-25%
• Customer intercepts: 30-60%
• Employee surveys: 60-85%
• Incentivized panels: 70-90%

Example: Need n = 400 completed responses

• Email survey with 20% expected response:
  Invitations = 400 / 0.20 = 2,000
• In-app survey with 25% expected response:
  Invitations = 400 / 0.25 = 1,600
• Employee survey with 80% expected response:
  Invitations = 400 / 0.80 = 500

Critical: Non-response bias (systematic differences between responders and non-responders) is more dangerous than non-response rate. A 20% response from a representative sample is better than 80% response from a self-selected group.

Q: Is there a minimum sample size that always works?

No—there is no universal minimum sample size. The required sample depends entirely on:

• Your required precision (margin of error)
• Confidence level
• Population variability
• Effect size you want to detect (for experiments)

Common misconceptions:

• "30 is enough because of the Central Limit Theorem" → CLT addresses distribution shape, not precision. You may need 30 or 3,000 depending on requirements.
• "10% of the population is sufficient" → For large populations, 10% is wasteful; for small populations, it may be insufficient.
• "100 responses is standard" → May give ±10% margin for proportions—often too imprecise for business decisions.

Rule of thumb for quick estimation:
For 95% confidence and ±5% margin of error estimating a proportion near 50%:
n ≈ 400
This works for populations >10,000 and is a reasonable starting point when precision requirements are moderate.

Always calculate the precise sample size for your specific requirements rather than relying on rules of thumb for important decisions.