The problem
Designing around the data you can get. Convenient data is fine when its limits are named. It becomes a problem when the question quietly shrinks to fit it.
Most weaknesses in a finished study were built in at the start. A question too broad to answer. A design that cannot separate the effect you care about from everything else going on. A sample size picked because it was affordable rather than because it was enough. Outcomes measured with instruments nobody checked for this population.
By the time these problems surface, in peer review or in an analysis that will not behave, they are expensive or impossible to fix. A reviewer can see in two paragraphs what took a team months to commit to.
The difficulty is rarely a lack of knowledge in your field. Design decisions involve trade-offs across statistics, measurement, logistics and ethics at the same time, and they are usually made under deadline by people who are also running everything else. An outside methods colleague can hold those trade-offs in view while you get on with the science.
Approximate power for comparing two groups at a two-sided alpha of 0.05, using the normal approximation. Reaching 80% takes about 25 per group for a large effect and about 175 for a small one.
View the numbers
| Effect size | n = 25 | n = 50 | n = 100 | n = 150 | n = 200 |
|---|---|---|---|---|---|
| d = 0.3 | 18% | 32% | 56% | 74% | 85% |
| d = 0.5 | 42% | 71% | 94% | 99% | 100% |
| d = 0.8 | 81% | 98% | 100% | 100% | 100% |
Who this is for
- Investigators planning a new study, pilot or fellowship project who want someone to test the plan before it is fixed.
- Teams preparing a grant whose approach section has to show that the design can deliver the aims.
- Nonprofits and organizations designing an evaluation of a program or service.
- Researchers whose design has been criticized by reviewers or an ethics board and who want to address the criticism properly rather than cosmetically.
What we do
Statistical power is the probability that a study detects an effect of a given size if that effect is real. It depends on sample size, variability, the size of effect you care about, and the error rate you accept.
- Sharpen the question and aims. We turn a broad interest into specific aims and, where they fit, testable hypotheses, with each aim tied to an outcome you can measure.
- Choose a design that fits. Experimental, quasi-experimental, observational, qualitative or mixed methods. We weigh what each can and cannot tell you in your setting, on your budget and timeline.
- Think through sampling and power. Who should be in the study, how they will be reached, and how many are needed to detect an effect that matters, with every assumption written down so a reviewer can check it.
- Plan measurement. Selecting or adapting instruments, defining outcomes and covariates, and checking that what you plan to measure matches what you plan to claim.
- Write or review the protocol. Procedures, data collection, quality checks, handling of missing data and an outline of the analysis, in a document your team can follow.
- Review feasibility. Recruitment assumptions, timelines, staffing and data access, tested against what is realistic before you commit to them.
- Support preregistration. Preparing a preregistration or registered report, and documenting any later deviations openly.
How it works
Power calculations need an effect size. Use the smallest effect that would change a decision, not the effect you hope to see. Hoped-for effects tend to be optimistic, and optimism shrinks samples.
Discovery conversation
We talk through your question, your setting, your constraints and deadlines, and what you already have in place.
Written scope
You receive a short proposal describing what we will review or draft, what you will receive, and how we will work together.
Design work
We work through the design decisions with you, usually in a shared document, setting out options and trade-offs rather than handing down answers.
Check-ins
At agreed points we meet to settle open questions, so decisions are made by the people who will live with them.
Final documents and handover
You receive the finished documents, a record of the main decisions and the reasons for them, and notes on what to watch once data collection starts.
What you receive
- A design memo covering the question, aims, design and the main trade-offs considered.
- A sample size or power rationale with every assumption stated, or a clear explanation of why a formal calculation does not apply.
- A measurement plan listing outcomes, instruments and timing.
- A draft or annotated protocol, ready for your team, an ethics review or a funder.
- An outline of the analysis plan, aligned with the aims.
- Preregistration text, if you plan to preregister.
How we can work together
- Project
- A defined piece of work with a written scope, such as designing a study from the ground up or preparing the approach section of a proposal.
- Retainer
- Ongoing access for a group that designs studies regularly and wants a methods colleague available across several projects.
- Hourly review
- A focused review of a design or protocol you have already drafted, with written comments and a follow-up call.
Timelines
Timelines depend on scope: how developed the idea is, how many people need to agree, and whether an external deadline applies. Typical ranges are agreed in the written proposal before any work starts.
If you have a fixed deadline, mention it in your first message. We will tell you honestly whether it is realistic for the work you have in mind.
Questions
Methods carry across fields, but context matters. We will ask a lot of questions about your setting and rely on you for subject knowledge. If a project needs specialist input we cannot provide, we will say so early.
Yes, though the options narrow. We can help you choose analyses that respect the design you have, describe the limitations honestly, and plan a stronger follow-up study. The data analysis service covers this in more detail.
Yes, and we will explain it so you can defend the numbers to reviewers. Where a formal calculation is not appropriate, as in some qualitative or pilot work, we will help you justify the sample in other terms.
We help prepare the methods content those submissions need. The submission itself, and every decision about approval, belong to you and your review board. Check the current requirements of your institution, since they vary.
Often, especially for confirmatory work. It puts your plan on record before results exist and protects you from accusations of searching for a result. We can talk through whether it suits your study and which format fits.
We will say so plainly and explain why. It is your study and the final decisions are yours. Our job is to make sure they are made with the trade-offs in view.