PhD mandatory courseworkEvery methods seminar assumes a vocabulary you either arrive with or pick up fast: validity, reliability, causal identification, and the difference between a construct and how you're actually measuring it. Here's that vocabulary, explained by the mistake each term exists to prevent.
Worth asking before your next design defense
Are you defending your design, or defending the design you already know how to code?
If your data is a single snapshot in time, what actually tells you which variable moved first?
01 / The vocabulary reviewers test first
The textbook definitions are easy to recite and hard to apply under questioning. What actually matters is the single question each one is really asking — and what it looks like when a design fails to answer it.
Does your measure actually capture the thing you claim it captures?
Failure looks like: You call a four-item scale "organizational commitment" because a prior paper did, without checking whether your sample understood the items the same way that paper's sample did.
Can you rule out the other explanations for what you observed?
Failure looks like: You find that firms with a practice perform better, and stop — without asking whether better-performing firms were simply more able to afford the practice in the first place.
Does the finding generalize past the exact sample and setting you studied?
Failure looks like: A result from one industry, one country, or one time period gets written up with language that implies it applies everywhere, because the paper never names the boundary.
Is the relationship you found real, or an artifact of how you tested for it?
Failure looks like: A significant result survives only because of an underpowered sample, an unreported number of model specifications tried, or a violated assumption the test quietly depends on.
02 / The gap that trips up first-years
A construct is the abstract idea you actually care about — leadership, trust, absorptive capacity. A measure is the specific instrument you used to stand in for it in this one study. The two are never identical, and the gap between them is the single most common thing a design defense probes. The tell is language: if you catch yourself writing "leadership was 4.2" instead of "scores on this leadership instrument averaged 4.2," you've quietly treated the measure as the construct itself — a slide reviewers will not let pass.
Ask this of your own instrument before anyone else does: which facets of the construct does this measure simply not reach, and have you said so?
03 / Causal identification, in plain language
A correlation is not, by itself, a causal claim. These are the four reasons why — and the one question to ask of any design, including your own, before trusting it.
A third variable causes both your predictor and your outcome, creating a relationship between them that isn't causal at all.
Ask: What else could explain both variables moving together?
The units in your sample ended up there through a process related to your outcome — so the comparison you're making was never fair to begin with.
Ask: Why did these specific cases end up in your data, and not others?
Your outcome may be causing your predictor, not the other way around — a risk that's easy to miss with cross-sectional data measured at a single point in time.
Ask: If you only have one snapshot in time, how do you know which variable moved first?
A variable that belongs in your model is missing, and its absence is correlated with something you did include — biasing the coefficient you actually report.
Ask: What would a reviewer say you left out, and why might that omission matter?
04 / Matching design to question
Pick the design your question needs, then live with what it can't tell you — every option below trades one kind of confidence for another.
Answers best: Whether a specific manipulation causes a specific effect, with the strongest possible claim to internal validity.
Cannot tell you: Whether the effect holds outside the artificial setting you created to isolate it.
Answers best: Causal questions in field settings where you can't randomize, using a naturally occurring comparison instead.
Cannot tell you: As clean a causal claim as a true experiment — you're always one assumption away from a confound your design didn't rule out.
Answers best: Whether variables are associated, and how strongly, across a broad or hard-to-manipulate sample.
Cannot tell you: Which variable caused which — association is not, by itself, a causal claim, no matter how large the sample.
Answers best: How and why a process unfolds the way it does, especially when the mechanism itself is what's poorly understood.
Cannot tell you: How common the pattern is, or how it would hold up against a large, representative sample.
Answers best: Both the size of an effect and the mechanism behind it, when a single method would leave one of those questions unanswered.
Cannot tell you: A shortcut — done well, it's two designs' worth of rigor, not half of each.
05 / What surfaces in proposal defenses
Choosing the design you're most comfortable coding before choosing the design your question actually needs.
Using cross-sectional data collected at a single point in time, then writing about the results in causal language anyway.
Running a confirmatory quantitative test without having pre-registered the hypothesis, so post-hoc pattern-finding gets described as if it had been predicted.
Comparing groups on a multi-item scale without first checking measurement invariance — whether the instrument means the same thing to every group being compared.
Take the design chapter of a paper you admire in your area and rewrite its validity threats from scratch, without looking at the paper's own limitations section. Then compare your list to theirs. The gaps in your list — the threats you didn't think to name — are exactly what a committee will ask you about first.
Guide
What each mandatory component is for, what to do before your first term, and how coursework connects to how I supervise once it's done.
See the full guideGuide
The three-pass method, a reading order that isn't the printed order, and the habit that turns scattered reading into an actual literature review.
Read the guide