Critical Thinking: How to Test Claims

Most arguments blur what was observed with the story told about it. Label each sentence. A fact is checkable against evidence. An inference explains or predicts from facts. An opinion expresses a preference or judgment. Inferences and opinions are not forbidden; they just should not borrow the certainty of observations.

FACT       38 of 100 invited users completed signup.
FACT       Last month, 51 of 100 completed it.
INFERENCE  The new identity step may have reduced completion.
OPINION    The extra verification is worth the lost signups.

Missing before causal confidence:
traffic mix · device mix · outages · experiment assignment · sample size

An argument is not a quarrel. It is a conclusion supported by premises. Rewrite prose into numbered lines. Then ask two separate questions: are the premises true, and would they support the conclusion if true? A valid shape can use false premises. True premises can still fail to reach the stated conclusion.

Statement: “We should cancel Friday support because few tickets arrive.”

P1  Only 3% of weekly tickets arrive on Friday evening.
P2  Staffing Friday evening costs $1,200 per month.
P3  Delaying those tickets until Monday creates no serious harm. UNTESTED
C   Therefore, cancel Friday evening support.

Test P3: severity, response promises, churn risk, and timezone.
The argument turns on P3, not on the easy 3% statistic.

Look for missing bridges such as “and that difference matters” or “therefore this policy is best.” People often agree on the visible facts and fight over an unstated value or prediction. Writing the bridge makes the real disagreement discussable.

The person making a claim owes evidence proportional to its consequence and surprise. “Prove me wrong” reverses that duty. Lack of disproof does not establish the claim. For low-risk experiments, the burden can be modest. For accusations, medical claims, or irreversible changes, demand much more.

“This supplement removes brain fog. Prove that it doesn't.”

Ask instead:
1. What outcome does “removes” mean, measured how and when?
2. Compared with placebo, sleep, or no treatment?
3. How many participants, selected and blinded how?
4. Was the study preregistered and replicated?
5. Who funded it, and what harms were tracked?

No evidence means “not established,” not automatically “false.”

When two things move together, several stories fit: A causes B, B causes A, a third factor causes both, selection created the pattern, or chance did. A timeline helps, but “after” is not “because.” Look for mechanism, comparison groups, dose response, natural experiments, and deliberate tests.

Claim: “Sales rose after the rebrand, so the rebrand worked.”

Competing explanations:
• holiday demand rose
• prices fell 12%
• a large reseller launched
• old ads kept converting
• measurement changed

Better test:
compare exposed and unexposed markets, keep price constant,
inspect channel-level lift, and define the success metric in advance.

A vivid case feels informative because it has detail. Ask how often the outcome happens before the special story. Then inspect who entered the sample, who left, what was measured, and whether the sample resembles the population in the claim.

Test claims 90% sensitivity and 95% specificity.
Condition affects 1 in 100 people.

Among 10,000 people:
100 have condition → 90 true positives
9,900 do not       → 495 false positives

A positive result is about 90 / 585 = 15% likely to be true
before other evidence. The low base rate matters.

Do not turn critical thinking into a vocabulary contest. Naming “straw man” or “ad hominem” does not answer the underlying claim. State the defective move in plain language, rebuild the argument, and ask for the missing evidence. A label is useful only if it points to a repair.

“You opposed the last redesign, so your accessibility concern is wrong.”
Repair: the speaker's history does not test the contrast data.
Measure contrast and keyboard navigation.

“Either ship Friday or admit we cannot execute.”
Repair: those are not the only options.
Compare Friday, staged Monday, and delayed full release.

“Three competitors use it, so it must be safe.”
Repair: popularity is not a security audit.
Inspect threat model, permissions, incidents, and controls.

Sometimes a person's incentives or expertise are relevant. That still does not make character a replacement for evidence. Say exactly why the conflict changes the weight you assign, then test the claim itself.

State the strongest version of the other person's position in terms they accept. Include their evidence and values. Ask them to confirm it. Then criticize that version. This is not politeness theater; it prevents you from winning against a claim nobody made.

Original: “We should not add mandatory two-factor login yet.”

Straw version:
“You don't care about security.”

Steelman:
“You agree account takeover matters. You expect mandatory setup
now to lock out users who share devices and lack reliable phone
access. You want recovery tested before enforcement. Correct?”

Now test: takeover rate, lockout rate, recovery completion,
alternative factors, pilot population, and launch threshold.

A useful factual claim risks failure. If every possible outcome is reinterpreted as support, the claim cannot guide action. Before seeing results, define the observation, threshold, and date that would lower confidence. “I will know it when I see it” gives your preferences control after the fact.

Claim: “The onboarding tour reduces first-week confusion.”

Before launch:
Metric: support contacts tagged setup per 100 new accounts
Comparison: randomized tour vs no tour
Success: at least 20% fewer, with no drop in activation
Failure: difference under 5% after 2,000 eligible accounts
Review: 2026-09-01

Also collect interview evidence for problems the tag misses.

Belief is rarely a switch. Put a rough number or range on confidence, predict what evidence would move it, and update by the diagnostic value of the evidence. One weak study should not turn ten percent into ninety. A failed prediction should move you more than a vague anecdote.

Claim: vendor can meet the June migration date

Prior from similar projects:                 35%
Detailed plan passes independent review:     50%
Pilot migrates 95% of messy records:         70%
Key engineer leaves; no replacement named:   48%
Second pilot fails permission mapping:        25%

Record why each update moved. Do not fake precision;
“20–30%” is honest when the evidence is rough.

Ask, “Would I have expected this evidence if the claim were false?” Evidence that appears under both stories should barely move you. Also notice asymmetric updates: if confirming anecdotes count but failed predictions never count, identity is protecting the belief.

Anger, fear, pride, and group loyalty are information about stakes, not proof about the claim. When a sentence threatens your identity, pause before searching. Write what you would accept as evidence if the same claim favored the other side. Critique privately before performing publicly. The goal is not emotional numbness; it is keeping emotion away from the evidence controls.

CLAIM-TESTING CHECKLIST
□ Write the exact claim. Define slippery words.
□ Label facts, inferences, and opinions.
□ List premises and the conclusion. Find the hidden bridge.
□ Put the burden of proof on the claimant.
□ Ask what else could produce the same observation.
□ Check base rate, sample selection, denominator, and missing data.
□ Trace important evidence to its source.
□ State the strongest opposing case and get confirmation.
□ Define what would lower confidence before seeing results.
□ Give current confidence and update size, with reasons.
□ Name the identity or incentive that makes this hard.
□ State what remains unknown and what action is justified now.

My rule: do not ask whether a person “is logical.” Ask whether this claim, from this person, survives this test. Good thinkers still defend bad ideas. Annoying people still bring true evidence.