CS Map

Bayesian Inference

Also known as: bayes, bayesian statistics, posterior inference

intermediate concept 5 min read · Updated 2026-06-17

Updating a belief about the world when new evidence arrives — Bayes' rule turned into a systematic method for learning from data while accounting for prior knowledge.

Primary domain
Algorithms & Mathematics
Sub-category
Discrete Mathematics, Probability & Statistics

In simple terms

Bayesian inference answers: “given what I already believed, and what I just observed, what should I believe now?” The answer is always a distribution — not a single number, but a range of possibilities with confidence attached. You start with a prior (your belief before seeing data), observe evidence, and the math outputs a posterior (your updated belief). Every new observation narrows or shifts the posterior further.

The Visual Map

graph LR
    prior["Prior P(H)\nbelief before data"] --> bayes{"Bayes' rule\nP(H|D) ∝ P(D|H)·P(H)"}
    like["Likelihood P(D|H)\nhow well H explains data"] --> bayes
    data["Observed data D"] --> bayes
    bayes --> post["Posterior P(H|D)\nupdated belief"]
    post -->|"more data arrives"| prior
    post --> approx["Intractable?\nMCMC · variational inference"]

More detail

The engine is Bayes’ rule:

P(hypothesis | data) = P(data | hypothesis) × P(hypothesis) / P(data)
  • Prior P(hypothesis) — what you believed before.
  • Likelihood P(data | hypothesis) — how probable is this data, assuming the hypothesis is true?
  • Posterior P(hypothesis | data) — the updated belief, combining both.
  • Evidence / marginal likelihood P(data) — a normalising constant.

The step from a formula to an inference method is: treat model parameters as random variables, write down a prior over them, observe data, and compute (or approximate) the posterior. Classical (frequentist) statistics instead treats parameters as fixed unknowns and asks whether data is consistent with them. Bayes treats uncertainty about parameters symmetrically with uncertainty about data.

In practice the posterior is often intractable and must be approximated — with Markov Chain Monte Carlo (MCMC) (drawing samples from the posterior) or variational inference (fitting a simpler distribution). Many ML models have Bayesian interpretations: Gaussian naive Bayes, a Bayesian neural network, or a Gaussian process are all posteriors over parameters.

Bayesian thinking changes how you design and interpret models: it forces you to declare what you believe before seeing data (avoiding overfitting to noise), it lets uncertainty propagate through a whole pipeline, and it naturally handles small data. Spam filters, recommendation systems, medical diagnosis, sensor fusion, and most of modern probabilistic ML lean on it. It also sharpens A/B testing — a Bayesian test reports “probability variant B is better” directly, instead of the often-misread p-value.

Under the Hood

The most instructive case is the medical test: even a 99%-accurate test for a rare disease yields a low posterior, because the prior (base rate) dominates. Bayes’ rule makes this precise:

def posterior(prior, sensitivity, false_pos_rate):
    # P(disease | positive) via Bayes' rule
    p_pos = sensitivity * prior + false_pos_rate * (1 - prior)
    return sensitivity * prior / p_pos

prior = 0.001          # 1 in 1000 has the disease
sens  = 0.99           # P(positive | disease)
fpr   = 0.01           # P(positive | healthy)

post = posterior(prior, sens, fpr)
print(f"prior belief:      {prior:.3%}")
print(f"after one positive: {post:.1%}")   # ~9% — far from 99%!

# A second independent positive test uses the first posterior as the new prior
post2 = posterior(post, sens, fpr)
print(f"after two positives: {post2:.1%}")  # belief compounds upward

Chaining observations — feeding each posterior back as the next prior — is exactly how a spam filter or a Kalman filter updates incrementally.

Engineering Trade-offs

  • Prior strength vs data. A strong prior regularises and shines under little data but biases results if it is wrong; a weak (flat) prior lets data speak but offers no protection against overfitting noise.
  • Exact vs approximate inference. Closed-form (conjugate) posteriors are instant but only exist for special prior/likelihood pairs. MCMC is general and asymptotically exact but slow and needs convergence diagnostics; variational inference is fast but only approximate.
  • Full posterior vs point estimate. Carrying the whole distribution propagates uncertainty honestly but costs compute and memory; collapsing to the maximum-a-posteriori point is cheap but throws the uncertainty away.
  • Interpretability vs cost. Bayesian outputs (“87% chance B beats A”) are directly actionable, but getting them reliably is more expensive than a frequentist test.

Real-world examples

  • A spam filter scores incoming email by updating the posterior probability of “spam” given the words it sees.
  • GPS fuses noisy sensor readings with a motion model using a Kalman filter — a linear-Gaussian form of Bayesian inference.
  • Drug-trial analysis starts with a prior over efficacy and updates with outcomes to get a posterior probability the drug works.
  • Bayesian optimisation tunes hyperparameters by maintaining a posterior over the loss surface and picking the next configuration to maximise expected improvement.

Common misconceptions

  • “The prior is subjective, so Bayes is unscientific.” The prior makes assumptions explicit; frequentist methods also have implicit assumptions that are just harder to see.
  • “You always need a lot of data for Bayesian methods.” The opposite is an advantage: priors regularise inference under small data, where maximum-likelihood estimation overfits.

Try it yourself

Feel the base-rate effect — see how a “99% accurate” test still leaves you probably healthy, then how repeated positives compound (python3 only):

python3 - <<'EOF'
def posterior(prior, sens, fpr):
    p_pos = sens * prior + fpr * (1 - prior)
    return sens * prior / p_pos

prior, sens, fpr = 0.001, 0.99, 0.01
print(f"base rate: {prior:.2%}")
for test in range(1, 5):
    prior = posterior(prior, sens, fpr)   # yesterday's posterior is today's prior
    print(f"after {test} positive test(s): P(disease) = {prior:.1%}")
EOF

Learn next

Neighborhood

A visual companion to the relationships above. Click any node to visit that topic.