CS Map

Neural Network

Also known as: neural networks, deep learning, artificial neural network

core intermediate concept 5 min read · Updated 2026-06-17

A family of machine learning models loosely inspired by the brain — layers of simple units that, together, can approximate complex functions.

Primary domain
Machine Learning
Sub-category
Supervised & Unsupervised Learning

In simple terms

A neural network is a stack of layers. Each layer is a pile of simple units (“neurons”) that each take a weighted sum of their inputs, push it through a non-linear function, and pass the result on. Stack enough of these layers together, train the weights on a lot of data, and the network can learn to recognise faces, translate languages, generate images, or play games — without anyone writing the rules by hand. The “deep” in deep learning just means many layers.

The Visual Map

graph LR
    x["Input x"] --> h1["Hidden layer 1\ny = f(W·x + b)"]
    h1 --> h2["Hidden layer 2"]
    h2 --> out["Output layer\n(prediction)"]
    out --> loss["Loss vs correct answer"]
    loss -. "backprop: ∂loss/∂W" .-> h2
    loss -. .-> h1
    h1 --- act["f = ReLU / sigmoid\n(non-linearity is essential)"]

More detail

A neuron computes y = f(W · x + b) — a weighted sum of inputs x with weights W and bias b, passed through a non-linear activation function f (ReLU is the default; older networks used sigmoid or tanh). The non-linearity is essential; without it, any stack of layers collapses into a single linear function.

Training works the same way for almost every modern network: a forward pass pushes input through and compares the output to the correct answer with a loss function; backpropagation uses the chain rule to compute how much each weight contributed to the loss; gradient descent nudges every weight a tiny step to reduce the loss; repeat for billions of examples. Major architectural families include the MLP (fully connected, the simplest deep model), the CNN (weight-sharing across spatial regions, the workhorse of vision in ~2012–2020), RNN/LSTM/GRU (for sequences, now largely replaced), the transformer (attention-based, under every modern LLM and a growing share of vision/audio), and diffusion models (iteratively denoise random noise into an image). Two realities shape everything: GPUs/TPUs are essential because every layer is dense matrix multiplication, and performance scales smoothly with more data, parameters, and compute (the “scaling laws”).

Under the Hood

A neural network is just matrix multiplies interleaved with a non-linearity. Here a tiny two-layer net does a forward pass on the XOR problem — something no single linear layer can solve, which is exactly why the hidden layer and non-linearity matter:

import math
def relu(x): return max(0.0, x)

# Hand-set weights that solve XOR: hidden layer builds OR and AND, output combines
def forward(a, b):
    h1 = relu(1*a + 1*b - 0.5)     # fires if a OR b
    h2 = relu(1*a + 1*b - 1.5)     # fires if a AND b
    out = 1*h1 - 2*h2              # OR minus 2*AND = XOR
    return out

for a in (0, 1):
    for b in (0, 1):
        print(f"XOR({a},{b}) = {forward(a, b):.0f}")

Training would discover these weights by gradient descent; here they’re set by hand to show that two linear layers plus a non-linearity can represent a function a single layer cannot.

Engineering Trade-offs

  • Depth/width vs compute. Bigger networks approximate more complex functions and keep improving with scale, but training and inference cost grows with every parameter.
  • Expressiveness vs interpretability. Deep nets learn powerful representations but are opaque — hard to audit or explain compared with linear models or small trees.
  • Generalisation vs memorisation. Enough capacity can memorise the training set; regularisation, dropout, and more data push it toward patterns that transfer.
  • Precision vs throughput. FP16/BF16 and quantised inference cut memory and double throughput at some risk to numerical accuracy the model usually tolerates.

Real-world examples

  • A CNN on your phone classifies photos on-device in milliseconds.
  • A transformer-based LLM writes much of the code in some IDE extensions.
  • A diffusion model generates a never-before-seen “raccoon astronaut on Mars” in seconds.
  • AlphaFold-2 predicts protein structures (a Nobel-winning result) with attention-based networks trained on the Protein Data Bank.

Common misconceptions

  • “Neural networks work like the brain.” Loosely inspired by biology, but the maths is linear algebra and calculus. The analogy stops being useful past introductory teaching.
  • “They reason.” They pattern-match extremely well, sometimes in ways that look like reasoning; whether that is reasoning is genuinely open. They have no introspective access to how they produced an answer.
  • “You need a PhD to use one.” Calling an API, fine-tuning, or running inference is an everyday programming task; inventing new architectures is the part needing deep expertise.

Try it yourself

Run a forward pass of a hand-built network that solves XOR — proof that one hidden layer plus a non-linearity beats any single linear layer (python3 only):

python3 - <<'EOF'
relu=lambda x:max(0.0,x)
def net(a,b):
    h1=relu(a+b-0.5); h2=relu(a+b-1.5)   # OR and AND
    return h1-2*h2                        # = XOR
for a in (0,1):
    for b in (0,1):
        print(f"XOR({a},{b}) = {net(a,b):.0f}")
EOF

Learn next

Neighborhood

A visual companion to the relationships above. Click any node to visit that topic.