Neural Network
Also known as: neural networks, deep learning, artificial neural network
A family of machine learning models loosely inspired by the brain — layers of simple units that, together, can approximate complex functions.
- Primary domain
- Machine Learning
- Sub-category
- Supervised & Unsupervised Learning
In simple terms
A neural network is a stack of layers. Each layer is a pile of simple units (“neurons”) that each take a weighted sum of their inputs, push it through a non-linear function, and pass the result on. Stack enough of these layers together, train the weights on a lot of data, and the network can learn to recognise faces, translate languages, generate images, or play games — without anyone writing the rules by hand. The “deep” in deep learning just means many layers.
The Visual Map
graph LR
x["Input x"] --> h1["Hidden layer 1\ny = f(W·x + b)"]
h1 --> h2["Hidden layer 2"]
h2 --> out["Output layer\n(prediction)"]
out --> loss["Loss vs correct answer"]
loss -. "backprop: ∂loss/∂W" .-> h2
loss -. .-> h1
h1 --- act["f = ReLU / sigmoid\n(non-linearity is essential)"]
More detail
A neuron computes y = f(W · x + b) — a weighted sum of inputs x with weights W and bias b, passed through a non-linear activation function f (ReLU is the default; older networks used sigmoid or tanh). The non-linearity is essential; without it, any stack of layers collapses into a single linear function.
Training works the same way for almost every modern network: a forward pass pushes input through and compares the output to the correct answer with a loss function; backpropagation uses the chain rule to compute how much each weight contributed to the loss; gradient descent nudges every weight a tiny step to reduce the loss; repeat for billions of examples. Major architectural families include the MLP (fully connected, the simplest deep model), the CNN (weight-sharing across spatial regions, the workhorse of vision in ~2012–2020), RNN/LSTM/GRU (for sequences, now largely replaced), the transformer (attention-based, under every modern LLM and a growing share of vision/audio), and diffusion models (iteratively denoise random noise into an image). Two realities shape everything: GPUs/TPUs are essential because every layer is dense matrix multiplication, and performance scales smoothly with more data, parameters, and compute (the “scaling laws”).
Under the Hood
A neural network is just matrix multiplies interleaved with a non-linearity. Here a tiny two-layer net does a forward pass on the XOR problem — something no single linear layer can solve, which is exactly why the hidden layer and non-linearity matter:
import math
def relu(x): return max(0.0, x)
# Hand-set weights that solve XOR: hidden layer builds OR and AND, output combines
def forward(a, b):
h1 = relu(1*a + 1*b - 0.5) # fires if a OR b
h2 = relu(1*a + 1*b - 1.5) # fires if a AND b
out = 1*h1 - 2*h2 # OR minus 2*AND = XOR
return out
for a in (0, 1):
for b in (0, 1):
print(f"XOR({a},{b}) = {forward(a, b):.0f}")
Training would discover these weights by gradient descent; here they’re set by hand to show that two linear layers plus a non-linearity can represent a function a single layer cannot.
Engineering Trade-offs
- Depth/width vs compute. Bigger networks approximate more complex functions and keep improving with scale, but training and inference cost grows with every parameter.
- Expressiveness vs interpretability. Deep nets learn powerful representations but are opaque — hard to audit or explain compared with linear models or small trees.
- Generalisation vs memorisation. Enough capacity can memorise the training set; regularisation, dropout, and more data push it toward patterns that transfer.
- Precision vs throughput. FP16/BF16 and quantised inference cut memory and double throughput at some risk to numerical accuracy the model usually tolerates.
Real-world examples
- A CNN on your phone classifies photos on-device in milliseconds.
- A transformer-based LLM writes much of the code in some IDE extensions.
- A diffusion model generates a never-before-seen “raccoon astronaut on Mars” in seconds.
- AlphaFold-2 predicts protein structures (a Nobel-winning result) with attention-based networks trained on the Protein Data Bank.
Common misconceptions
- “Neural networks work like the brain.” Loosely inspired by biology, but the maths is linear algebra and calculus. The analogy stops being useful past introductory teaching.
- “They reason.” They pattern-match extremely well, sometimes in ways that look like reasoning; whether that is reasoning is genuinely open. They have no introspective access to how they produced an answer.
- “You need a PhD to use one.” Calling an API, fine-tuning, or running inference is an everyday programming task; inventing new architectures is the part needing deep expertise.
Try it yourself
Run a forward pass of a hand-built network that solves XOR — proof that one hidden layer plus a non-linearity beats any single linear layer (python3 only):
python3 - <<'EOF'
relu=lambda x:max(0.0,x)
def net(a,b):
h1=relu(a+b-0.5); h2=relu(a+b-1.5) # OR and AND
return h1-2*h2 # = XOR
for a in (0,1):
for b in (0,1):
print(f"XOR({a},{b}) = {net(a,b):.0f}")
EOF
Learn next
- Transformer — the attention-based architecture behind the current AI wave
- Large language model — what transformers are scaled into
- GPU — the parallel hardware that makes training tractable
- Convolutional neural network — the architecture that dominated computer vision
Read this in a learning path
All paths →This topic is part of a learning path. Start in context to keep prev/next and progress tracking.
Relationships
- Requires
- Next
- Leads to
Neighborhood
A visual companion to the relationships above. Click any node to visit that topic.