Q&A

What is a neural network?

By AlgoInsight Team2026-06-18

Overview

Neural networks are the foundation of modern AI. Their power comes from two properties: they can represent arbitrarily complex functions (universal approximation), and they can learn those functions directly from data end-to-end. Adding depth - more layers - allows the network to learn increasingly abstract features: edges → shapes → objects in images, characters → words → meaning in text.

graph LR
    I["Input layer<br/>raw features"] --> H1["Hidden layer 1<br/>learned features"]
    H1 --> H2["Hidden layer 2<br/>more abstract features"]
    H2 --> O["Output layer<br/>prediction"]
    L["Loss function"] -.backpropagation.-> H2
    L -.backpropagation.-> H1
    style H2 fill:#1e1b4b,color:#fff

Anatomy of a Neural Network

Neuron (node)

Each neuron computes: output = activation(Σ wᵢxᵢ + b)

• Weights (w) and bias (b) are learnable parameters.
• Activation function introduces non-linearity (without it, any number of layers collapses to a single linear transformation).

Layers

• Input layer - receives the raw features.
• Hidden layers - intermediate transformations; "deep" means many hidden layers.
• Output layer - produces the prediction (logits for classification, a value for regression).

Key Activation Functions

Forward Pass and Loss

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),    # fully connected layer
    nn.ReLU(),
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Linear(128, 10),     # 10 class outputs
)

output = model(input)                          # forward pass
loss = nn.CrossEntropyLoss()(output, targets)  # compute loss

Backpropagation

Backpropagation is the algorithm for computing gradients - how much each weight contributed to the loss - by applying the chain rule backward through the network. It answers: "if I increase this weight by a tiny amount, how much does the loss change?"

∂L/∂w = ∂L/∂output × ∂output/∂w   (chain rule, applied layer by layer)

Modern frameworks (PyTorch, TensorFlow) compute this automatically (autograd).

Gradient Descent and Optimization

Once gradients are computed, weights are updated to reduce the loss:

w ← w − η × ∂L/∂w

where η is the learning rate. In practice, mini-batch stochastic gradient descent (SGD) or Adam (adaptive learning rates) is used:

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss.backward()   # compute gradients
optimizer.step()  # update weights
optimizer.zero_grad()  # clear accumulated gradients

Key Architectures

Hyperparameters vs Parameters

• Parameters - learned from data: weights and biases.
• Hyperparameters - set before training: learning rate, number of layers, neurons per layer, batch size, dropout rate.

Why This Matters in Interviews

Neural networks are the foundation of all modern AI. An interviewer expects you to explain the forward pass (weighted sum → activation), the role of non-linearity (without it, networks are just linear models), backpropagation (chain-rule gradient computation), and gradient descent (iterative weight updates). Knowing the major architectures (MLP, CNN, RNN, Transformer) and when to use each, and understanding the parameters-vs-hyperparameters distinction, separates a surface-level answer from a deep one.

Common Pitfalls

• Thinking more layers always helps - deeper networks are harder to train (vanishing gradients, need residual connections) and can overfit.
• Forgetting that without activation functions, any depth of layers reduces to a single linear transformation.
• Confusing parameters (learned) with hyperparameters (hand-tuned).
• Not zeroing gradients between batches - they accumulate by default in PyTorch.

References

• 3Blue1Brown neural networks series: https://www.youtube.com/watch?v=aircAruvnKk
• PyTorch tutorial: https://pytorch.org/tutorials/beginner/basics/intro.html

More answers