Q&A

How to create a simple neural network in Python

By AlgoInsight Team2026-06-07data-structuresmachine-learningnumpypython

Overview

A neural network maps inputs to outputs through layers of neurons. Each neuron computes a weighted sum of its inputs plus a bias, then applies a non-linear activation (ReLU, sigmoid). Training adjusts the weights via backpropagation + gradient descent to reduce a loss. Understanding the from-scratch version illuminates what high-level frameworks do for you.

graph LR
    I[Input layer] --> H[Hidden layer: weighted sum + activation]
    H --> O[Output layer]
    O --> L[Loss vs target]
    L -->|backprop gradients| H
    H -->|update weights| H
    style H fill:#1e1b4b,color:#fff

Level 1: From Scratch with NumPy (a single hidden layer)

Shows the mechanics - forward pass, sigmoid, and gradient-based weight updates:

import numpy as np

# XOR-like toy data
X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([[0],[1],[1],[0]])

def sigmoid(x):  return 1 / (1 + np.exp(-x))
def dsigmoid(x): return x * (1 - x)        # derivative given sigmoid output

np.random.seed(1)
w1 = np.random.randn(2, 4)    # input → hidden (4 neurons)
w2 = np.random.randn(4, 1)    # hidden → output
lr = 0.5

for epoch in range(10000):
    # forward pass
    h = sigmoid(X @ w1)
    out = sigmoid(h @ w2)
    # backpropagation
    err = y - out
    d_out = err * dsigmoid(out)
    d_h   = (d_out @ w2.T) * dsigmoid(h)
    # gradient descent updates
    w2 += h.T @ d_out * lr
    w1 += X.T @ d_h   * lr

print(out.round(3))   # approaches [[0],[1],[1],[0]]

This is the whole loop: forward (compute prediction), compute error, backward (propagate gradients), update weights - repeated over epochs.

Level 2: Keras / TensorFlow (concise, production-friendly)

import tensorflow as tf
from tensorflow.keras import layers, models

model = models.Sequential([
    layers.Dense(16, activation="relu", input_shape=(4,)),
    layers.Dense(8,  activation="relu"),
    layers.Dense(3,  activation="softmax"),   # 3-class classification
])
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
model.fit(X_train, y_train, epochs=50, validation_split=0.2)
model.evaluate(X_test, y_test)

Keras hides the backprop loop: compile sets the optimizer/loss, fit trains.

Level 3: PyTorch (explicit, flexible)

import torch, torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4, 16)
        self.fc2 = nn.Linear(16, 3)
    def forward(self, x):
        x = torch.relu(self.fc1(x))
        return self.fc2(x)

model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(50):
    optimizer.zero_grad()           # reset gradients
    out = model(X)                  # forward
    loss = criterion(out, y)
    loss.backward()                 # backprop
    optimizer.step()                # update weights

PyTorch makes the training loop explicit (zero_grad → forward → backward → step).

The Universal Ingredients

Tips

• Scale/normalize inputs (as in scikit-learn) for stable training.
• Pick the right output + loss: softmax + cross-entropy for multiclass, sigmoid + binary cross-entropy for binary, linear + MSE for regression.
• Watch for overfitting (use validation split, dropout, regularization).

Why This Matters in Interviews

ML interviews use this to gauge whether you understand neural nets, not just call libraries. The from-scratch NumPy version demonstrates the forward/backward/update cycle and why activations introduce non-linearity. The framework versions show practical skill: layer stacking, choosing loss/optimizer, and the training loop (zero_grad → forward → backward → step in PyTorch; compile/fit in Keras). Naming the universal ingredients and the output-layer/loss pairing per task type signals real understanding.

Common Pitfalls

• Omitting activations (the network collapses to a linear model).
• Not scaling inputs → slow/unstable training.
• Wrong output activation + loss for the task (e.g. softmax for regression).
• In PyTorch, forgetting optimizer.zero_grad() (gradients accumulate).
• Judging the model on training data instead of a held-out set (overfitting).

References

• Keras Sequential guide: https://keras.io/guides/sequential_model/
• PyTorch beginner tutorial: https://pytorch.org/tutorials/beginner/basics/intro.html

More answers