Neural Networks and Deep Learning

The neuron

An artificial neuron is a very small calculation: multiply each input by a weight, add them up with a bias, and pass the result through a non-linear activation function. Training adjusts the weights and biases so that the network’s outputs match the examples.

z = w₁x₁ + w₂x₂ + … + b output = f(z)

Layers and activations

Neurons are arranged in layers: an input layer (your features), one or more hidden layers and an output layer. Each neuron takes all the outputs of the layer before. A network with several hidden layers is “deep”.

Activation	Formula	Use
ReLU	max(0, z)	The default for hidden layers: simple, fast, trains well
Sigmoid	1 ÷ (1 + e−z)	Output of a yes/no classifier (a probability from 0 to 1)
Softmax	ezk ÷ Σ ez	Output for several classes: probabilities that add to 1
Tanh	(ez − e−z) ÷ (ez + e−z)	Outputs between −1 and 1; used in recurrent networks
Linear (none)	z	Output for a regression (any number)

A forward pass by hand

Training: loss and gradient descent

Training measures how wrong the network is with a loss function and nudges every weight in the direction that reduces it. For regression the usual loss is the mean squared error; for classification it is the cross-entropy, −[y ln p + (1 − y) ln(1 − p)], which punishes confident wrong answers heavily. The direction comes from the gradient, the slope of the loss with respect to each weight, computed efficiently by backpropagation (the chain rule applied layer by layer). The update is:

w ← w − η · ∂L/∂w (η = the learning rate)

Loss	For three predictions 0.9, 0.6, 0.2 with labels 1, 1, 0	Used for
Mean squared error	0.070	Regression; smooth, but weak at punishing wrong confident classes
Cross-entropy	0.280	Classification: the standard choice

The learning rate

The learning rate is the most important setting. Too small and training crawls; too large and the weights overshoot and the loss grows. A simple bowl-shaped loss L = (w − 3)² shows the whole story, because each step multiplies the distance from the optimum by (1 − 2η).

Counting parameters

A dense layer with n_in inputs and n_out neurons has n_in × n_out weights plus n_out biases. The parameter count sets the memory (flash and RAM, see Deployment, Monitoring and Limits), and the number of multiply-accumulate operations (MACs) sets the time and energy of one inference.

Convolutional networks for signals and images

A convolutional network (CNN) slides small filters across the input and shares the same weights at every position. It needs far fewer parameters than a dense network on the same input, and detects a pattern wherever it occurs: an edge anywhere in an image, an impact anywhere in a vibration window. Convolutions on one-dimensional signals (1-D CNNs) and on spectrograms or images (2-D CNNs) are the workhorse for sensor and vision tasks.

• 2-D CNNs work on images and spectrograms; well-known designs (ResNet, MobileNet, EfficientNet) are available pretrained.
• Pooling shrinks the signal between layers; global pooling at the end removes the dependence on the input length.
• Batch normalization and residual connections make deeper networks train reliably.

Sequences

For data where order matters (time series, text) there are recurrent networks (LSTM, GRU), which carry a memory from step to step, and transformers, which relate every position to every other with attention. They are powerful and heavy; for sensor windows, a well-featured random forest or a small 1-D CNN is often as good and far cheaper (see Features for Signals and Machines and Trees, Clustering and Anomaly Detection). Large language models and foundation models belong here too: use them through a service or as a pretrained component, not as something to train from scratch.

Pretrained models and transfer learning

Training a large network from scratch needs a great deal of data and computing. Transfer learning starts from a network already trained on millions of images (or hours of audio), keeps its early layers, which have learned generally useful patterns such as edges and textures, and trains only a new final layer, or fine-tunes the whole network gently, on your own examples. A few hundred labeled images can then be enough to recognize defects on your product.

• Freeze the base, train the head first; unfreeze with a small learning rate (such as 10⁻⁵) only if you have enough data.
• Match the pre-processing (image size, normalization) to what the pretrained network expects.
• Check licenses and data rights of the pretrained weights and of any training images.
• Collect examples of your own conditions (lighting, angle, dirt, camera): a network that learned on clean images fails on a dusty lens (see Robotics Tools).

import tensorflow as tf
base = tf.keras.applications.MobileNetV2(input_shape=(160, 160, 3), include_top=False, weights="imagenet")
base.trainable = False                                      # keep the pretrained layers fixed at first
model = tf.keras.Sequential([base, tf.keras.layers.GlobalAveragePooling2D(), tf.keras.layers.Dropout(0.2),
                             tf.keras.layers.Dense(1, activation="sigmoid")])      # new head: defect / no defect
model.compile(optimizer=tf.keras.optimizers.Adam(1e-3), loss="binary_crossentropy", metrics=["AUC"])
model.fit(train_ds, validation_data=val_ds, epochs=10,
          callbacks=[tf.keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)])

Training in practice

• Normalize the inputs (zero mean, unit variance, from the training data only; see Data Preparation, Splitting and Leakage).
• Watch both loss curves. Training loss falling while validation loss rises is overfitting (see Overfitting, Validation and Tuning): use early stopping, dropout, weight decay, augmentation or a smaller network.
• Start small. Make a tiny network overfit a tiny set (a sanity check that the code works), then scale up.
• Fix the random seeds and save the model with its version of the data and code so that a result can be reproduced.
• Class imbalance needs class weights or a balanced sampler (see Data Preparation, Splitting and Leakage).
• GPUs speed up training by 10 to 100 times on large networks; small networks on tables train fine on a laptop CPU.
• Deep learning is rarely the first thing to try on a table of numbers; try the line, a forest and good features first (see What Machine Learning Is and When to Use It).

Related

• Machine Learning: the overview of this guide.
• Features for Signals and Machines: features, or spectrograms, to feed a network.
• Deployment, Monitoring and Limits: fitting a trained network on a microcontroller.
• Overfitting, Validation and Tuning: keeping a network honest.
• Robotics Tools: perception networks in robots.

FAQ

How many layers and neurons do I need?

Start small (one or two hidden layers of 16 to 128 neurons for tabular data) and grow only if the validation error says you are underfitting. For images and signals use a small CNN or a pretrained network.

Why is my loss not falling?

Usually the learning rate is too high or too low, the inputs are not normalized, the labels are wrong or the code has a bug. Check that the network can overfit a tiny set, then try learning rates across a log scale.

What is the difference between an epoch, a batch and a step?

An epoch is one pass through all training data; a batch is the group of examples used for one weight update; a step is one update. Steps per epoch = data size ÷ batch size.

Can I trust a neural network’s probability?

Not blindly. Networks are often over-confident, and on inputs unlike their training data they can be confidently wrong. Calibrate the probabilities, test on unfamiliar data and keep a human in the loop for important decisions.

References

• Goodfellow, I., Bengio, Y. and Courville, A., Deep Learning, MIT Press (free online).
• Géron, A., Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow, O’Reilly; Chollet, F., Deep Learning with Python, Manning.
• Kingma, D. P. and Ba, J., “Adam: a method for stochastic optimization”, ICLR 2015; the PyTorch and TensorFlow/Keras documentation.

Last reviewed: October 2026. The small networks here are for illustration and are computed by hand; real networks use the same arithmetic at much larger scale.