Neural Network Parameter Calculator

The number of parameters in a neural network is the count of every learnable weight and bias across all of its layers. This neural network parameter calculator adds that count up layer by layer and returns the total, the trainable versus non-trainable split, and the model size in MB for float32, float16 or int8 weights.

What counts as a parameter in a neural network?

A parameter is a single number the optimizer is allowed to change during training. In a dense layer those numbers are the entries of the weight matrix and the bias vector. In a convolution they are the kernel values, in an embedding they are the rows of the lookup table, and in a recurrent layer they are the gate matrices.

Activation functions, pooling, dropout and flattening add nothing, because they have no learned values. Batch normalization is the one common layer that stores numbers of both kinds: a trainable scale and shift, and running statistics that are saved with the model but never touched by the optimizer.

What is the formula for the number of parameters in each layer?

A dense (fully connected) layer multiplies an input vector by a weight matrix, which you can check with the matrix multiplication calculator, and then adds a bias. Its count is:

Here \(n_{\text{in}}\) is the number of inputs and \(n_{\text{out}}\) the number of units; the second term is the bias and drops out when the bias is switched off. A 2D convolution replaces the full matrix with a small kernel that is reused at every position:

\(k_h\) and \(k_w\) are the kernel height and width, \(C_{\text{in}}\) and \(C_{\text{out}}\) the input and output channels, and \(g\) the number of groups (1 for an ordinary convolution). The other layer types follow the same idea, and the table lists each formula with a checked value.

The Transformer row is one encoder layer in the PyTorch layout: 1,050,624 parameters in self-attention, 2,099,712 in the feed-forward block and 2,048 in the two LayerNorms. It does not include token embeddings or an output head, and the number of attention heads does not change it.

Worked example: how many parameters does a 784-128-64-10 MLP have?

Take a classifier for 28 × 28 images flattened to 784 inputs, with hidden layers of 128 and 64 units and 10 output scores (the ones a softmax turns into class probabilities). This is the calculator’s default model, and each layer is one application of the dense formula.

Stored as float32, the weights take 109,386 × 4 = 437,544 bytes, which is 0.437544 MB or about 0.4173 MiB. Set the decimal places to 6 above to see the same figure.

Where do a model’s parameters actually live?

The table shows the usual pattern: the first dense layer holds almost 92% of the network, because its count grows with the product of two sizes and 784 is the largest size in the model. Embeddings behave the same way. A 30,522-token vocabulary at 768 dimensions is 23,440,896 parameters before a single hidden layer has been added.

Convolutions are cheap by comparison because the same kernel is reused at every image position. A 7 × 7 convolution from 3 to 64 channels has 9,472 parameters whether the image is 32 or 1,000 pixels wide. A dense layer mapping a flattened 224 × 224 × 3 image (150,528 values) to 64 units would need 150,528 × 64 + 64 = 9,633,856. That is why the dense layer placed after the last convolution is often the largest row in a CNN; the Conv2d output size calculator tells you how many values reach it.

Why does the LSTM count differ between PyTorch and Keras?

An LSTM has four gate blocks, each with input weights (i × h), recurrent weights (h × h) and a bias. Keras stores one bias vector of length h per gate. PyTorch stores two, bias_ih and bias_hh, which are simply added together inside the gate, so the second adds parameters without adding modelling capacity.

With i = 10 and h = 20 that is 2,560 against 2,480, a gap of exactly 4h = 80. The layout is described in the PyTorch nn.LSTM documentation. A GRU has three gate blocks; PyTorch and Keras with reset_after=True both keep two bias vectors, so both give 3(i·h + h² + 2h) = 1,920 for the same sizes.

Is parameter count the same as model size or training memory?

No. They are three different quantities. The parameter count is a pure number. The model size is that number times the bytes per value: 4 for float32, 2 for float16, 1 for int8. A saved file is slightly larger than this because it also holds layer names and metadata, and it includes the non-trainable BatchNorm statistics, which is why this calculator counts them in the size.

Training needs more memory again. The framework stores a gradient for every trainable parameter, the optimizer may keep its own state (Adam keeps two extra values per parameter), and the activations of every layer are held for the backward pass. Activation memory grows with batch size and input size, not with parameter count, so it cannot be read off this page. For inference on large language models, the LLM VRAM calculator starts from a parameter count and adds those overheads.

Parameter count does not set training time either. That depends on how many updates you run, which the training steps calculator works out from dataset size, batch size and epochs.

How do you count parameters in PyTorch?

Sum numel() over the model’s parameter tensors. This reproduces the worked example:

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 128), nn.ReLU(),
    nn.Linear(128, 64), nn.ReLU(),
    nn.Linear(64, 10),
)

total = sum(p.numel() for p in model.parameters())
print(total)                 # 109386
print(total * 4 / 1e6)       # 0.437544  (MB as float32)

for name, p in model.named_parameters():
    print(name, tuple(p.shape), p.numel())
# 0.weight (128, 784) 100352
# 0.bias (128,) 128
# 2.weight (64, 128) 8192
# 2.bias (64,) 64
# 4.weight (10, 64) 640
# 4.bias (10,) 10

lstm = nn.LSTM(10, 20)
print(sum(p.numel() for p in lstm.parameters()))   # 2560

Add if p.requires_grad to the generator to count trainable parameters only. In Keras the equivalent is model.count_params(), and model.summary() prints the per-layer breakdown with trainable and non-trainable totals. The weight and bias shapes used above are those given in the PyTorch nn.Linear documentation.

Common mistakes

• Leaving out the biases. 784 × 128 is 100,352, not 100,480. Each layer with a bias adds one value per output unit or filter.
• Putting the image size into a convolution. Height and width change the output shape and the compute cost, never the kernel’s parameter count.
• Comparing an LSTM across frameworks. A PyTorch summary and a Keras summary of the same architecture differ by 4h per LSTM layer, and nothing is wrong.
• Looking for BatchNorm statistics in model.parameters(). PyTorch keeps the running mean and variance as buffers, so they are missing from that sum but present in the saved file.
• Counting tied weights twice. If the output layer shares its matrix with the embedding, that matrix exists once.
• Mixing MB and MiB. 1 MB is 1,000,000 bytes and 1 MiB is 1,048,576 bytes, a difference of almost 5%.

Frequently asked questions

How do you calculate the number of parameters in a neural network?

Count the weights and biases of every layer and add them up. A dense layer with n inputs and m outputs has n times m weights plus m biases. For a network with layer sizes 784, 128, 64 and 10 that gives 100,480 + 8,256 + 650 = 109,386 parameters.

Why does an LSTM have more parameters in PyTorch than in Keras?

PyTorch stores two bias vectors for each of the four gate blocks and Keras stores one. The PyTorch count is 4(ih + h^2 + 2h) and the Keras count is 4(ih + h^2 + h), so they differ by 4h. With input size 10 and hidden size 20 that is 2,560 against 2,480.

How do you convert a parameter count to model size in MB?

Multiply the parameter count by the bytes per value (4 for float32, 2 for float16, 1 for int8) and divide by 1,000,000. A model with 109,386 float32 parameters takes 437,544 bytes, which is 0.437544 MB. Divide by 1,048,576 instead to get MiB.

Are BatchNorm running statistics counted as parameters?

They are stored with the model but not trained. A BatchNorm layer with C channels has 2C trainable values (scale and shift) and 2C non-trainable values (running mean and running variance). Keras reports them as non-trainable params, while PyTorch keeps them as buffers that model.parameters() does not return.

Related calculators

This page is part of the deep learning calculators collection. These tools pick up where a parameter count leaves off:

• Estimate the GPU memory a language model needs
• Feature-map size after a convolution
• Steps and iterations per epoch
• Plan a learning rate schedule