Learning Rate Schedule Calculator

A learning rate schedule is a rule that sets the optimizer’s learning rate at every training step, typically raising it during a short warmup and then lowering it toward a minimum. This learning rate schedule calculator returns the exact rate at any step for five common schedules, plus the average rate, the half-peak step and a plot of the curve.

What is a learning rate schedule?

The learning rate is the number that scales every weight update: new weight = old weight − learning rate × gradient. A schedule replaces that single fixed number with a function of the step count, so the optimizer takes differently sized steps at different points in training. The schedule is defined by a peak (or base) rate, the total number of steps, and a handful of shape parameters such as the warmup length or a decay factor.

The tool above works as a learning rate scheduler visualizer: it evaluates the same closed-form expressions that training frameworks use, so the number it shows for step t is the number your optimizer would be using at that step.

Why is the learning rate changed during training?

Two different problems sit at the two ends of a training run. At the start, the weights are random, the gradients are large and point in inconsistent directions, and adaptive optimizers such as Adam have only a few batches behind their running averages. A full-sized step at that moment can throw the weights far from any sensible region, and the loss diverges or stalls. Warmup avoids this by starting near zero and increasing the rate linearly over the first W steps.

At the end, the opposite holds. The model is close to a good solution and gradient noise from mini-batches is now the main thing moving the weights. A large rate keeps the weights bouncing around the minimum instead of settling into it, so the rate is decayed: smaller steps average out the noise and let the loss finish its descent. If you are watching the loss fall with the cross-entropy loss calculator open beside your logs, the late plateau that follows a rate drop is this effect.

Learning rate schedule formulas

Every schedule here is a function of the step index \(t\), which runs from 0 to \(T\). The three warmup schedules share the same first phase, for \(t < W\):

From \(t = W\) onward the schedules differ. Cosine decay and linear decay are:

Constant with warmup just holds \(\eta_t = \eta_{\text{peak}}\). The two schedules without warmup multiply a base rate by a factor \(\gamma\):

The symbols are: \(\eta_t\), the learning rate at step \(t\); \(\eta_{\text{peak}}\) or \(\eta_{\text{base}}\), the highest rate; \(\eta_{\min}\), the floor the decay ends at; \(T\), the total number of steps; \(W\), the number of warmup steps; \(s\), the step size (how many steps pass between drops); and \(\gamma\), the decay factor between 0 and 1. With \(W = 0\) the warmup phase is empty and the decay formula applies from the first step.

Worked example: cosine annealing with warmup

Take the calculator’s default input: peak rate 1e-3, minimum 0, \(W = 100\) warmup steps, \(T = 1000\) total steps, and query step \(t = 325\). Step 325 is past the warmup, so the cosine formula applies. The progress through the decay phase is (325 − 100) / (1000 − 100) = 225 / 900 = 0.25, and cos(0.25π) = 0.707107.

So a quarter of the way through the decay, the rate is still 85.4% of its peak. Other steps from the same run: \(t = 50\) is halfway through warmup and gives 5e-4, \(t = 100\) gives the peak 1e-3, \(t = 550\) is the midpoint of the decay and gives 5e-4 again, and \(t = 1000\) gives 0. The average over the 1000 steps is exactly 5e-4, half the peak.

Which learning rate schedule should you use?

The choice mostly comes down to the model family and to how many extra numbers you are willing to tune.

One result worth knowing: for the same peak, minimum, warmup and total steps, cosine and linear decay have the same average learning rate. They differ in where they spend it. In the default example the cosine schedule is at 9.330e-4 a quarter of the way through training and 1.786e-4 at three quarters, while linear decay is at 8.333e-4 and 2.778e-4 at the same points. Cosine stays high for longer and then finishes lower.

Exponential decay is the same curve you would get by fitting data with the exponential regression calculator: a constant ratio between consecutive values. Because that ratio is applied every step, gamma must be very close to 1 when you count optimizer steps. A gamma of 0.95 halves the rate in 14 steps.

Do schedulers count steps or epochs?

A scheduler counts how many times you call it, nothing more. In PyTorch each call to scheduler.step() advances \(t\) by one. Call it after every optimizer update and \(T\), \(W\) and the step size are all measured in optimizer steps; call it once per epoch and the same numbers mean epochs. The familiar “step size 30, gamma 0.1” recipe assumes per-epoch calls, while warmup schedules for transformers are almost always stepped per batch.

That is why the totals matter. One epoch is (number of training examples / batch size) optimizer steps, divided again by any gradient accumulation factor. The training steps calculator does that conversion and gives you the \(T\) and \(W\) to enter here. Batch size itself is usually capped by memory, which depends on model size; the neural network parameter calculator gives the weight count behind that limit.

Learning rate schedules in PyTorch

The built-in schedulers cover step, exponential and plain cosine decay. Each wraps an optimizer whose lr is the base rate:

import torch

model = torch.nn.Linear(10, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

# step decay: 0.1 -> 0.01 at step 30 -> 0.001 at step 60
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)

# exponential decay: lr = 0.1 * 0.95**t
# scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.95)

# cosine decay with no warmup, reaching eta_min after T_max steps
# scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=900, eta_min=0.0)

For cosine annealing with warmup, LambdaLR lets you write the formula yourself. It multiplies the optimizer’s base rate by whatever your function returns, so the function returns the rate divided by the peak. This reproduces the worked example:

import math
import torch

peak, lr_min, W, T = 1e-3, 0.0, 100, 1000

def lr_at(t):
    if t < W:
        return peak * t / W
    progress = (t - W) / (T - W)
    return lr_min + 0.5 * (peak - lr_min) * (1 + math.cos(math.pi * progress))

print(lr_at(325))                      # 0.000853553...

model = torch.nn.Linear(10, 1)
optimizer = torch.optim.AdamW(model.parameters(), lr=peak)
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda t: lr_at(t) / peak)

for step in range(325):
    optimizer.step()                   # normally preceded by loss.backward()
    scheduler.step()

print(scheduler.get_last_lr()[0])      # 0.000853553...

In Keras the equivalent is keras.optimizers.schedules.CosineDecay, passed to the optimizer as its learning rate. The PyTorch classes are documented in the official torch.optim reference, under “How to adjust learning rate”, with a separate page for LambdaLR.

Common mistakes

• Mixing up steps and epochs. Setting T_max or the step size in epochs and then calling the scheduler every batch finishes the whole schedule inside the first epoch or two.
• Calling the scheduler before the optimizer. In PyTorch, optimizer.step() comes first and scheduler.step() second; the other order skips the first value of the schedule.
• Reusing an exponential gamma across units. A gamma of 0.95 per epoch is gentle. The same 0.95 per optimizer step leaves under 1% of the base rate after 90 steps.
• Forgetting that warmup starts at zero. At \(t = 0\) the formula gives a rate of exactly 0, so the very first update does nothing. Some implementations use \((t + 1) / W\) to avoid that; this calculator uses \(t / W\).
• Resuming with the wrong total. If a run is extended, the old \(T\) no longer matches. A cosine curve evaluated past \(T\) starts rising again.
• Treating the peak as model-independent. A larger model or batch size usually needs a different peak rate; the schedule only shapes how that peak is reached and left.

Frequently asked questions

How many warmup steps should I use?

There is no universal value. A common starting point is a few percent of the total steps: the default example uses 100 warmup steps out of 1000, which is 10 percent. Treat the warmup length as a hyperparameter and increase it if the loss spikes or diverges in the first few hundred steps.

What is the learning rate at the end of warmup?

Exactly the peak learning rate. At step t = W the cosine and linear decay formulas both return the peak, and the warmup line reaches the same value, so the schedule has no jump where the two phases meet.

Is cosine annealing better than linear decay?

Neither is better in every case. With the same peak, minimum, warmup and total steps they have the same average learning rate. Cosine keeps the rate near the peak for longer and flattens out at the end, while linear decay falls at a constant pace. Cosine is the usual choice for training from scratch and linear decay is common for fine-tuning.

Does a learning rate scheduler count steps or epochs?

It counts calls. Each call to the scheduler advances the schedule by one, so if you call it after every optimizer update the units are optimizer steps, and if you call it once per epoch the units are epochs. Total steps, warmup steps and step size must all be in the same unit.

Related calculators

Browse every tool for network training in the deep learning calculators hub, or check whether the model you are scheduling will fit on your GPU with the LLM VRAM calculator.

• Total and warmup steps from dataset size, batch size and epochs
• Cross-entropy loss for a set of predictions
• Count the parameters in a neural network
• Sigmoid activation and its derivative