Modern neural networks are rarely trained with plain stochastic gradient descent alone. In practice, teams rely on adaptive methods because they reduce manual tuning and often converge faster in the early stages. If you are learning optimisation concepts through an AI course in Kolkata, it helps to understand why different adaptive optimisers behave differently, and what “convergence” really means in theory versus real training runs.
Adaptive optimisers adjust the effective step size per parameter using gradient statistics. That can stabilise training when gradients vary across layers, or when features are sparse. However, adaptivity also changes the convergence guarantees you may expect from classical SGD analysis.
What “convergence” means in adaptive optimisation
In convex optimisation, convergence often means reaching the global optimum and having theoretical bounds on regret or suboptimality. In deep learning (a largely non-convex setting), convergence typically means reaching a stationary point (where gradients are small) and doing so reliably without exploding or stalling.
A key detail: adaptive methods scale updates using either a cumulative sum of squared gradients (Adagrad-style) or an exponential moving average (RMSProp/Adam-style). These choices directly influence:
- Stability (how sensitive training is to noisy gradients)
- Speed (how quickly loss decreases early on)
- Long-run behaviour (whether the effective learning rate shrinks too much or remains responsive)
Adagrad and its variants: strong for sparse features, but shrinking steps
Adagrad accumulates squared gradients over time and divides the learning rate by the square root of that accumulation. Parameters with frequent large gradients get smaller future updates; rarely-updated parameters keep relatively larger steps.
Convergence behaviour (high level):
- In convex settings, Adagrad has strong regret bounds and is well suited to sparse problems (e.g., NLP with one-hot features).
- In practice, the effective learning rate decays monotonically, which can cause training to slow down later. You may see quick initial progress, then a “plateau” because steps become tiny.
Common variants:
- Adadelta and RMSProp-like ideas were partly motivated by fixing Adagrad’s aggressive decay by replacing the full cumulative sum with a moving window/average.
- FTRL-style variants extend Adagrad for online learning with regularisation, often used in large-scale linear models.
A practical takeaway: Adagrad can be excellent when gradients are sparse and you want strong early convergence, but it may require a higher initial learning rate or a reset strategy if training drags late in optimisation.
RMSProp family: steadier progress via exponential averaging
RMSProp replaces Adagrad’s cumulative accumulator with an exponential moving average of squared gradients. This prevents the denominator from growing without bound, so learning rates do not collapse over time.
Convergence behaviour (high level):
- The moving average makes RMSProp responsive to recent curvature and gradient scale, which often stabilises training in non-stationary settings.
- RMSProp tends to be less prone to late-stage stalling than Adagrad because the effective step size does not shrink indefinitely.
Limitations to note:
- RMSProp’s theoretical guarantees are less straightforward than Adagrad’s in classic convex analysis, largely because the exponential averaging complicates the bound structure.
- Performance is sensitive to the decay rate (often denoted ρ) and the epsilon term added for numerical stability.
If you are applying these ideas after an AI course in Kolkata, RMSProp is a good baseline for recurrent networks and noisy objectives, especially when you want stable, consistent progress without the “Adagrad slowdown”.
Adam and variants: fast and robust, but subtle convergence caveats
Adam combines:
- a momentum-like term (exponential moving average of gradients; first moment), and
- an RMSProp-like term (exponential moving average of squared gradients; second moment),
- with bias correction to counteract early-iteration underestimation.
Convergence behaviour (high level):
- Adam often shows rapid initial loss reduction and handles ill-conditioned problems well.
- However, classical results show that Adam can fail to converge in some constructed cases, even in convex settings, because the adaptive scaling may produce problematic update directions over time.
Common fixes and variants:
- AMSGrad modifies the second-moment accumulator to be non-decreasing, which restores certain convergence guarantees in convex scenarios.
- AdamW decouples weight decay from the gradient update, improving generalisation and making regularisation behaviour more predictable.
- Nadam adds Nesterov-style momentum, sometimes improving responsiveness.
In practical deep learning, Adam/AdamW are widely used because they are forgiving and efficient. Still, if you observe instability or poor final performance, switching to AdamW, tuning β₂, or using a scheduler can materially change the convergence trajectory.
How schedulers interact with adaptive optimisers
Even though these optimisers are “adaptive,” external learning rate schedules often help. Common patterns include:
- Warmup: reduces early instability, especially for Adam-family methods on large batch training.
- Cosine decay or step decay: improves late-stage convergence by shrinking the global learning rate once progress slows.
- Reduce-on-plateau: reacts to validation stagnation and can prevent wasted epochs.
A reliable workflow is: use AdamW (or RMSProp) for fast stabilisation, then apply a decay schedule to improve the final basin quality, especially when chasing better generalisation rather than only faster loss reduction.
Conclusion
Adagrad offers strong behaviour for sparse gradients but can slow down as its step sizes shrink. RMSProp avoids that shrinkage with exponential averaging and often converges steadily in noisy, non-stationary training. Adam combines momentum and adaptive scaling, delivering fast early convergence, but its theoretical convergence can be weaker without variants like AMSGrad, and its practical results often improve with AdamW and sensible scheduling. If your optimisation toolkit is growing through an AI course in Kolkata, focusing on how each method scales updates over time will help you choose the right optimiser and scheduler combination for stable, efficient training.