How Far is Adam from Natural Gradient Descent?

Author1, Author2, Author3, Author4, Author5

optimizationdeep learningAdam optimizernatural gradient descent

2610.00004

Builder Relevance

70%

20h ago

Abstract

This paper investigates the geometric relationship between the Adam optimizer and natural gradient descent, revealing context-dependent deviations in optimization performance.

Reality Card

Core Claim

Adam's geometric trajectory deviates significantly from natural gradient descent under ill-conditioning, but it consistently achieves low loss despite this misalignment.

Method / Result

Deviation reaches misalignments of approximately 10^3 in a non-convex small neural network.

Limitations

The study's findings may not be easily reproducible due to the context-dependent nature of Adam's performance across different loss landscapes.

Paper to code

Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.

No verified implementation link has been attached yet. AIBuzzHub will keep this panel separate from unverified search results.

← Back to all papers