Learning Rate

The size of each gradient descent step: too large overshoots and bounces around, too small crawls and wastes time, just right converges steadily.

When to use

Use it when asked what goes wrong in training: an oversized step overshoots the minimum of the loss, an undersized one is very slow.

See Lec 8.

Appears in

No references yet.