Local Extremas and Nesterov Accelerated Gradient

Python. To find a global minimum for a selected function, I compared brute force sweeping with Nesterov Accelerated Gradient (NAG)optimisation technique. By applying a small deltaX increments, I obtained local extremas in the range of selected lower and upper bounds. Then, I obtained the global minimum by iterating a nested NAG function. Finally, I compared iteration counts. I used the function y = x^2 + sin(4x) – 1.5 in the range of [-2, 2] with the deltaXs = 0.001 (brute force) and 0.2 (nested NAG) respectively. The brute force method’s iteration count is 4000 whereas Nesterov Accelerated Gradient’s iteration count is 1137 with its specific parameters.

[1] On the Importance of Initialization and Momentum in Deep Learning (2013), I. Sutskever, J. Martens, G. Dahl, G. Hinton

[2] Nesterov Accelerated Gradient

[3] Top Optimizers for Neural Networks

[4] Desmos Calculator