Unit 2: Multivariate Problems, Gradient-Based Methods

Feb 20: Lecture 4 — Local Descent

Contents:

  • Descent Direction Iteration
  • Line Search
  • Approximate Line Search
  • Trust Region Methods
  • Termination Conditions

Resources:

  • Slides
  • Chapter 4 and Appendix of [KW]

Feb 25: Lecture 5 — First-Order Methods

Contents:

  • Gradient Descent
  • Conjugate Gradient
  • Momentum
  • Nesterov Momentum
  • Adagrad
  • RMSProp
  • Adadelta
  • Adam
  • Hypergradient Descent

Resources:

Feb 27: Exercise Session 2

March 3: Lecture 6 — Second-Order Methods and Direct Methods

Second-Order Methods:

  • Newton’s Method
  • Secant Method
  • Quasi-Newton Methods

Resources:

Direct Methods:

  • Cyclic Coordinate Search
  • Powell’s Method
  • Hooke-Jeeves
  • Generalized Pattern Search
  • Nelder-Mead Simplex Method
  • (Divided Rectangles)

Resources:

March 6: Exercise Session 3