When the [[domain]] is "continuous" (ie [[subset]] of some $\mathbb{R}^{D}$). Typically [[discrete time|iterative]] [[algorithm]]s. A continuous optimization algorithm used in [[machine learning]] is usually called a **fitting method** [[zeroth order optimization]] - [[natural selection]] [[gradient based optimization]] (first order) - [[backpropagation]] algorithm for computing gradients of [[artificial neural network]]s [[Hessian-based optimization]] (second order) # implementation - [[autodiff]] - [[efficient training]] - [[ResNet#^resnet-training]] - [[efficient deep learning]] - [[efficient finetuning]] - [[artificial neural network]] [`torch.optim` documentation](https://pytorch.org/docs/stable/optim.html#module-torch.optim) - can vary per-layer learning rates by passing in list of `{ "params": [], ...keywords }` as the learning rate - can adjust [[learning rate]] over time https://github.com/ClashLuke/TrueGrad [[Optax]] has good list # sources [[2018HeathScientificComputing]]