BitWithBite
Applied Mathematics · Quick Reference
Machine Learning Mathematics Cheat Sheet
In one line: machine learning relies on a handful of core mathematical ideas from linear algebra, calculus, and statistics — understanding these foundations demystifies how models actually learn from data.
Key Ideas
1Loss Function. A function that measures how wrong a model's predictions are compared to the actual values; training a model means trying to minimize this function.
2Gradient Descent. An optimization algorithm that repeatedly adjusts a model's parameters in the direction that most reduces the loss function, using its gradient (derivative).
3Learning Rate. A small constant that controls how big each step is during gradient descent; too large risks overshooting, too small makes training very slow.
4Vectors and Matrices in ML. Data (like an image or a row of features) is typically represented as a vector, and datasets as matrices, so that models can use fast matrix operations to make predictions.
5Overfitting. When a model learns the training data too precisely, including its noise and quirks, causing it to perform poorly on new, unseen data.
Worked Examples
A model has a parameter w=5, learning rate 0.1, and the gradient of the loss at w=5 is 4. Find the updated parameter.
w_new = 4.6
Why is the learning rate important in gradient descent?
It balances training speed against the risk of overshooting the optimal solution
A model achieves 99% accuracy on training data but only 60% on new test data. What problem does this indicate?
Overfitting
Formulas
Gradient descent update: parameter_new = parameter_old - (learning_rate * gradient)
Practice Yourself
What does a loss function measure?
How wrong a model's predictions are compared to actual values
What algorithm minimizes a loss function by following its gradient?
Gradient descent
What happens if the learning rate is set too high?
The algorithm may overshoot and fail to converge