SenatorovAI | IT-разработка и консалтинг | Школа Data Science Руслана Сенаторова

My first deep dive into Linear Regression I'm on a mission to destroy the black box beliefs about ML in my head, and Linear Regression is where I started. Before this I had no idea there were actually two different approaches to solving it, analytical methods and numerical methods, so that alone was a big realisation for me. I learned the closed form solution, the normal equation, and how to derive it completely from scratch instead of just using it as a ready made formula. I saw how it comes from the loss function, the sum of squared errors, by taking partial derivatives with respect to theta zero and theta one and setting them to zero. That gives you a system of two equations with two unknowns, which is the scalar form of the normal equation. I also figured out how this scalar form connects to the matrix form, since writing out X^T X and X^T y explicitly for simple linear regression gives you the exact same system. On top of that I learned when the analytical approach actually makes sense and when it doesn't. Inverting X^T X costs O(n^3) where n is the number of features, so it works fine with a lot of samples but becomes slow and unstable once you have a lot of features. That's exactly where numerical methods like Gradient Descent come in, since they don't need matrix inversion and scale much better.