Introduction

The least squares method is a fundamental technique in statistical inference and data analysis, designed to estimate the parameters of a linear model by minimizing the sum of squared residuals. In regression analysis, this method seeks to find the best-fit line or curve that minimizes the vertical distance between the observed data points and the predicted values. The core idea revolves around the principle that the optimal coefficients for a linear model are those that reduce the overall error in the predictions. This approach is widely applied in fields such as economics, engineering, and machine learning, where the goal is to quantify relationships between variables and make predictions based on historical data.

The least squares method is particularly relevant in scenarios where the relationship between variables is linear, and the goal is to determine the coefficients that best approximate the data. Unlike other methods, such as maximum likelihood estimation, the least squares method does not require complex computational models or assumptions about the distribution of errors. Instead, it relies on a straightforward optimization process that can be efficiently implemented using algebraic techniques. This simplicity makes the method both robust and versatile, enabling its application in a wide range of contexts.

Mathematical Foundations

The least squares method is rooted in the mathematical formulation of linear regression. Given a set of observed data points $(x_i, y_i)$, where $x_i$ represents the independent variable and $y_i$ the dependent variable, the goal is to determine the coefficients $a$ and $b$ of the linear model $y = a + bx$ that minimize the sum of squared residuals. The residual for each data point is defined as $r_i = y_i - (a + bx_i)$, and the total residual sum is given by:
$$ \sum_{i=1}^n (y_i - a - bx_i)^2 $$
To find the optimal coefficients, the problem reduces to minimizing this function with respect to $a$ and $b$. This involves taking partial derivatives of the residual sum with respect to each coefficient and setting them equal to zero.

The partial derivative with respect to $a$ yields:
$$ \frac{\partial}{\partial a} \sum_{i=1}^n (y_i - a - bx_i)^2 = -2 \sum_{i=1}^n (y_i - a - bx_i) $$
Setting this equal to zero and simplifying leads to the equation:
$$ \sum_{i=1}^n (y_i - a - bx_i) = 0 $$
Similarly, the partial derivative with respect to $b$ results in:
$$ \frac{\partial}{\partial b} \sum_{i=1}^n (y_i - a - bx_i)^2 = -2b \sum_{i=1}^n (y_i - a - bx_i) $$
Setting this equal to zero and simplifying gives:
$$ \sum_{i=1}^n (y_i - a - bx_i) = 0 $$
These two equations form the normal equations, which provide the values of $a$ and $b$ that minimize the sum of squared residuals. Solving these equations involves solving a system of linear equations, which can be achieved using matrix algebra.

Derivation of the Estimators

The least squares estimators $a$ and $b$ are derived by solving the normal equations. For a linear model $y = a + bx$, the normal equations are:
$$ \sum_{i=1}^n y_i = a \sum_{i=1}^n 1 + b \sum_{i=1}^n x_i $$
$$ \sum_{i=1}^n x_i y_i = a \sum_{i=1}^n x_i + b \sum_{i=1}^n x_i^2 $$
These equations can be written in matrix form as:
$$ \begin{bmatrix} \sum_{i=1}^n 1 & \sum_{i=1}^n x_i \ \sum_{i=1}^n x_i & \sum_{i=1}^n x_i^2 \end{bmatrix} \begin{bmatrix} a \ b \end{bmatrix}

\begin{bmatrix} \sum_{i=1}^n y_i \ \sum_{i=1}^n x_i y_i \end{bmatrix} $$
The solution to this system involves computing the determinants of the coefficient matrix. The value of $a$ is given by:
$$ a = \frac{\left( \sum_{i=1}^n x_i y_i \right) \sum_{i=1}^n x_i^2 - \left( \sum_{i=1}^n x_i \right) \left( \sum_{i=1}^n x_i y_i \right)}{\left( \sum_{i=1}^n x_i^2 \right) \left( \sum_{i=1}^n 1^2 \right) - \left( \sum_{i=1}^n x_i \right)^2} $$
Similarly, $b$ is calculated as:
$$ b = \frac{\left( \sum_{i=1}^n y_i \right) \sum_{i=1}^