Method of Parameter Estimation (such as Maximum Likelihood Estimation)

Introduction

Parameter estimation is a fundamental concept in statistical inference, involving the determination of values for unknown parameters in a model that best describe observed data. This process is critical for making predictions, testing hypotheses, and understanding underlying relationships in data. Among the various methods employed in parameter estimation, the Maximum Likelihood Estimation (MLE) stands out as a widely used and robust approach. MLE seeks to find the parameter values that maximize the likelihood function, which quantifies the probability of observing the given data under the assumed model. This method is particularly valuable in scenarios where the model's parameters are unknown and the data is noisy or incomplete.

The theoretical foundation of MLE is rooted in the principles of probability and optimization. Given a dataset $ \mathbf{y} = (y_1, y_2, \dots, y_n) $ and a parametric model $ \mathbf{y} \sim \mathcal{M}(\boldsymbol{\theta}) $, where $ \boldsymbol{\theta} $ represents the parameters to be estimated, MLE aims to solve the optimization problem:
$$ \hat{\boldsymbol{\theta}} = \arg\max_{\boldsymbol{\theta}} \left[ \ell(\boldsymbol{\theta}) \right], $$
where $ \ell(\boldsymbol{\theta}) $ is the likelihood function. This function measures the probability of observing the data given the parameters, and its maximization ensures that the parameters are chosen to best fit the data.

Core Content

Mathematical Framework of Maximum Likelihood Estimation

The likelihood function is derived from the joint probability distribution of the observed data, conditioned on the parameters $ \boldsymbol{\theta} $. For a sample $ \mathbf{y} = (y_1, y_2, \dots, y_n) $, the likelihood function is defined as:
$$ \ell(\boldsymbol{\theta}) = f(\mathbf{y} | \boldsymbol{\theta}), $$
where $ f(\mathbf{y} | \boldsymbol{\theta}) $ is the probability density function (PDF) or probability mass function (PMF) of the model. In the case of continuous distributions, this function is often expressed using the multivariate normal distribution:
$$ f(\mathbf{y} | \boldsymbol{\theta}) = \frac{1}{\sqrt{(2\pi)^n \det(\boldsymbol{\Sigma})}} \exp\left(-\frac{1}{2} (\mathbf{y} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{y} - \boldsymbol{\mu}) \right), $$
where $ \boldsymbol{\mu} $ is the mean vector and $ \boldsymbol{\Sigma} $ is the covariance matrix. For discrete distributions, the likelihood function is similarly constructed using the binomial or Poisson PMF.

The MLE procedure involves finding the parameter values $ \boldsymbol{\theta} $ that maximize $ \ell(\boldsymbol{\theta}) $. This is typically achieved by taking the partial derivatives of the likelihood function with respect to each parameter, setting them equal to zero, and solving the resulting system of equations. For example, in the case of the normal distribution, the MLE for the mean $ \boldsymbol{\mu} $ is the sample mean $ \bar{y} $, while the MLE for the variance $ \boldsymbol{\sigma}^2 $ is the sample variance $ s^2 $. These results are derived from the fact that the likelihood function is maximized when the parameters align with the data's central tendency and spread.

Properties of Maximum Likelihood Estimators

The MLE estimator possesses several important properties that make it a cornerstone of statistical inference. One of the most notable is consistency, which guarantees that the MLE converges to the true parameter values as the sample size increases. This property is supported by the Law of Large Numbers and is critical for ensuring reliable parameter estimation.

Another key property is asymptotic normality, which states that the distribution of the MLE approaches a normal distribution with mean $ \boldsymbol{\theta} $ and variance $ \boldsymbol{\Sigma}^{-1} $ as the sample size grows. This allows for the construction of confidence intervals and hypothesis tests, providing a robust framework for statistical inference. Additionally, the MLE is efficient, meaning it achieves the lowest possible variance among all unbiased estimators, as dictated by the Cramér–Rao Lower Bound.

Applications of Maximum Likelihood Estimation

MLE is applied across diverse fields, from natural sciences to engineering, to model complex data structures and make informed decisions. In econometrics, MLE is used to estimate parameters in models such as the linear regression, where the goal is to minimize the sum of squared residuals.