Measure of the influence of a data point in regression analysis
In statistics, Cook's distance or Cook's D is a commonly used estimate of the influence of a data point when performing a least-squares regression analysis.[1] In a practical ordinary least squares analysis, Cook's distance can be used in several ways: to indicate influential data points that are particularly worth checking for validity; or to indicate regions of the design space where it would be good to be able to obtain more data points. It is named after the American statistician R. Dennis Cook, who introduced the concept in 1977.[2][3]
Definition
Data points with large residuals (outliers) and/or high leverage may distort the outcome and accuracy of a regression. Cook's distance measures the effect of deleting a given observation. Points with a large Cook's distance are considered to merit closer examination in the analysis.
For the algebraic expression, first define
![{\displaystyle {\underset {n\times 1}{\mathbf {y} }}={\underset {n\times p}{\mathbf {X} }}\quad {\underset {p\times 1}{\boldsymbol {\beta }}}\quad +\quad {\underset {n\times 1}{\boldsymbol {\varepsilon }}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2e76835cb891f972fcb8a587bad8a4e75bdbc2b0)
where
is the error term,
is the coefficient matrix,
is the number of covariates or predictors for each observation, and
is the design matrix including a constant. The least squares estimator then is
, and consequently the fitted (predicted) values for the mean of
are
![{\displaystyle \mathbf {\widehat {y}} =\mathbf {X} \mathbf {b} =\mathbf {X} \left(\mathbf {X} ^{\mathsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {y} =\mathbf {H} \mathbf {y} }](https://wikimedia.org/api/rest_v1/media/math/render/svg/f042a9503d85a48dea4a730c515bf3c8ccd774f5)
where
is the projection matrix (or hat matrix). The
-th diagonal element of
, given by
,[4] is known as the leverage of the
-th observation. Similarly, the
-th element of the residual vector
is denoted by
.
Cook's distance
of observation
is defined as the sum of all the changes in the regression model when observation
is removed from it[5]
![{\displaystyle D_{i}={\frac {\sum _{j=1}^{n}\left({\widehat {y\,}}_{j}-{\widehat {y\,}}_{j(i)}\right)^{2}}{ps^{2}}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c6ee0a7e775082373eefada5e6dc60110ba1a4e1)
where p is the rank of the model (i.e., number of independent variables in the design matrix) and
is the fitted response value obtained when excluding
, and
is the mean squared error of the regression model.[6]
Equivalently, it can be expressed using the leverage[5] (
):
![{\displaystyle D_{i}={\frac {e_{i}^{2}}{ps^{2}}}\left[{\frac {h_{ii}}{(1-h_{ii})^{2}}}\right].}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0e1940d0d869a2ec32a4628045c36feca472dbe6)
Detecting highly influential observations
There are different opinions regarding what cut-off values to use for spotting highly influential points. Since Cook's distance is in the metric of an F distribution with
and
(as defined for the design matrix
above) degrees of freedom, the median point (i.e.,
) can be used as a cut-off.[7] Since this value is close to 1 for large
, a simple operational guideline of
has been suggested.[8]
The
-dimensional random vector
, which is the change of
due to a deletion of the
-th observation, has a covariance matrix of rank one and therefore it is distributed entirely over one dimensional subspace (a line, say
) of the
-dimensional space. The distributional property of
mentioned above implies that information about the influence of the
-th observation provided by
should be obtained not from outside of the line
but from the line
itself. However, in the introduction of Cook’s distance, a scaling matrix of full rank
is chosen and as a result
is treated as if it is a random vector distributed over the whole space of
dimensions. This means that information about the influence of the
-th observation provided by
through the Cook’s distance comes from the whole space of
dimensions. Hence the Cook's distance measure is likely to distort the real influence of observations, misleading the right identification of influential observations.[9][10]
Relationship to other influence measures (and interpretation)
can be expressed using the leverage[5] (
) and the square of the internally Studentized residual (
), as follows:
![{\displaystyle {\begin{aligned}D_{i}&={\frac {e_{i}^{2}}{ps^{2}}}\cdot {\frac {h_{ii}}{(1-h_{ii})^{2}}}={\frac {1}{p}}\cdot {\frac {e_{i}^{2}}{{1 \over n-p}\sum _{j=1}^{n}{\widehat {\varepsilon \,}}_{j}^{\,2}(1-h_{ii})}}\cdot {\frac {h_{ii}}{1-h_{ii}}}\\[5pt]&={\frac {1}{p}}\cdot t_{i}^{2}\cdot {\frac {h_{ii}}{1-h_{ii}}}.\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d3cd49482b87a4c146dc0cf6031d6fad8faf6b1c)
The benefit in the last formulation is that it clearly shows the relationship between
and
to
(while p and n are the same for all observations). If
is large then it (for non-extreme values of
) will increase
. If
is close to 0 then
will be small, while if
is close to 1 then
will become very large (as long as
, i.e.: that the observation
is not exactly on the regression line that was fitted without observation
).
is related to DFFITS through the following relationship (note that
is the externally studentized residual, and
are defined here):
![{\displaystyle {\begin{aligned}D_{i}&={\frac {1}{p}}\cdot t_{i}^{2}\cdot {\frac {h_{ii}}{1-h_{ii}}}\\&={\frac {1}{p}}\cdot {\frac {{\widehat {\sigma }}_{(i)}^{2}}{{\widehat {\sigma }}^{2}}}\cdot {\frac {{\widehat {\sigma }}^{2}}{{\widehat {\sigma }}_{(i)}^{2}}}\cdot t_{i}^{2}\cdot {\frac {h_{ii}}{1-h_{ii}}}={\frac {1}{p}}\cdot {\frac {{\widehat {\sigma }}_{(i)}^{2}}{{\widehat {\sigma }}^{2}}}\cdot \left(t_{i(i)}{\sqrt {\frac {h_{ii}}{1-h_{ii}}}}\right)^{2}\\&={\frac {1}{p}}\cdot {\frac {{\widehat {\sigma }}_{(i)}^{2}}{{\widehat {\sigma }}^{2}}}\cdot {\text{DFFITS}}^{2}\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8315da28cfe1ec00baffe74c6a6089b173e0b311)
can be interpreted as the distance one's estimates move within the confidence ellipsoid that represents a region of plausible values for the parameters.[clarification needed] This is shown by an alternative but equivalent representation of Cook's distance in terms of changes to the estimates of the regression parameters between the cases, where the particular observation is either included or excluded from the regression analysis.
An alternative to
has been proposed. Instead of considering the influence a single observation has on the overall model, the statistics
serves as a measure of how sensitive the prediction of the
-th observation is to the deletion of each observation in the original data set. It can be formulated as a weighted linear combination of the
's of all data points. Again, the projection matrix is involved in the calculation to obtain the required weights:
![{\displaystyle S_{i}={\frac {\sum _{j=1}^{n}\left({\widehat {y}}_{i}-{{\widehat {y}}_{i}}_{(j)}\right)^{2}}{ps^{2}h_{ii}}}=\sum _{j=1}^{n}{\frac {h_{ij}^{2}\cdot D_{j}}{h_{ii}\cdot h_{jj}}}=\sum _{j=1}^{n}\rho _{ij}^{2}\cdot D_{j}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3dfcbe1e3114fdc797a736f9d057ba81c44aee17)
In this context,
(
) resembles the correlation between the predictions
and
[a].
In contrast to
, the distribution of
is asymptotically normal for large sample sizes and models with many predictors. In absence of outliers the expected value of
is approximately
. An influential observation can be identified if
![{\displaystyle \left|S_{i}-\operatorname {med} (S)\right|\geq 4.5\cdot \operatorname {MAD} (S)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dafe479b0e767c47cfd5fdc20dc838738f153172)
with
as the median and
as the median absolute deviation of all
-values within the original data set, i.e., a robust measure of location and a robust measure of scale for the distribution of
. The factor 4.5 covers approx. 3 standard deviations of
around its centre.
When compared to Cook's distance,
was found to perform well for high- and intermediate-leverage outliers, even in presence of masking effects for which
failed.[12]
Interestingly,
and
are closely related because they can both be expressed in terms of the matrix
which holds the effects of the deletion of the
-th data point on the
-th prediction:
![{\displaystyle {\begin{aligned}&\mathbf {T} =\left[{\begin{matrix}{\widehat {y}}_{1}-{{\widehat {y}}_{1}}_{\left(1\right)}&{\widehat {y}}_{1}-{{\widehat {y}}_{1}}_{\left(2\right)}&{\widehat {y}}_{1}-{{\widehat {y}}_{1}}_{\left(3\right)}&\cdots &{\widehat {y}}_{1}-{{\widehat {y}}_{1}}_{\left(n-1\right)}&{\widehat {y}}_{1}-{{\widehat {y}}_{1}}_{\left(n\right)}\\{\widehat {y}}_{2}-{{\widehat {y}}_{2}}_{\left(1\right)}&{\widehat {y}}_{2}-{{\widehat {y}}_{2}}_{\left(2\right)}&{\widehat {y}}_{2}-{{\widehat {y}}_{2}}_{\left(3\right)}&\cdots &{\widehat {y}}_{2}-{{\widehat {y}}_{2}}_{\left(n-1\right)}&{\widehat {y}}_{2}-{{\widehat {y}}_{2}}_{\left(n\right)}\\\vdots &\vdots &\vdots &\ddots &\vdots &\vdots \\{\widehat {y}}_{n-1}-{{\widehat {y}}_{n-1}}_{\left(1\right)}&{\widehat {y}}_{n-1}-{{\widehat {y}}_{n-1}}_{\left(2\right)}&{\widehat {y}}_{n-1}-{{\widehat {y}}_{n-1}}_{\left(3\right)}&\cdots &{\widehat {y}}_{n-1}-{{\widehat {y}}_{n-1}}_{\left(n-1\right)}&{\widehat {y}}_{n-1}-{{\widehat {y}}_{n-1}}_{\left(n\right)}\\{\widehat {y}}_{n}-{{\widehat {y}}_{n}}_{\left(1\right)}&{\widehat {y}}_{n}-{{\widehat {y}}_{n}}_{\left(2\right)}&{\widehat {y}}_{n}-{{\widehat {y}}_{n}}_{\left(3\right)}&\cdots &{\widehat {y}}_{n}-{{\widehat {y}}_{n}}_{\left(n-1\right)}&{\widehat {y}}_{n}-{{\widehat {y}}_{n}}_{\left(n\right)}\end{matrix}}\right]\\\\&\ \ =\mathbf {H} \mathbf {E} \mathbf {G} =\mathbf {H} \left[{\begin{matrix}e_{1}&0&0&\cdots &0&0\\0&e_{2}&0&\cdots &0&0\\\vdots &\vdots &\vdots &\ddots &\vdots &\vdots \\0&0&0&\cdots &e_{n-1}&0\\0&0&0&\cdots &0&e_{n}\end{matrix}}\right]\left[{\begin{matrix}{\frac {1}{1-h_{11}}}&0&0&\cdots &0&0\\0&{\frac {1}{1-h_{22}}}&0&\cdots &0&0\\\vdots &\vdots &\vdots &\ddots &\vdots &\vdots \\0&0&0&\cdots &{\frac {1}{1-h_{n-1,n-1}}}&0\\0&0&0&\cdots &0&{\frac {1}{1-h_{nn}}}\end{matrix}}\right]\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b37a8d61763a401e6a219554356b2559268215a2)
With
at hand,
is given by:
![{\displaystyle \mathbf {D} =\left[{\begin{matrix}D_{1}\\D_{2}\\\vdots \\D_{n-1}\\D_{n}\end{matrix}}\right]={\frac {1}{ps^{2}}}\operatorname {diag} \left(\mathbf {T} ^{\mathsf {T}}\mathbf {T} \right)={\frac {1}{ps^{2}}}\operatorname {diag} \left(\mathbf {G} \mathbf {E} \mathbf {H} ^{\mathsf {T}}\mathbf {H} \mathbf {E} \mathbf {G} \right)=\operatorname {diag} (\mathbf {M} )}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ff183354e5e3b140be61cb57a5ee0dd0cd91007b)
where
if
is symmetric and idempotent, which is not necessarily the case. In contrast,
can be calculated as:
![{\displaystyle {\begin{aligned}&\mathbf {S} =\left[{\begin{matrix}S_{1}\\S_{2}\\\vdots \\S_{n-1}\\S_{n}\end{matrix}}\right]={\frac {1}{ps^{2}}}\mathbf {F} \operatorname {diag} \left(\mathbf {T} \mathbf {T} ^{\mathsf {T}}\right)={\frac {1}{ps^{2}}}\left[{\begin{matrix}{\frac {1}{h_{11}}}&0&0&\cdots &0&0\\0&{\frac {1}{h_{22}}}&0&\cdots &0&0\\\vdots &\vdots &\vdots &\ddots &\vdots &\vdots \\0&0&0&\cdots &{\frac {1}{h_{n-1n-1}}}&0\\0&0&0&\cdots &0&{\frac {1}{h_{nn}}}\end{matrix}}\right]\operatorname {diag} \left(\mathbf {T} \mathbf {T} ^{\mathsf {T}}\right)\\\\&\ \ ={\frac {1}{ps^{2}}}\mathbf {F} \operatorname {diag} \left(\mathbf {H} \mathbf {E} \mathbf {G} \mathbf {G} \mathbf {E} \mathbf {H} ^{\mathsf {T}}\right)=\mathbf {F} \operatorname {diag} (\mathbf {P} )\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/db58307889e9df073493850073316089e68196d4)
where
extracts the main diagonal of a square matrix
. In this context,
is referred to as the influence matrix whereas
resembles the so-called sensitivity matrix. An eigenvector analysis of
and
- which both share the same eigenvalues – serves as a tool in outlier detection, although the eigenvectors of the sensitivity matrix are more powerful. [13]
Software implementations
Many programs and statistics packages, such as R, Python, Julia, etc., include implementations of Cook's distance.
Language/Program | Function | Notes |
Stata | predict, cooksd | See [1] |
R | cooks.distance(model, ...) | See [2] |
Python | CooksDistance().fit(X, y) | See [3] |
Julia | cooksdistance(model, ...) | See [4] |
Extensions
High-dimensional Influence Measure (HIM) is an alternative to Cook's distance for when
(i.e., when there are more predictors than observations).[14] While the Cook's distance quantifies the individual observation's influence on the least squares regression coefficient estimate, the HIM measures the influence of an observation on the marginal correlations.
See also
Notes
- ^ The indices
and
are often interchanged in the original publication as the projection matrix
is symmetric in ordinary linear regression, i.e.,
. Since this is not always the case, e.g., in weighted linear regression, the indices have been written consistently here to account for potential asymmetry and thus allow for direct usage.
References
- ^ Mendenhall, William; Sincich, Terry (1996). A Second Course in Statistics: Regression Analysis (5th ed.). Upper Saddle River, NJ: Prentice-Hall. p. 422. ISBN 0-13-396821-9.
A measure of overall influence an outlying observation has on the estimated
coefficients was proposed by R. D. Cook (1979). Cook's distance, Di, is calculated...
- ^ Cook, R. Dennis (February 1977). "Detection of Influential Observations in Linear Regression". Technometrics. 19 (1). American Statistical Association: 15–18. doi:10.2307/1268249. JSTOR 1268249. MR 0436478.
- ^ Cook, R. Dennis (March 1979). "Influential Observations in Linear Regression". Journal of the American Statistical Association. 74 (365). American Statistical Association: 169–174. doi:10.2307/2286747. hdl:11299/199280. JSTOR 2286747. MR 0529533.
- ^ Hayashi, Fumio (2000). Econometrics. Princeton University Press. pp. 21–23. ISBN 1400823838.
- ^ a b c "Cook's Distance".
- ^ "Statistics 512: Applied Linear Models" (PDF). Purdue University. Archived from the original (PDF) on 2016-11-30. Retrieved 2016-03-25.
- ^ Bollen, Kenneth A.; Jackman, Robert W. (1990). "Regression Diagnostics: An Expository Treatment of Outliers and Influential Cases". In Fox, John; Long, J. Scott (eds.). Modern Methods of Data Analysis. Newbury Park, CA: Sage. pp. 266. ISBN 0-8039-3366-5.
- ^ Cook, R. Dennis; Weisberg, Sanford (1982). Residuals and Influence in Regression. New York, NY: Chapman & Hall. hdl:11299/37076. ISBN 0-412-24280-X.
- ^ Kim, Myung Geun (31 May 2017). "A cautionary note on the use of Cook's distance". Communications for Statistical Applications and Methods. 24 (3): 317–324. doi:10.5351/csam.2017.24.3.317. ISSN 2383-4757.
- ^ On deletion diagnostic statistic in regression
- ^ Peña, Daniel (2005). "A New Statistic for Influence in Linear Regression". Technometrics. 47 (1). American Society for Quality and the American Statistical Association: 1–12. doi:10.1198/004017004000000662. S2CID 1802937.
- ^ Peña, Daniel (2006). Pham, Hoang (ed.). Springer Handbook of Engineering Statistics. Springer London. pp. 523–536. doi:10.1007/978-1-84628-288-1. ISBN 978-1-84628-288-1. S2CID 60460007.
- ^ High-dimensional influence measure
Further reading
- Atkinson, Anthony; Riani, Marco (2000). "Deletion Diagnostics". Robust Diagnostics and Regression Analysis. New York: Springer. pp. 22–25. ISBN 0-387-95017-6.
- Heiberger, Richard M.; Holland, Burt (2013). "Case Statistics". Statistical Analysis and Data Display. Springer Science & Business Media. pp. 312–27. ISBN 9781475742848.
- Krasker, William S.; Kuh, Edwin; Welsch, Roy E. (1983). "Estimation for dirty data and flawed models". Handbook of Econometrics. Vol. 1. Elsevier. pp. 651–698. doi:10.1016/S1573-4412(83)01015-6. ISBN 9780444861856.
- Aguinis, Herman; Gottfredson, Ryan K.; Joo, Harry (2013). "Best-Practice Recommendations for Defining Identifying and Handling Outliers". Organizational Research Methods. 16 (2). Sage: 270–301. doi:10.1177/1094428112470848. S2CID 54916947. Retrieved 4 December 2015.