Probability distribution
In statistics, the inverse Wishart distribution, also called the inverted Wishart distribution, is a probability distribution defined on real-valued positive-definite matrices. In Bayesian statistics it is used as the conjugate prior for the covariance matrix of a multivariate normal distribution.
We say
follows an inverse Wishart distribution, denoted as
, if its inverse
has a Wishart distribution
. Important identities have been derived for the inverse-Wishart distribution.[2]
Density
The probability density function of the inverse Wishart is:[3]
![{\displaystyle f_{\mathbf {X} }({\mathbf {X} };{\mathbf {\Psi } },\nu )={\frac {\left|{\mathbf {\Psi } }\right|^{\nu /2}}{2^{\nu p/2}\Gamma _{p}({\frac {\nu }{2}})}}\left|\mathbf {X} \right|^{-(\nu +p+1)/2}e^{-{\frac {1}{2}}\operatorname {tr} (\mathbf {\Psi } \mathbf {X} ^{-1})}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0e27654b7f744b5f318d02962515551c7f0f785b)
where
and
are
positive definite matrices,
is the determinant, and Γp(·) is the multivariate gamma function.
Theorems
Distribution of the inverse of a Wishart-distributed matrix
If
and
is of size
, then
has an inverse Wishart distribution
.[4]
Marginal and conditional distributions from an inverse Wishart-distributed matrix
Suppose
has an inverse Wishart distribution. Partition the matrices
and
conformably with each other
![{\displaystyle {\mathbf {A} }={\begin{bmatrix}\mathbf {A} _{11}&\mathbf {A} _{12}\\\mathbf {A} _{21}&\mathbf {A} _{22}\end{bmatrix}},\;{\mathbf {\Psi } }={\begin{bmatrix}\mathbf {\Psi } _{11}&\mathbf {\Psi } _{12}\\\mathbf {\Psi } _{21}&\mathbf {\Psi } _{22}\end{bmatrix}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/bfad0b50e2063e19095052409d3df324b8dcb8db)
where
and
are
matrices, then we have
is independent of
and
, where
is the Schur complement of
in
;
;
, where
is a matrix normal distribution;
, where
;
Conjugate distribution
Suppose we wish to make inference about a covariance matrix
whose prior
has a
distribution. If the observations
are independent p-variate Gaussian variables drawn from a
distribution, then the conditional distribution
has a
distribution, where
.
Because the prior and posterior distributions are the same family, we say the inverse Wishart distribution is conjugate to the multivariate Gaussian.
Due to its conjugacy to the multivariate Gaussian, it is possible to marginalize out (integrate out) the Gaussian's parameter
, using the formula
and the linear algebra identity
:
![{\displaystyle f_{\mathbf {X} \,\mid \,\Psi ,\nu }(\mathbf {x} )=\int f_{\mathbf {X} \,\mid \,\mathbf {\Sigma } \,=\,\sigma }(\mathbf {x} )f_{\mathbf {\Sigma } \,\mid \,\mathbf {\Psi } ,\nu }(\sigma )\,d\sigma ={\frac {|\mathbf {\Psi } |^{\nu /2}\Gamma _{p}\left({\frac {\nu +n}{2}}\right)}{\pi ^{np/2}|\mathbf {\Psi } +\mathbf {A} |^{(\nu +n)/2}\Gamma _{p}({\frac {\nu }{2}})}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/86cc1f7be1df2265cd133a473e163c8a8ac5302a)
(this is useful because the variance matrix
is not known in practice, but because
is known a priori, and
can be obtained from the data, the right hand side can be evaluated directly). The inverse-Wishart distribution as a prior can be constructed via existing transferred prior knowledge.[5]
Moments
The following is based on Press, S. J. (1982) "Applied Multivariate Analysis", 2nd ed. (Dover Publications, New York), after reparameterizing the degree of freedom to be consistent with the p.d.f. definition above.
Let
with
and
, so that
.
The mean:[4]: 85
![{\displaystyle \operatorname {E} (\mathbf {X} )={\frac {\mathbf {\Psi } }{\nu -p-1}}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/998971401b15abb77575953aafd91e626312a37d)
The variance of each element of
:
![{\displaystyle \operatorname {Var} (x_{ij})={\frac {(\nu -p+1)\psi _{ij}^{2}+(\nu -p-1)\psi _{ii}\psi _{jj}}{(\nu -p)(\nu -p-1)^{2}(\nu -p-3)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6a27c8d9dadc0803bce3b6910416c6317091d17e)
The variance of the diagonal uses the same formula as above with
, which simplifies to:
![{\displaystyle \operatorname {Var} (x_{ii})={\frac {2\psi _{ii}^{2}}{(\nu -p-1)^{2}(\nu -p-3)}}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c74d6f32f5e26b6f5775cff5478203b018cfab77)
The covariance of elements of
are given by:
![{\displaystyle \operatorname {Cov} (x_{ij},x_{k\ell })={\frac {2\psi _{ij}\psi _{k\ell }+(\nu -p-1)(\psi _{ik}\psi _{j\ell }+\psi _{i\ell }\psi _{kj})}{(\nu -p)(\nu -p-1)^{2}(\nu -p-3)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6648e8c3fe4c530a587042ab8c8ea2d21e6e067c)
The same results are expressed in Kronecker product form by von Rosen[6] as follows:
![{\displaystyle {\begin{aligned}\mathbf {E} \left(W^{-1}\otimes W^{-1}\right)&=c_{1}\Psi \otimes \Psi +c_{2}Vec(\Psi )Vec(\Psi )^{T}+c_{2}K_{pp}\Psi \otimes \Psi \\\mathbf {Cov} _{\otimes }\left(W^{-1},W^{-1}\right)&=(c_{1}-c_{3})\Psi \otimes \Psi +c_{2}Vec(\Psi )Vec(\Psi )^{T}+c_{2}K_{pp}\Psi \otimes \Psi \end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/65c6735989c22bc4076affd807bb70bfe47e8bdf)
where
![{\displaystyle {\begin{aligned}c_{2}&=\left[(\nu -p)(\nu -p-1)(\nu -p-3)\right]^{-1}\\c_{1}&=(\nu -p-2)c_{2}\\c_{3}&=(\nu -p-1)^{-2},\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b4df0a39589293ddf4418b75da91da88b0023b06)
commutation matrix ![{\displaystyle \mathbf {Cov} _{\otimes }\left(W^{-1},W^{-1}\right)=\mathbf {E} \left(W^{-1}\otimes W^{-1}\right)-\mathbf {E} \left(W^{-1}\right)\otimes \mathbf {E} \left(W^{-1}\right).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b9fca94f9cf577c97f36dc2d5d61df0024595d67)
There appears to be a typo in the paper whereby the coefficient of
is given as
rather than
, and that the expression for the mean square inverse Wishart, corollary 3.1, should read
![{\displaystyle \mathbf {E} \left[W^{-1}W^{-1}\right]=(c_{1}+c_{2})\Sigma ^{-1}\Sigma ^{-1}+c_{2}\Sigma ^{-1}\mathbf {tr} (\Sigma ^{-1}).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f7491beb9f835af410859020c015b0b787ec30c9)
To show how the interacting terms become sparse when the covariance is diagonal, let
and introduce some arbitrary parameters
:
![{\displaystyle \mathbf {E} \left(W^{-1}\otimes W^{-1}\right)=u\Psi \otimes \Psi +v\,\mathrm {vec} (\Psi )\,\mathrm {vec} (\Psi )^{T}+wK_{pp}\Psi \otimes \Psi .}](https://wikimedia.org/api/rest_v1/media/math/render/svg/63e46a1801fbd0100a6436bb3faba368579e45e9)
where
denotes the matrix vectorization operator. Then the second moment matrix becomes
![{\displaystyle \mathbf {E} \left(W^{-1}\otimes W^{-1}\right)={\begin{bmatrix}u+v+w&\cdot &\cdot &\cdot &v&\cdot &\cdot &\cdot &v\\\cdot &u&\cdot &w&\cdot &\cdot &\cdot &\cdot &\cdot \\\cdot &\cdot &u&\cdot &\cdot &\cdot &w&\cdot &\cdot \\\cdot &w&\cdot &u&\cdot &\cdot &\cdot &\cdot &\cdot \\v&\cdot &\cdot &\cdot &u+v+w&\cdot &\cdot &\cdot &v\\\cdot &\cdot &\cdot &\cdot &\cdot &u&\cdot &w&\cdot \\\cdot &\cdot &w&\cdot &\cdot &\cdot &u&\cdot &\cdot \\\cdot &\cdot &\cdot &\cdot &\cdot &w&\cdot &u&\cdot \\v&\cdot &\cdot &\cdot &v&\cdot &\cdot &\cdot &u+v+w\\\end{bmatrix}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e6748bb5415ae87070ba7f37be9232a8601c9274)
which is non-zero only when involving the correlations of diagonal elements of
, all other elements are mutually uncorrelated, though not necessarily statistically independent. The variances of the Wishart product are also obtained by Cook et al.[7] in the singular case and, by extension, to the full rank case.
Muirhead[8] shows in Theorem 3.2.8 that if
is distributed as
and
is an arbitrary vector, independent of
then
and
, one degree of freedom being relinquished by estimation of the sample mean in the latter. Similarly, Bodnar et.al. further find that
and setting
the marginal distribution of the leading diagonal element is thus
![{\displaystyle {\frac {[A^{-1}]_{1,1}}{[\Sigma ^{-1}]_{1,1}}}\sim {\frac {2^{-k/2}}{\Gamma (k/2)}}x^{-k/2-1}e^{-1/(2x)},\;\;k=\nu -p+1}](https://wikimedia.org/api/rest_v1/media/math/render/svg/dcc4301c2da69a196a7231328b5365db6a7c355e)
and by rotating
end-around a similar result applies to all diagonal elements
.
A corresponding result in the complex Wishart case was shown by Brennan and Reed[9] and the uncorrelated inverse complex Wishart
was shown by Shaman[10] to have diagonal statistical structure in which the leading diagonal elements are correlated, while all other element are uncorrelated.
- A univariate specialization of the inverse-Wishart distribution is the inverse-gamma distribution. With
(i.e. univariate) and
,
and
the probability density function of the inverse-Wishart distribution becomes matrix
![{\displaystyle p(x\mid \alpha ,\beta )={\frac {\beta ^{\alpha }\,x^{-\alpha -1}\exp(-\beta /x)}{\Gamma _{1}(\alpha )}}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2517dd8be76024d532e4b9d0d99895f8eb6ef4a2)
- i.e., the inverse-gamma distribution, where
is the ordinary Gamma function.
- The Inverse Wishart distribution is a special case of the inverse matrix gamma distribution when the shape parameter
and the scale parameter
. - Another generalization has been termed the generalized inverse Wishart distribution,
. A
positive definite matrix
is said to be distributed as
if
is distributed as
. Here
denotes the symmetric matrix square root of
, the parameters
are
positive definite matrices, and the parameter
is a positive scalar larger than
. Note that when
is equal to an identity matrix,
. This generalized inverse Wishart distribution has been applied to estimating the distributions of multivariate autoregressive processes.[11] - A different type of generalization is the normal-inverse-Wishart distribution, essentially the product of a multivariate normal distribution with an inverse Wishart distribution.
- When the scale matrix is an identity matrix,
is an arbitrary orthogonal matrix, replacement of
by
does not change the pdf of
so
belongs to the family of spherically invariant random processes (SIRPs) in some sense.[clarification needed]
- Thus, an arbitrary p-vector
with
length
can be rotated into the vector
without changing the pdf of
, moreover
can be a permutation matrix which exchanges diagonal elements. It follows that the diagonal elements of
are identically inverse chi squared distributed, with pdf
in the previous section though they are not mutually independent. The result is known in optimal portfolio statistics, as in Theorem 2 Corollary 1 of Bodnar et al,[12] where it is expressed in the inverse form
.
- As is the case with the Wishart distribution linear transformations of the distribution yield a modified inverse Wishart distribution. If
and
are full rank matrices then[13] ![{\displaystyle \mathbf {\Theta } \mathbf {X} {\mathbf {\Theta } }^{T}\sim {\mathcal {W}}_{p}^{-1}\left({\mathbf {\Theta } }{\mathbf {\Psi } }{\mathbf {\Theta } }^{T},\nu \right).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/29b7289162d2696a8a5743515313fc00f89a24f7)
- If
and
is
of full rank
then[13] ![{\displaystyle \mathbf {\Theta } \mathbf {X} {\mathbf {\Theta } }^{T}\sim {\mathcal {W}}_{m}^{-1}\left({\mathbf {\Theta } }{\mathbf {\Psi } }{\mathbf {\Theta } }^{T},\nu \right).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/83a717098c0f95f09fc14eac5c252a5190fb0143)
See also
References
- ^ A. O'Hagan, and J. J. Forster (2004). Kendall's Advanced Theory of Statistics: Bayesian Inference. Vol. 2B (2 ed.). Arnold. ISBN 978-0-340-80752-1.
- ^ Haff, LR (1979). "An identity for the Wishart distribution with applications". Journal of Multivariate Analysis. 9 (4): 531–544. doi:10.1016/0047-259x(79)90056-3.
- ^ Gelman, Andrew; Carlin, John B.; Stern, Hal S.; Dunson, David B.; Vehtari, Aki; Rubin, Donald B. (2013-11-01). Bayesian Data Analysis, Third Edition (3rd ed.). Boca Raton: Chapman and Hall/CRC. ISBN 9781439840955.
- ^ a b Kanti V. Mardia, J. T. Kent and J. M. Bibby (1979). Multivariate Analysis. Academic Press. ISBN 978-0-12-471250-8.
- ^ Shahrokh Esfahani, Mohammad; Dougherty, Edward (2014). "Incorporation of Biological Pathway Knowledge in the Construction of Priors for Optimal Bayesian Classification". IEEE Transactions on Bioinformatics and Computational Biology. 11 (1): 202–218. doi:10.1109/tcbb.2013.143. PMID 26355519. S2CID 10096507.
- ^ Rosen, Dietrich von (1988). "Moments for the Inverted Wishart Distribution". Scand. J. Stat. 15: 97–109 – via JSTOR.
- ^ Cook, R D; Forzani, Liliana (August 2019). Cook, Brian (ed.). "On the mean and variance of the generalized inverse of a singular Wishart matrix". Electronic Journal of Statistics. 5. doi:10.4324/9780429344633. ISBN 9780429344633. S2CID 146200569.
- ^ Muirhead, Robb (1982). Aspects of Multivariate Statistical Theory. USA: Wiley. p. 93. ISBN 0-471-76985-1.
- ^ Brennan, L E; Reed, I S (January 1982). "An Adaptive Array Signal Processing Algorithm for Communications". IEEE Transactions on Aerospace and Electronic Systems. 18 (1): 120–130. Bibcode:1982ITAES..18..124B. doi:10.1109/TAES.1982.309212. S2CID 45721922.
- ^ Shaman, Paul (1980). "The Inverted Complex Wishart Distribution and Its Application to Spectral Estimation" (PDF). Journal of Multivariate Analysis. 10: 51–59. doi:10.1016/0047-259X(80)90081-0.
- ^ Triantafyllopoulos, K. (2011). "Real-time covariance estimation for the local level model". Journal of Time Series Analysis. 32 (2): 93–107. arXiv:1311.0634. doi:10.1111/j.1467-9892.2010.00686.x. S2CID 88512953.
- ^ Bodnar, T.; Mazur, S.; Podgórski, K. (January 2015). "Singular Inverse Wishart Distribution with Application to Portfolio Theory". Department of Statistics, Lund University. (Working Papers in Statistics, Nr. 2): 1–17.
- ^ a b Bodnar, T; Mazur, S; Podgorski, K (2015). "Singular Inverse Wishart Distribution with Application to Portfolio Theory". Journal of Multivariate Analysis. 143: 314–326.
Discrete univariate | with finite support | |
---|
with infinite support | |
---|
|
---|
Continuous univariate | supported on a bounded interval | |
---|
supported on a semi-infinite interval | |
---|
supported on the whole real line | |
---|
with support whose type varies | |
---|
|
---|
Mixed univariate | |
---|
Multivariate (joint) | |
---|
Directional | |
---|
Degenerate and singular | |
---|
Families | |
---|
Category Commons |