Overview - Outlier Tests

Introduction

Outliers are extreme values that stand out from the other values of a sample. Outliers normally have a considerable influence on the calculation of statistics (see e.g. the leverage effect with linear regression) and should be removed in most cases. You should also note that outliers may result simply from the fact that you assume a distribution which does not fit the real distribution of the data.

Typical examples of outliers are errors in measurement, errors in acquisition (human influences...), or (rare) outstanding values. An important question concerning outliers is whether it is legitimate to remove a particular value after it has been recognized as an outlier. Of course, statistical tests cannot decide, if it is appropriate to remove such values. They can only give you a hint if a significant deviation exists (basically, outlier tests are based on the probabilities to belong to the assumed distribution).

Tests on outliers in data sets can be used to

Check if methods of measurement are reliable
Check the reliability of data sets

Several outlier tests are available, each of them having its own special advantages and drawbacks. Following is a short description of the most commonly used strategies to identify outliers:

Basic rules using the population standard deviation (S.D.) and the interquartile range
Grubbs test for outliers
Generalized Extreme Studentized Deviate (ESD) outlier test
Dean-Dixon Test
Significance of extreme values (Pearson-Hartley)
Nalimov test
Outlier test according to Walsh (for not normally distributed data, not provided in SampleSTAT)

Nonetheless there is one common rule which should be strictly obeyed: Do not eliminate outliers successively. To be specific, do not identify an outlier, eliminate it and recalculate the statistics to perform another outlier detection and elimination:

All outliers must be removed in one single calculation.

The outlier tests of SampleSTAT are applicable for normally distributed data ONLY.

Test recommendations

Grubbs (ST_grubbs):

Standard test for small samples sizes (<30) and just one outlier expected which is not a problem for small sample sizes. An iterative version of Grubbs (ST_grubbs_iterat) is also available. It identifies more than one outlier per data record. This iterated version is sometimes used in labs, but it is not recommended as the statistical confidence deteriorates with each iteration step.

Generalized Extreme Studentized Deviate - ESD (ST_esd):

The Generalized Extreme Studentized Deviate (ESD) test for outliers is eminently suitable for medium to big sample sizes (>25) and also widely used and accepted. It can detect more than one outlier per data set.

Dean-Dixon (ST_deandixon):

Best for small samples sizes (less than 30). It can just consider one value on each side of the sorted data an outlier. Grubbs is more common for that sample size.

Pearson-Hartley's Significance of Extreme Values (ST_personhardley):

For larger sample sizes (more than 30). The Generalized Extreme Studentized Deviate (ESD) is the standard test for bigger sample sizes today.

Nalimov (ST_nalimov):

For small and larger sample sizes. It has a very strict algorithm and is controversially discussed. It was very common in Germany and Eastern Europe. Use it with care.

Basic rules (ST_outlier):

Up from medium sample sizes (larger than 10, better more than 25).

Threshold are based on population standard deviation ("sd" mode) or interquantile range distance ("iqr15", "iqr30" modes). The iqr-options are robust against skewed data (non-normally distributed data).

For SMALL sample sizes - less than 10 or 30 - where only one outlier is expected, the classic Grubbs test is recommended. For LARGER data sets (>25) and also when more than one outlier is expected, the ESD test is appropriate.

In QUALITY CONTROL or LAB ANALYSIS, the Nalimov test is often the standard. But use it with care.

The Dean-Dixon and Pearson-Hartley tests are used less frequently today and have been largely replaced by the Grubbs and ESD tests.

The basic outlier test rounds out the portfolio and is only practical for use with medium to large sample sizes. While the option using the standard deviation "sd" needs normally distributed data, the option using the quantile "iqr15" and "iqr30" are more robust against non-normally distruted data like skewed data. If you are not sure if your data is normally distributed and has more than 25 values ST_outlier with one of the iqr-Options can be a quick solution.

Statistical Fundamentals

$\begin{eqnarray} \bar{x} &:& \text{arithmetic mean} \quad \Rightarrow \quad \bar{x} = {1 \over{n}}\sum_{i=1}^{n}x_i \\ s &:& \text{sample standard deviation} \quad \Rightarrow \quad s = \sqrt{{1 \over{n-1}}\sum_{i=1}^{n}(x_i-\bar{x})^2} \\ \sigma &:& \text{population standard deviation, S.D.} \quad \Rightarrow \quad \sigma = \sqrt{{1 \over{n}}\sum_{i=1}^{n}(x_i-\bar{x})^2} \end{eqnarray}$

Grubbs (ST_grubbs)

The Grubbs test, also known as the maximum normed residual test, is a classical statistical procedure for detecting a single outlier in a univariate data set that is assumed to follow an approximately normal distribution. It was originally developed by Frank E. Grubbs, whose early work on outlier detection was published in 1950 and later refined in a widely cited 1969 paper. The test examines the observation with the largest absolute deviation from the sample mean relative to the sample standard deviation. It can be performed as a two-sided test for an unusually high or low value, or as a one-sided test when only one direction is of interest.

The classical Grubbs test is designed to detect one outlier at a time. An iterative version can be implemented by removing a detected outlier, recalculating the mean and standard deviation, and repeating the test on the remaining observations. However, this sequential approach should be interpreted with caution: repeatedly applying the standard Grubbs test without adjusting the critical limits changes the overall Type I error rate and does not provide the same statistical guarantees as a dedicated multiple-outlier procedure. In addition, multiple outliers may mask each other, causing the first Grubbs test to fail even when several outliers are present. For this reason, the iterative implementation should include an appropriate warning, and the Generalized ESD test is generally preferable when multiple outliers are suspected because it explicitly adjusts the sequential critical values for that purpose.

The application of the test is quite simple and straightforward: one searches the maximum of the absolute differences between the values xi and the mean. The result is divided by the standard deviation of the sample. If the resulting test statistic G is greater than the critical value, the corresponding value can be regarded to be an outlier. An extract of the critical values is shown in the following table:

$\begin{eqnarray} G_{two-sided} &=& \frac{\max_i |x_i-\overline{x}|}{s} \\ G_{left} &=& \frac{\overline{x}-x_{min}}{s} \\ G_{right} &=& \frac{x_{max}-\overline{x}}{s} \end{eqnarray}$

Critical value:

$\begin{eqnarray} G_{crit} &=& \frac{n-1}{\sqrt{n}} \sqrt{\frac{t^2}{n-2+t^2}} \\ t &=& t_{1-\alpha/(2n),\,n-2}\quad\text{Student t quantile for a two-sided test} \\ t &=& t_{1-\alpha/n,\,n-2}\quad\text{Student t quantile for a one-sided test} \\ \text{with} \\ x_i &:& \text{test value} \\ n &:& \text{number of values} \\ s &:& \text{sample standard deviation} \\ \bar{x} &:& \text{arithmetic mean} \\ x_{max} &:& \text{max. value} \\ x_{min} &:& \text{min value} \\ \alpha &:& \text{statistical confidence level} \end{eqnarray}$

Only one outlier can be found in a data set.

Generalized Extreme Studentized Deviate - ESD (ST_esd)

The Generalized Extreme Studentized Deviate (ESD) test, commonly known as the Rosner test, was introduced by Bernard Rosner in 1983 as a method for detecting multiple outliers in approximately normally distributed univariate data. It was developed as an improvement over earlier single-outlier procedures such as the Grubbs test and over the original ESD procedure. Its main purpose is to determine whether a data set contains one or more outliers while controlling the overall Type I error rate.

A key feature of the Generalized ESD test is that the exact number of suspected outliers does not have to be known in advance. Instead, the user specifies only an upper limit, and the test sequentially examines the most extreme observations while recalculating the test statistic and appropriately adjusting the critical values. This makes it more suitable for detecting multiple outliers than simply applying the Grubbs test repeatedly. Rosner's simulations indicated that the approximation performs particularly well for sample sizes of about 25 or more, although it can also be reasonably accurate for samples of around 15 observations. The test assumes approximately normally distributed data and should ideally be combined with graphical diagnostics, such as a normal probability plot or boxplot.

$\begin{eqnarray} n_i &=& n-i+1 \\ R_i &=& \frac{\max_j |x_j-\overline{x}_i|}{s_i} \quad\text{for a two-sided test} \\ R_i &=& \frac{\overline{x}_i-x_{min,i}}{s_i} \quad\text{for a left-sided test} \\ R_i &=& \frac{x_{max,i}-\overline{x}_i}{s_i} \quad\text{for a right-sided test} \end{eqnarray}$

Critical value at step i

$\begin{eqnarray} \lambda_i &=& \frac{(n_i-1)t_i}{\sqrt{(n_i-2+t_i^2)n_i}} \\ t_i &=& t_{1-\alpha/(2n_i),\,n_i-2} \quad \text{for a two-sided test} \\ t_i &=& t_{1-\alpha/n_i,\,n_i-2} \quad \text{for a one-sided test} \\ \text{with} \\ x_i &:& \text{test value} \\ n &:& \text{number of values} \\ s &:& \text{sample standard deviation} \\ \bar{x} &:& \text{arithmetic mean} \\ x_{max} &:& \text{max. value} \\ x_{min} &:& \text{min value} \\ t &:& \text{student factor} \\ \alpha &:& \text{statistical confidence level} \end{eqnarray}$

More than one outlier can be found in a data set.

Nalimov (ST_nalimov)

Nalimov is applicable for data sets between 3 and 1002 values. It calculates for all values the test value "q". It compares these q-values with the appropriate qcrit value from a table (Kaiser/Gottschalk 1972).

$\begin{eqnarray} q = \left | \frac{1}{s}(x_i- \bar{x}) \right | \sqrt{\frac{n}{n-1}} \quad;\quad > q_{crit}\;\Rightarrow \; x_i=\text{outlier} \\ x_i: \text{test value} \quad ; \quad \bar{x}: \text{arithmetic mean} \\ s: \text{sample standard deviation} \quad ; \quad n: \text{number of values} \end{eqnarray}$

The Nalimov test was widely used in quality cotrol and laboratory analysis in Germany and German-speaking countries as well as in Eastern Europe. But it has it flaws. Especially the version implemented in sampleSTAT from Kaiser/Gottschalk 1972. It indicates outliers very strict. That could eliminates valid data in some circumstances. There is some scientific discussion about this test. It is thus recommended to use other tests instead of the Nalimov test (Grubbs test for small samples, the ESD test for larger ones).

In contrast to the classical sequential elimination procedure, this implementation may therefore identify more than one value as an outlier in a single call.

If you want to use Nalimov, use it with care.

Dean-Dixon (ST_deandixon)

A test for outliers of normally distributed data which is particularly simple to apply has been developed by J.W. Dixon. This test is eminently suitable for small sample sizes (less than 30 samples); for samples having more than 30 observations the test for significance of Pearson and Hartley can be used as well.

It sorts the distribution in ascending or descending order, then takes the minimum and maximum values (xi) and calculates the respective Q value for both xi values. This is compared with the critical value from a table (Qcrit). If one of the two or both Q values greater than the corresponding Qcrit value, one or both xi values are outliers.

$\begin{eqnarray} Q = \left | x_{i+1}-x_i \right |/\left | x_n-x_i \right | \quad ; \quad Q > Q_{crit} \quad \Rightarrow \quad x_i = \text{outlier} \end{eqnarray}$

Only one outlier can be found on each side of the sorted distribution.

Pearson-Hartley's Significance of Extreme Values (ST_personhardley)

For random samples larger than 30 objects possible outliers may be identified by using the significance thresholds of Pearson and Hartley.

xi is regarded to be an outlier if the test statistic q exceeds the critical threshold qcrit for a given level of significance alpha (statistical confidence level) and a sample size n.

$\begin{eqnarray} q=\left| \frac{1}{s}(x_i - \bar{x}) \right| \quad ; \quad q > q_{crit} \quad \Rightarrow \quad x_i = \text{outlier} \\ x_i: \text{test value} \quad ; \quad \bar{x}: \text{arithmetic mean} \quad ; \quad s: \text{sample standard deviation} \end{eqnarray}$

Basic rules (ST_outlier)

Provide two basic tests on outliers based on population standard deviation (S.D.) or inter-quartile range (IQR). It should be applied to sample sizes larger than 10 better more than 25. For small sample sizes Grubbs or the Dean-Dixon test is recommended.

Based on Standard deviation:

If we assume a normal distribution, a single value may be considered as an outlier if it falls outside a certain range of the population standard deviation. In many cases a factor of 2.5 is used, which means that approx. 99 % of the data belonging to a normal distribution fall inside this range.

$\begin{eqnarray} (\bar{x} - 2.5\sigma) > x_i > (\bar{x} + 2.5\sigma) \; \text{with} \quad \sigma = \sqrt{{1 \over n}\sum_{i=1}^{n}(x_i-\bar{x})^2} \quad \Rightarrow \quad x_i = \text{outlier}\\ x_i: \text{value} \quad ; \quad \bar{x}: \text{arithmetic mean} \\ \sigma: \text{population standard deviation} \quad ; \quad n: \text{number of values} \end{eqnarray}$

Based on he interquartile range (IQR)

This approach is used for skewed data in the first place but is - of course - applicable for normally distributed data as well. It allows to distinguish between weak and strong outliers. It is the only outlier test which is applicable to non-normally distributed data.

ST_outlier offers in both iqr-options two different quantile interpolations: The Hydman-Fan type 7, which is widley used e.g. in the statistic software R and the Hyndman-Fan type 8 which is distribution independent and median-unbiased. So, it is more robust against non-normally distributed data. Therefore type 8 is the default. Type 8 is implemented for better comparison to results from statistic software, like R.

x0.25 is the first quartile and x0.75 the third and IQR is the difference.

$\begin{eqnarray} IQR = x_{0.75} - x_{0.25} \\ (x_{0.25} - 1.5 \cdot IQR) < x_i < (x_{0.25} + 1.5 \cdot IQR) \quad \Rightarrow \quad x_i = \text{outliers (iqr15 mode)} \\ (x_{0.25} - 3.0 \cdot IQR) < x_i < (x_{0.25} + 3.0 \cdot IQR) \quad \Rightarrow \quad x_i = \text{strong outlier (iqr30 mode)} \end{eqnarray}$

Values between the thresholds 3.0xIQR and 1.5xIQR are called weak outliers. SampleSTAT provides the modes "iqr15" (1.5xIQR) and "iqr30" (3.0xIQR) for testing on weak or strong outliers.

Please note that these basic tests require at least 10 observations (better 25, or more).

Authors

Hani A. Ibrahim - hani.ibrahim@gmx.de

Bibliography

R. Kaiser, G. Gottschalk; "Elementare Tests zur Beurteilung von Meßdaten", BI Hochschultaschenbücher, Bd. 774, Mannheim 1972.

Lohringer, H., "Grundlagen der Statistik", Oct, 10th, 2012, http://www.statistics4u.info/

Shapiro, Wilk: "An Analysis of Variance Test for Normality", Biometrika, Vol. 52, No. 3/4. (Dec., 1965), pp. 591-611.

Grubbs, F. E. (1950). Sample criteria for testing outlying observations. Annals of Mathematical Statistics, 21(1), 27-58.

Grubbs, F. E. (1969). Procedures for detecting outlying observations in samples. Technometrics, 11(1), 1-21.

NIST/SEMATECH e-Handbook of Statistical Methods.

Rosner, B. (1983). Percentage Points for a Generalized ESD Many-Outlier Procedure. Technometrics, 25(2), 165-172.

Report an issue
<< Outlier-Tests	Outlier-Tests	ST_deandixon >>