<< ST_nalimov Outlier-Tests ST_pearsonhartley >>
samplestat >> samplestat > Outlier-Tests > ST_outlier

ST_outlier

Basic outlier tests with standard deviation or quantile

Syntax

[outlierfree] = ST_outlier(v)
[outlierfree] = ST_outlier(v, mod)
[outlierfree] = ST_outlier(v, mod, q)
[outlierfree, outlier] = ST_outlier(v)
[outlierfree, outlier] = ST_outlier(v, mod)
[outlierfree, outlier] = ST_outlier(v, mod, q)

Arguments

v:

n-by-1 or 1-by-m matrix of doubles, numerical values (n>10, better n>25)

mod:

1-by-1 matrix of strings, "sd", "iqr15" or "iqr30" mode

q:

1-by-1 matrix of doubles. OPTIONAL. Quantile interpolation type for IQR modes 7 = Hyndman-Fan type 7, 8 = Hyndman-Fan type 8 (default)

outlierfree :

input vector with all detected outliers removed; unchanged if the test does not identify an outlier

outlier :

detected outliers in their original input order; [] if no outlier is detected

Description

Performs basic outlier tests.

SD-MODE: If you have a normal, symetric and unimodal distribution you can use the "sd" mode (population standard deviation, S.D. or sigma). In this mode a value is presented as an outlier when it is more than 2.5xS.D. off the arithmetic mean in both directions.

The data has to be normally distributed when using the "sd" option!

x_i < (\bar{x} - 2.5\sigma) \; \text{or} \; x_i > (\bar{x} + 2.5\sigma) \; \text{with} \quad \sigma = \sqrt{{1 \over n}\sum_{i=1}^{n}(x_i-\bar{x})^2} \quad \Rightarrow \quad x_i = \text{outlier}\\ \begin{eqnarray} x_i: \text{value} \quad &;& \quad \bar{x}: \text{arithmetic mean} \\ \sigma: \text{population standard deviation} \quad &;& \quad n: \text{number of values} \end{eqnarray}

IQR-MODES:Testing on outliers with interquartile range (IQR) distance is recommended for skewed data in the first place. But it is also applicaple for normally distributed data.

The IQR-Mode is more robust against non-normally distributed data.

IQR15-MODE: It is common to consider a value an outlier when it is more than 1.5xIQR (inter-quartile range) off from the lower or upper quartile. The "iqr15"-mode make use of this.

IQR30-MODE: But with a border of 1,5xIQR 0.7% of the distribution can be expected as an outlier automatically. This means that a distribution of 143 values or more could have at least one outlier in any case. To avoid this, values between 1.5xIQR and 3.0xIQR from the lower or upper quartile are called extreme values or weak outliers and just values outside of 3.0xIQR are strong outliers. SampleSTAT toolbox take care of this by introducing the "iqr30" mode.

For quantile interpolation two types are available. Hyndman-Fan type 7 - widely used, e.g. in the statistic software R and Hyndman-Fan type 8 - distribution independent, median-unbiased (default).

IQR = x_{0.75} - x_{0.25} \\ x_i < (x_{0.25} - 1.5 \cdot IQR) \; \text{or} \; x_i > (x_{0.75} + 1.5 \cdot IQR) \quad \Rightarrow \quad x_i = \text{outlier (iqr15 mode)} \\ x_i < (x_{0.25} - 3.0 \cdot IQR) \; \text{or} \; x_i > (x_{0.75} + 3.0 \cdot IQR) \quad \Rightarrow \quad x_i = \text{strong outlier (iqr30 mode)}

Examples

data = [
0.4827129   0.3431706  -0.4127328    0.3843994 ..
-0.7107495  -0.2547306   0.0290803    0.1386087 ..
-0.7698385   1.0743628   1.0945652    0.4365680 ..
-0.5913411  -0.7426987   1.609719     0.8079680 ..
-2.1700554  -4.7361261   0.0069708    14.626386 ..
-2.5036545  -2.9046385 ..
];
of              = ST_outlier(data')             // outlier-free values with sd-mode
[of, o]         = ST_outlier(data', "sd")       // outlier and outlier-free values
[of15, o15]     = ST_outlier(data', "iqr15")    // IQR, quantile type 8 (default)
[of15t7, o15t7] = ST_outlier(data', "iqr15", 7) // IQR, quantile type 7
[of30, o30]     = ST_outlier(data', "iqr30", 8) // IQR, quantile type 8

See also

Authors

Bibliography

Lohringer, H., "Grundlagen der Statistik", Oct, 10th, 2012, http://www.statistics4u.info/fundstat_germ/cc_outlier_tests_4sigma.html


Report an issue
<< ST_nalimov Outlier-Tests ST_pearsonhartley >>