I have a scenario where I have several thousand instances of data. The data itself is represented as a single integer value. I want to be able to detect when an instance is an extreme outlier.

我有一个场景，我有几千个数据实例。数据本身表示为单个整数值。我希望能够检测到一个实例何时是极端离群值。

For example, with the following example data:

例如，使用以下示例数据：

a = 10
b = 14
c = 25
d = 467
e = 12

d is clearly an anomaly, and I would want to perform a specific action based on this.

D显然是一个反常现象，我想在此基础上执行特定的操作。

I was tempted to just try an use my knowledge of the particular domain to detect anomalies. For instance, figure out a distance from the mean value that is useful, and check for that, based on heuristics. However, I think it's probably better if I investigate more general, robust anomaly detection techniques, which have some theory behind them.

我想试着利用我对特定领域的知识来检测异常。例如，计算出与平均值之间的有用距离，并基于启发式方法进行检查。然而，我认为如果我研究更通用、更健壮的异常检测技术，可能会更好，这些技术背后有一些理论支持。

Since my working knowledge of mathematics is limited, I'm hoping to find a technique which is simple, such as using standard deviation. Hopefully the single-dimensioned nature of the data will make this quite a common problem, but if more information for the scenario is required please leave a comment and I will give more info.

由于我的数学应用知识有限，我希望找到一种简单的方法，比如使用标准差。希望数据的单维性会使这成为一个相当常见的问题，但如果需要关于该场景的更多信息，请留下评论，我会提供更多信息。

---

Edit: thought I'd add more information about the data and what I've tried in case it makes one answer more correct than another.

编辑：我想我会添加更多关于数据和我尝试的内容的信息，以防它使一个答案比另一个更正确。

The values are all positive and non-zero. I expect that the values will form a normal distribution. This expectation is based on an intuition of the domain rather than through analysis, if this is not a bad thing to assume, please let me know. In terms of clustering, unless there's also standard algorithms to choose a k-value, I would find it hard to provide this value to a k-Means algorithm.

这些值都是正数且非零。我预计这些值将形成正态分布。这种期望是基于对领域的直觉，而不是通过分析，如果这不是一件坏事，请让我知道。在集群方面，除非也有标准的算法来选择k值，否则我会发现很难将这个值提供给k-均值算法。

The action I want to take for an outlier/anomaly is to present it to the user, and recommend that the data point is basically removed from the data set (I won't get in to how they would do that, but it makes sense for my domain), thus it will not be used as input to another function.

我想对离群值/异常采取的操作是将其呈现给用户，并建议基本上从数据集中删除数据点(我不会深入了解他们将如何做到这一点，但这对我的域是有意义的)，因此它不会用作另一个函数的输入。

So far I have tried three-sigma, and the IQR outlier test on my limited data set. IQR flags values which are not extreme enough, three-sigma points out instances which better fit with my intuition of the domain.

到目前为止，我已经在我有限的数据集上尝试了三西格玛和IQR离群值测试。IQR标记了不够极端的值，三西格玛指出了更符合我对该领域的直觉的实例。

---

Information on algorithms, techniques or links to resources to learn about this specific scenario are valid and welcome answers.

有关算法、技术或资源链接的信息，以了解这一特定场景是有效和受欢迎的答案。

What is a recommended anomaly detection technique for simple, one-dimensional data?

对于简单的一维数据，推荐的异常检测技术是什么？

优秀答案推荐

Check out the three-sigma rule:

看看三西格玛规则：

mu  = mean of the data
std = standard deviation of the data
IF abs(x-mu) > 3*std  THEN  x is outlier

An alternative method is the IQR outlier test:

另一种方法是IQR离群值检验：

Q25 = 25th_percentile
Q75 = 75th_percentile
IQR = Q75 - Q25         // inter-quartile range
IF (x < Q25 - 1.5*IQR) OR (Q75 + 1.5*IQR < x) THEN  x is a mild outlier
IF (x < Q25 - 3.0*IQR) OR (Q75 + 3.0*IQR < x) THEN  x is an extreme outlier

this test is usually employed by Box plots (indicated by the whiskers):

此测试通常由盒子图(由胡须指示)使用：

---

EDIT:

编辑：

For your case (simple 1D univariate data), I think my first answer is well suited.
That however isn't applicable to multivariate data.

对于您的情况(简单的一维单变量数据)，我认为我的第一个答案非常合适。然而，这并不适用于多变量数据。

@smaclell suggested using K-means to find the outliers. Beside the fact that it is mainly a clustering algorithm (not really an outlier detection technique), the problem with k-means is that it requires knowing in advance a good value for the number of clusters K.

@smaclell建议使用K-Means来寻找离群值。除了它主要是一种聚类算法(不是真正的离群点检测技术)这一事实之外，k-Means的问题在于它需要预先知道聚类数目K的好值。

A better suited technique is the DBSCAN: a density-based clustering algorithm. Basically it grows regions with sufficiently high density into clusters which will be maximal set of density-connected points.

一种更适合的技术是DBSCAN：一种基于密度的集群算法。基本上，它将具有足够高密度的区域生长成簇，这将是密度连接点的最大集合。

DBSCAN requires two parameters: epsilon and minPoints. It starts with an arbitrary point that has not been visited. It then finds all the neighbor points within distance epsilon of the starting point.

DBSCAN需要两个参数：Epsilon和minPoints。它从一个尚未被访问的任意点开始。然后，它找到起点距离epsilon内的所有邻近点。

If the number of neighbors is greater than or equal to minPoints, a cluster is formed. The starting point and its neighbors are added to this cluster and the starting point is marked as visited. The algorithm then repeats the evaluation process for all the neighbors recursively.

如果邻居的数量大于或等于minPoints，则形成集群。起始点及其邻居被添加到该集群，并且起始点被标记为已访问。然后，该算法递归地对所有邻居重复评估过程。

If the number of neighbors is less than minPoints, the point is marked as noise.

如果邻居的数量小于minPoints，则将该点标记为噪波。

If a cluster is fully expanded (all points within reach are visited) then the algorithm proceeds to iterate through the remaining unvisited points until they are depleted.

如果簇被完全扩展(访问范围内的所有点)，则算法继续迭代剩余的未访问点，直到它们被耗尽。

Finally the set of all points marked as noise are considered outliers.

最后，标记为噪声的所有点的集合被认为是异常值。

There are a variety of clustering techniques you could use to try to identify central tendencies within your data. One such algorithm we used heavily in my pattern recognition course was K-Means. This would allow you to identify whether there are more than one related sets of data, such as a bimodal distribution. This does require you having some knowledge of how many clusters to expect but is fairly efficient and easy to implement.

您可以使用各种集群技术来尝试识别数据中的中心趋势。我们在模式识别课程中经常使用的一种算法是K-Means。这将允许您识别是否存在多个相关的数据集，例如双峰分布。这确实需要您对预期的集群数量有一定的了解，但这是相当高效且易于实现的。

After you have the means you could then try to find out if any point is far from any of the means. You can define 'far' however you want but I would recommend the suggestions by @Amro as a good starting point.

在你有了方法之后，你可以试着找出任何一点离任何方法都很远。你可以随心所欲地定义“Far”，但我会推荐@Amro的建议作为一个很好的起点。

For a more in-depth discussion of clustering algorithms refer to the wikipedia entry on clustering.

有关集群算法的更深入讨论，请参考关于集群的维基百科条目。

This is an old topic but still it lacks some information.

这是一个古老的话题，但仍然缺乏一些信息。

Evidently, this can be seen as a case of univariate outlier detection. The approaches presented above have several pros and cons. Here are some weak spots:

显然，这可以被视为单变量异常值检测的情况。上面提出的方法有几个利弊。以下是一些薄弱环节：

1. Detection of outliers with the mean and sigma has the obvious disadvantage of dependence of mean and sigma on the outliers themselves.


2. The case of the small sample limit (see question for example) is not adequately covered by, 3 sigma, K-Means, IQR etc.
   And I could go on... However the statistical literature offers a simple metric: the median absolute deviation. (Medians are insensitive to outliers)
   Details can be found here: https://www.sciencedirect.com/book/9780128047330/introduction-to-robust-estimation-and-hypothesis-testing

I think this problem can be solved in a few lines of python code like this:

我认为这个问题可以通过如下所示的几行Python代码来解决：

import numpy as np
import scipy.stats as sts

x = np.array([10, 14, 25, 467, 12]) # your values
np.abs(x - np.median(x))/(sts.median_abs_deviation(x)/0.6745) #MAD criterion

Subsequently you reject values above a certain threshold (97.5 percentile of the distribution of data), in case of an assumed normal distribution the threshold is 2.24. Here it translates to:

随后，您拒绝超过某个阈值(数据分布的97.5%)的值，在假设正态分布的情况下，阈值是2.24。在这里，它翻译为：

array([ 0.6745  ,  0.      ,  1.854875, 76.387125,  0.33725 ])

or the 467 entry being rejected.

或被拒绝的467条目。

Of course, one could argue, that the MAD (as presented) also assumes a normal dist. Therefore, why is it that argument 2 above (small sample) does not apply here? The answer is that MAD has a very high breakdown point. It is easy to choose different threshold points from different distributions and come to the same conclusion: 467 is the outlier.

当然，人们可以争辩说，疯子(如上所述)也假定有正常的离散性。因此，为什么上面的论点2(小样本)不适用于这里？答案是MAD有一个非常高的故障点。很容易从不同的分布中选择不同的阈值，并得出相同的结论：467是异常值。

Both three-sigma rule and IQR test are often used, and there are a couple of simple algorithms to detect anomalies.

三西格玛规则和IQR检验都是经常使用的，而且有几个简单的算法来检测异常。

The three-sigma rule is correct
mu  = mean of the data
std = standard deviation of the data
IF abs(x-mu) > 3*std  THEN  x is outlier

The IQR test should be:

IQR测试应该是：

Q25 = 25th_percentile
Q75 = 75th_percentile
IQR = Q75 - Q25         // inter-quartile range
If x >  Q75  + 1.5 * IQR or  x   < Q25 - 1.5 * IQR THEN  x is a mild outlier
If x >  Q75  + 3.0 * IQR or  x   < Q25 – 3.0 * IQR THEN  x is a extreme outlier

The anomaly detection of one-dimensional data is an open challenge. I have published a Python package, named xiezhi, which can be applied to detect the abnormal data in a list, especially when the list is large while only a few data in it are anomalies. This tool is based on one of my research papers, and it has been proven to be theoretically robust. Here is a tutorial for xiezhi: https://medium.com/@hellojerrywong18/xiezhi-the-anomaly-detection-tool-for-one-dimensional-data-9108c539e692

一维数据的异常检测是一个开放的挑战。我已经发布了一个名为协志的Python包，它可以用来检测列表中的异常数据，特别是当列表很大而其中只有几个数据是异常的时候。这个工具是基于我的一篇研究论文，它已经被证明在理论上是健壮的。这里有一个谢志的教程：https://medium.com/@hellojerrywong18/xiezhi-the-anomaly-detection-tool-for-one-dimensional-data-9108c539e692

If you have any problem or suggestion, please let me know.

如果您有任何问题或建议，请让我知道。