java - ElasticSearch 内存不足

Question

我正在尝试在 ES 中索引一些数据，但收到内存不足异常:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.elasticsearch.common.jackson.core.util.BufferRecycler.balloc(BufferRecycler.java:155)
    at org.elasticsearch.common.jackson.core.util.BufferRecycler.allocByteBuffer(BufferRecycler.java:96)
    at org.elasticsearch.common.jackson.core.util.BufferRecycler.allocByteBuffer(BufferRecycler.java:86)
    at org.elasticsearch.common.jackson.core.io.IOContext.allocWriteEncodingBuffer(IOContext.java:152)
    at org.elasticsearch.common.jackson.core.json.UTF8JsonGenerator.<init>(UTF8JsonGenerator.java:123)
    at org.elasticsearch.common.jackson.core.JsonFactory._createUTF8Generator(JsonFactory.java:1284)
    at org.elasticsearch.common.jackson.core.JsonFactory.createGenerator(JsonFactory.java:1016)
    at org.elasticsearch.common.xcontent.json.JsonXContent.createGenerator(JsonXContent.java:68)
    at org.elasticsearch.common.xcontent.XContentBuilder.<init>(XContentBuilder.java:96)
    at org.elasticsearch.common.xcontent.XContentBuilder.builder(XContentBuilder.java:77)
    at org.elasticsearch.common.xcontent.json.JsonXContent.contentBuilder(JsonXContent.java:38)
    at org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:122)
    at org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder(XContentFactory.java:49)
    at EsController.importProductEs(EsController.java:60)
    at Parser.fromCsvToJson(Parser.java:120)
    at CsvToJsonParser.parseProductFeeds(CsvToJsonParser.java:43)
    at MainParser.main(MainParser.java:49)

这就是我实例化 ES 客户端的方式:

System.out.println("Elastic search client is instantiated");
Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "elasticsearch_brew").build();
client = new TransportClient(settings);
String hostname = "localhost";
int port = 9300; 
((TransportClient) client).addTransportAddress(new InetSocketTransportAddress(hostname, port));     
bulkRequest = client.prepareBulk();

然后我运行批量请求:

// for each product in the list, we need to include the fields in the bulk request 
for(HashMap<String, String> productfields : products)
        try {
            bulkRequest.add(client.prepareIndex(index,type,productfields.get("Product_Id"))
                    .setSource(jsonBuilder()
                                .startObject()
                                    .field("Name",productfields.get("Name") )
                                    .field("Quantity",productfields.get("Quantity"))
                                    .field("Make", productfields.get("Make"))
                                    .field("Price", productfields.get("Price"))
                                .endObject()
                              )
                    );                  

        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
//execute the bulk request
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures()) {
    // process failures by iterating through each bulk response item
}

我正在尝试对各个商店的产品进行索引。每个商店都有不同的索引。当我到达包含大约 60000 种产品的第 6 家商店时，我得到了上述异常。我将批量请求分成 10000 个 block ，试图避免内存不足问题。我无法理解瓶颈到底在哪里。如果我以某种方式刷新批量请求或重新启动客户端会有帮助吗？我看过类似的帖子，但对我不起作用。

编辑

当我每次处理新的批量请求时实例化一个新客户端时，我不会遇到内存不足异常。但每次实例化一个新客户端似乎并不正确..

谢谢

Answer 1

所以我弄清楚出了什么问题。

每个新的批量请求都会与前一个请求相加，最终导致内存不足。

所以现在在开始新的批量请求之前我运行bulkRequest = client.prepareBulk();这会刷新之前的请求。

谢谢大家的评论