elasticsearch: cluster_block_exception TOO_MANY_REQUESTS/12/index 只读/允许删除 (api)

Question

我正在运行带有多进程索引的爬虫，我想解决这个问题。

我总是收到 TOO_MANY_REQUESTS，有没有我可以更改的配置以允许更大的阈值来处理并发请求？

这是完整的错误

2020-12-29 06:24:13 [scrapy.core.scraper] ERROR: Spider error processing <GET https://github.com/topics/python?o=desc&s=updated&page=5> (referer: None)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/scrapy/spiders/__init__.py", line 90, in _parse
    return self.parse(response, **kwargs)
  File "/usr/src/app/xxxx/spiders/python_topic.py", line 85, in parse
    print_data(repository)
  File "/usr/src/app/xxxx/spiders/python_topic.py", line 42, in print_data
    repository.save(repository_doc)
  File "/usr/src/app/xxxx/model/repositories.py", line 18, in save
    repo.save()
  File "/usr/local/lib/python3.8/site-packages/toute/document.py", line 90, in save
    saved_document = self.get_es(es).index(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 398, in index
    return self.transport.perform_request(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/transport.py", line 392, in perform_request
    raise e
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/transport.py", line 358, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 269, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 315, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.TransportError: TransportError(429, 'cluster_block_exception', 'index [xxxxxx] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];')

更新:发现洪水盘水印超出了，但似乎没有超出!

elasticsearch_1  | {"type": "server", "timestamp": "2020-12-29T06:40:51,701Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "talks-cluster", "node.name": "n1_it", "message": "flood stage disk watermark [5gb] exceeded on [Z8G4W5UlQYy1HOk0JWGP7w][n1_it][/usr/share/elasticsearch/data/nodes/0] free: 3.8gb[1.6%], all indices on this node will be marked read-only", "cluster.uuid": "uo5vK64tQoyfIeGiUhlStg", "node.id": "Z8G4W5UlQYy1HOk0JWGP7w"  }

df 命令的输出

Filesystem    512-blocks      Used Available Capacity iused      ifree %iused  Mounted on
/dev/disk1s6   489825072  22025208  12057544    65%  488254 2448637106    0%   /
devfs                385       385         0   100%     666          0  100%   /dev
/dev/disk1s1   489825072 413862176  12057544    98% 4477496 2444647864    0%   /System/Volumes/Data
/dev/disk1s4   489825072  17879464  12057544    60%       8 2449125352    0%   /private/var/vm
/dev/disk1s5   489825072  21349800  12057544    64%  487049 2448638311    0%   /Volumes/Ohne Titel
map auto_home          0         0         0   100%       0          0  100%   /System/Volumes/Data/home

Answer 1

根据您收到的错误消息，您的磁盘已满

flood stage disk watermark [5gb] exceeded on [Z8G4W5UlQYy1HOk0JWGP7w][n1_it][/usr/share/elasticsearch/data/nodes/0] free: 3.8gb[1.6%], all indices on this node will be marked read-only

您需要增加该节点的可用磁盘空间。