hadoop - HBase 旧 WALs : what it is and how can I clean it?

Question

我们的小型 hadoop 集群空间不足，因此我检查了 HDFS 上的磁盘使用情况，发现大部分空间都被/hbase/oldWALs 文件夹占用。

我查看了“HBase Definitive Book”和其他书籍、网站，我还在谷歌上搜索了我的问题，但我没有找到合适的回复...

所以我想知道这个文件夹是做什么的，有什么用，以及如何在不破坏一切的情况下从这个文件夹中释放空间...

如果它与特定版本有关...我们的集群位于 Cloudera (hbase 0.98.6) 的 5.3.0-1.cdh5.3.0.p0.30 下。

感谢您的帮助!

Answer 1

仅供引用

我已经在 hbase 用户列表中发布了这个问题。这是 Enis Söztutar(一名 hbase 提交者)的回答以及我是如何解决它的:

The folder gets cleaned regularly by a chore in master. When a WAL file is not needed any more for recovery purposes (when HBase can guaratee HBase has flushed all the data in the WAL file), it is moved to the oldWALs folder for archival. The log stays there until all other references to the WAL file are finished. There is currently two services which may keep the files in the archive dir. First is a TTL process, which ensures that the WAL files are kept at least for 10 min. This is mainly for debugging. You can reduce this time by setting hbase.master.logcleaner.ttl configuration property in master. It is by default 600000. The other one is replication. If you have replication setup, the replication processes will hang on to the WAL files until they are replicated. Even if you disabled the replication, the files are still referenced.

You can look at the logs from master from classes (LogCleaner, TimeToLiveLogCleaner, ReplicationLogCleaner) to see whether the master is actually running this chore and whether it is getting any exceptions.

复制确实无法在所有集群上进行，但过去它是启用的，因为我们使用 hbase-indexer 将数据从 HBase 复制到 Solr，这种机制是基于复制的。

我已经在 hbase shell 上运行了这个命令:

hbase(main):005:0> list_peers
PEER_ID CLUSTER_KEY STATE TABLE_CFS
Indexer_profilesIndexer m1.prod.ps,m2.prod.ps,m3.prod.ps:2181:/ngdata/sep/hbase-slave/Indexer_profilesIndexer DISABLED nil
1 row(s) in 0.0070 seconds

hbase(main):006:0> remove_peer 'Indexer_profilesIndexer'
0 row(s) in 0.0050 seconds


hbase(main):007:0> list_peers
PEER_ID CLUSTER_KEY STATE TABLE_CFS
0 row(s) in 0.0020 seconds

最后我删除了 hdfs 上的 oldsWALs 文件夹!

文件夹不再增长!