hadoop - HDFS上的Solr集合创建失败

Question

我正在尝试在Hadoop上运行Solr 7.4.0(我已经在Windows 10上成功安装了Hadoop 2.7.6)。
按照Solr用户指南，我打开了_default config的文件“solrconfig.xml”，并替换了该行

<directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>

通过

<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
  <str name="solr.hdfs.home">hdfs://localhost:9000/solr</str>
  <bool name="solr.hdfs.blockcache.enabled">true</bool>
  <int name="solr.hdfs.blockcache.slab.count">1</int>
  <bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
  <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
  <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
  <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
  <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
  <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
</directoryFactory>

我使用命令行在HDFS上启动了SolrCloud，它成功运行

solr start -c -Dsolr.directoryFactory=HdfsDirectoryFactory -Dsolr.lock.type=hdfs -Dsolr.hdfs.home=hdfs://localhost:9000/solr

一些论点如下

-Djava.io.tmpdir=d:\Project\Java\hadoop-2.7.6\solr-7.4.0\server\tmp
-Djetty.home=d:\Project\Java\hadoop-2.7.6\solr-7.4.0\server
-Djetty.host=0.0.0.0
-Djetty.port=8983
-Dlog4j.configurationFile=file:d:\Project\Java\hadoop-2.7.6\solr-7.4.0\server\resources\log4j2.xml
-Dsolr.default.confdir=d:\Project\Java\hadoop-2.7.6\solr-7.4.0\server\solr\configsets\_default\conf
-Dsolr.directoryFactory=HdfsDirectoryFactory
-Dsolr.hdfs.home=hdfs://localhost:9000/solr
-Dsolr.install.dir=d:\Project\Java\hadoop-2.7.6\solr-7.4.0
-Dsolr.lock.type=hdfs
-Dsolr.log.dir=d:\Project\Java\hadoop-2.7.6\solr-7.4.0\server\logs
-Dsolr.solr.home=d:\Project\Java\hadoop-2.7.6\solr-7.4.0\server\solr

在下一步中，我尝试通过执行此请求来创建集合“测试”

http://localhost:8983/solr/admin/collections?action=CREATE&name=Test&numShards=1&replicationFactor=1

和Solr返回错误

............
Caused by: java.nio.file.InvalidPathException: Illegal char <:> at index 4: hdfs://localhost:9000/solr/Test/core_node2/data\
    at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
    at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
    at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
    at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
    at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
    at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
    at org.apache.solr.core.SolrCore.initUpdateLogDir(SolrCore.java:1341)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:923)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:867)
    at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1135)
    ... 45 more

有人遇到过同样的问题吗？如何创建SolrCloud集合并存储在Hadoop上？

Answer 1

您将需要在solrconfig.xml中使用以下目录工厂。

<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
<str name="solr.hdfs.home">hdfs://namenodeserver:8020/apps/solr</str>
<str name="solr.hdfs.confdir">/etc/hadoop/conf</str>
<str name="solr.hdfs.security.kerberos.enabled">true</str>
<str name="solr.hdfs.security.kerberos.keytabfile">/etc/security/keytabs/solr.service.keytab</str>
<str name="solr.hdfs.security.kerberos.principal">@principal</str>
<bool name="solr.hdfs.blockcache.enabled">true</bool>
<bool name="solr.hdfs.blockcache.global">true</bool>
<int name="solr.hdfs.blockcache.slab.count">2</int>
<bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
<int name="solr.hdfs.blockcache.blocksperbank">16384</int>
<bool name="solr.hdfs.blockcache.read.enabled">true</bool>
<bool name="solr.hdfs.blockcache.write.enabled">false</bool>
<bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
<int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
<int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
</directoryFactory>

将配置上传到zookeeper，如下所示。

sh /opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh --zkhost hostname1.com,hostname2.com,hostname3.com:2181/solr -cmd upconfig --collection Event --confname webconf --solrhome /home/solr/Event/ --confdir /home/solr/Event/conf

创建如下的hdfs集合。

http://solrhostname:8983/solr/admin/collections?action=CREATE&name=Event&numShards=13&replicationFactor=3&maxShardsPerNode=10&instanceDir=hdfs://namenode/apps/solr/Event&config=hdfs://namenode/apps/solr/Event/solrconfig.xml&schema=hdfs://namenode/apps/solr/Event/managed-schema&dataDir=hdfs://namenode/apps/solr/Event/data&collection.configName=webconf