hadoop - CDH 5.7 上的 Streamsets solrcloud 无法连接到 Solr

Question

我在 CDH 5.7.0 版上使用流集

从 HDFS(源)加载文件并在 Solr(目标)上创建记录的示例工作流程。验证失败 - SOLR_03 - 无法连接到 Solr 实例:org.apache.solr.common.cloud.ZooKeeperException::ZooKeeper 连接字符串。

但是，从日志中，我看到它能够连接到 zookeeper，但是，在调用/live_nodes zookeeper api 后以某种方式关闭了 session 。

2016-06-03 16:30:21,336 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper

2016-06-03 16:30:21,336 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkACLProvider

2016-06-03 16:30:21,338 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 1,3  replyHeader:: 1,42949673777,0  request:: '/clusterstate.json,F  response:: s{38654846788,38654846788,1464989414801,1464989414801,0,0,0,0,0,0,38654846788} 

2016-06-03 16:30:21,339 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 2,3  replyHeader:: 2,42949673777,0  request:: '/aliases.json,F  response:: s{38654846789,38654846789,1464989414814,1464989414814,0,0,0,0,0,0,38654846789} 

2016-06-03 16:30:21,339 INFO org.apache.solr.common.cloud.ZkStateReader: Updating cluster state from ZooKeeper... 

2016-06-03 16:30:21,339 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 3,3  replyHeader:: 3,42949673777,0  request:: '/clusterstate.json,T  response:: s{38654846788,38654846788,1464989414801,1464989414801,0,0,0,0,0,0,38654846788} 

2016-06-03 16:30:21,340 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 4,8  replyHeader:: 4,42949673777,-101  request:: '/live_nodes,T  response:: v{} 

2016-06-03 16:30:21,340 DEBUG org.apache.zookeeper.ZooKeeper: Closing session: 0x355187691370039

2016-06-03 16:30:21,340 DEBUG org.apache.zookeeper.ClientCnxn: Closing client for session: 0x355187691370039

2016-06-03 16:30:21,352 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 5,-11  replyHeader:: 5,42949673778,0  request:: null response:: null

2016-06-03 16:30:21,352 DEBUG org.apache.zookeeper.ClientCnxn: Disconnecting client for session: 0x355187691370039

2016-06-03 16:30:21,352 INFO org.apache.zookeeper.ZooKeeper: Session: 0x355187691370039 closed

2016-06-03 16:30:21,352 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down

2016-06-03 16:30:21,352 INFO com.streamsets.pipeline.cluster.ControlChannel: Producer complete

2016-06-03 16:30:21,352 DEBUG com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Returning private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib'

2016-06-03 16:30:21,352 DEBUG com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Returned a private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib', active private ClassLoaders='0'

2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager is shutting down

2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager shut down

2016-06-03 16:30:21,353 DEBUG com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager: Status of previewer with id: '0bec83af-c152-4519-98f4-201394bebc36' changed to status: 'INVALID'

2016-06-03 16:30:21,353 INFO com.streamsets.pipeline.cluster.ControlChannel: Producer complete

2016-06-03 16:30:21,353 DEBUG com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Returning private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib'

2016-06-03 16:30:21,353 WARN com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Could not return a private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib', active private ClassLoaders='0'

2016-06-03 16:30:21,353 WARN com.streamsets.datacollector.runner.Pipeline: Exception thrown during pipe 'StagePipe[instance='com_streamsets_pipeline_stage_origin_hdfs_cluster_ClusterHdfsDSource_01' input='[]' output='[com_streamsets_pipeline_stage_origin_hdfs_cluster_ClusterHdfsDSource_01OutputLane14649766295360::s]']' destroy: java.lang.RuntimeException: java.lang.IllegalStateException: Object has already been returned to this pool or is invalid

java.lang.RuntimeException: java.lang.IllegalStateException: Object has already been returned to this pool or is invalid

at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.releaseStageClassLoader(ClassLoaderStageLibraryTask.java:448)

at com.streamsets.datacollector.runner.preview.PreviewStageLibraryTask.releaseStageClassLoader(PreviewStageLibraryTask.java:92)

at com.streamsets.datacollector.creation.StageBean.releaseClassLoader(StageBean.java:60)

at com.streamsets.datacollector.runner.StageRuntime.destroy(StageRuntime.java:203)

at com.streamsets.datacollector.runner.StagePipe.destroy(StagePipe.java:275)

at com.streamsets.datacollector.runner.Pipeline.destroy(Pipeline.java:187)

at com.streamsets.datacollector.runner.preview.PreviewPipeline.destroy(PreviewPipeline.java:71)

at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:205)

at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer$2.call(AsyncPreviewer.java:96)

at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:233)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.IllegalStateException: Object has already been returned to this pool or is invalid

at org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:484)

at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.releaseStageClassLoader(ClassLoaderStageLibraryTask.java:442)

... 15 more

2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager is shutting down

2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager shut down

因此，为了验证，我连接到 zookeeper 并查询了 live_nodes:

[root@hadoop3 ~]# zookeeper-client

Connecting to localhost:2181

Welcome to ZooKeeper!

[zk: localhost:2181(CONNECTED) 5] ls /solr

[configs, overseer, aliases.json, live_nodes, collections, overseer_elect, clusterstate.json, solr.xml, zkdtsm, clusterprops.json, token]

[zk: localhost:2181(CONNECTED) 6] ls /solr/live_nodes

[hadoop3:8983_solr, hadoop4:8983_solr, hadoop2:8983_solr, hadoop5:8983_solr]

不确定，为什么当 streamsets 调用 API 时响应为空以及为什么它在能够连接时提示“Zookeeper Connection String”...有帮助吗？

Answer 1

将 /solr 添加到您的 ZooKeeper URL，使其看起来像:host:port/solr。