hadoop - 槽不适用于hdfs

Question

我将flume设置为两个节点，我想将数据从slave01加载到hdfs。
slave01:example-conf.properties

 agent.sources = baksrc
agent.channels = memoryChannel
agent.sinks =avro-forward-sink 
agent.sources.baksrc.type = exec
agent.sources.baksrc.command = tail -F /root/hadoo/test/data.txt
agent.sources.baksrc.checkperiodic = 1000
agent.sources.baksrc.channels =memoryChannel 
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.keep-alive = 30
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity = 10000
agent.sinks.avro-forward-sink.type = avro
agent.sinks.avro-forward-sink.hostname = master
agent.sinks.avro-forward-sink.port = 23004
agent.sinks.avro-forward-sink.channel = memoryChannel

母版:example-conf.properties

agent.sources = avrosrc
agent.sinks =hdfs-write 
agent.channels = memoryChannel
agent.sources.avrosrc.type =avro
agent.sources.avrosrc.bind =master
agent.sources.avrosrc.port =23004
agent.sources.avrosrc.channels=memoryChannel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.keep-alive = 30
agent.channels.memoryChannel.capacity = 10000
agent.channels.memoryChannel.transactionCapacity =10000
agent.sinks.hdfs-write.type = hdfs
agent.sinks.hdfs-write.hdfs.path =hdfs://172.16.86.38:9000/flume/webdata
agent.sinks.hdfs-write.hdfs.rollInterval = 0   
agent.sinks.hdfs-write.hdfs.rollSize = 4000000  
agent.sinks.hdfs-write.hdfs.rollCount = 0   
agent.sinks.hdfs-write.hdfs.writeFormat = Text  
agent.sinks.hdfs-write.hdfs.fileType = DataStream  
agent.sinks.hdfs-write.hdfs.batchSize = 10  
agent.sinks.hdfs-write.channel=memoryChannel  

然后我运行一个shell脚本:像这样:

#!/bin/sh
for i in {1..1000000}; do
    echo "test flume to Hbase $i" >>/root/hadoop/test/data.txt; 
    sleep 0.1; 
done

开始水槽:
flume-ng代理--conf conf --conf文件示例-conf.properties --name代理-Dflume.root.logger = DEBUG，控制台
我在控制台上没有错误。

14/05/06 16:38:44 INFO source.AvroSource: Avro source avrosrc stopping: Avro source avrosrc: { bindAddress: master, port: 23004 }
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] DISCONNECTED
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] UNBOUND
14/05/06 16:38:44 INFO ipc.NettyServer: [id: 0x49f2de1b, /172.16.86.39:9359 :> /172.16.86.38:23004] CLOSED

但是我在hdfs中看不到文件，我的配置有问题吗？
我已经在master上测试过了，工作正常。

Answer 1

您使用哪个版本的水槽？
您设置了HADOOP_HOME吗？
flume是否从HADOOP_HOME用hadoop jar输出类路径？
如果您使用的是 Apache 水槽，那么请逐步进行操作:
1.设置HADOOP_HOME
2.编辑hadoop core-site.xml，确保namenode ip正确。
3.使用hdfs路径:agent.sinks.hdfs-write.hdfs.path =/flume/webdata4.开始水槽