Python写入hdfs文件

Question

从本地 python 脚本在远程 HDFS 中创建/写入/更新文件的最佳方法是什么？

我能够列出文件和目录，但写入似乎是个问题。

我搜索了hdfs和 snakebite但他们都没有给出一个干净的方法来做到这一点。

Answer 1

试试 HDFS liberary..它真的很好你可以使用 write()。 https://hdfscli.readthedocs.io/en/latest/api.html#hdfs.client.Client.write

例子:

创建连接:

from hdfs import InsecureClient
client = InsecureClient('http://host:port', user='ann')

from json import dump, dumps
records = [
  {'name': 'foo', 'weight': 1},
  {'name': 'bar', 'weight': 2},
]

# As a context manager:
with client.write('data/records.jsonl', encoding='utf-8') as writer:
  dump(records, writer)

# Or, passing in a generator directly:
client.write('data/records.jsonl', data=dumps(records), encoding='utf-8')

对于 CSV 你可以做

import pandas as pd
df=pd.read.csv("file.csv")
with client.write('path/output.csv', encoding = 'utf-8') as writer:
  df.to_csv(writer)