python-3.x - 使用 Python 通过 Kerberos 身份验证连接到 HDFS

Question

我正在尝试连接到受 Kerberos 身份验证保护的 HDFS。我有以下详细信息，但不知道如何继续。

User
Password
Realm
HttpFs Url

我尝试了以下代码，但出现身份验证错误:

from hdfs.ext.kerberos import KerberosClient
import requests
import logging

logging.basicConfig(level=logging.DEBUG)

session = requests.Session()
session.verify = False

client = KerberosClient(url='http://x.x.x.x:abcd', session=session, 
mutual_auth='REQUIRED',principal='abcdef@LMNOPQ')

print(client.list('/'))

错误

INFO:hdfs.client:Instantiated   
<KerberosClient(url=http://x.x.x.x:abcd)>.
INFO:hdfs.client:Listing '/'.
DEBUG:hdfs.client:Resolved path '/' to '/'.
DEBUG:hdfs.client:Resolved path '/' to '/'.
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 
DEBUG:urllib3.connectionpool:http://x.x.x.x:abcd "GET /webhdfs/v1/? 
op=LISTSTATUS HTTP/1.1" 401 997
DEBUG:requests_kerberos.kerberos_:handle_401(): Handling: 401
ERROR:requests_kerberos.kerberos_:generate_request_header(): authGSSClientInit() failed:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_kerberos/kerberos_.py", line 213, in generate_request_header
gssflags=gssflags, principal=self.principal)
kerberos.GSSError: ((' No credentials were supplied, or the credentials were unavailable or inaccessible.', 458752), ('unknown mech-code 0 for mech unknown', 0))
ERROR:requests_kerberos.kerberos_:((' No credentials were supplied, or the credentials were unavailable or inaccessible.', 458752), ('unknown mech-code 0 for mech unknown', 0))
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_kerberos/kerberos_.py", line 213, in generate_request_header
gssflags=gssflags, principal=self.principal)
kerberos.GSSError: ((' No credentials were supplied, or the credentials were unavailable or inaccessible.', 458752), ('unknown mech-code 0 for mech unknown', 0))
DEBUG:requests_kerberos.kerberos_:handle_401(): returning <Response [401]>
DEBUG:requests_kerberos.kerberos_:handle_response(): returning <Response [401]>

我也有密码，但不知道在哪里提供它。

Answer 1

假设您有原则:hdfs/localhost@HADOOP.COM 并且您的 key 表文件是:/var/run/cloudera-scm-agent/process/39-hdfs-NAMENODE/hdfs.keytab 并且如果您想阅读 hdfs csv文件已在:/hadoop_test_data/filecount.csv，然后使用以下代码，您将获得包含 filecount.csv 内容的 Pandas 数据框

在这里，我使用了 python 版本: 3.7.6

import io 
from csv import reader
from krbcontext import krbcontext
import subprocess 
import pandas as pd

try:
    with krbcontext(using_keytab=True,
                    principal='hdfs/localhost@HADOOP.COM',
                    keytab_file='/var/run/cloudera-scm-agent/process/39-hdfs-NAMENODE/hdfs.keytab') as krb:
                    print(krb)
                    print('kerberos authentication successful') 
                    output = subprocess.Popen(["hadoop", "fs", "-cat", "/hadoop_test_data/filecount.csv"], stdout=subprocess.PIPE)
                    stdout,stderr = output.communicate()
                    data = str(stdout,'utf-8').split('\r\n')
                    df = pd.DataFrame( list(reader(data[1:])),columns=data[0].split(','))
                    print(df.shape)
                    print(df)

except Exception as e:
    print("Kerberos authentication unsuccessful")
    print("Detailed error is : "+e)

如果您想了解更多信息，请告诉我。