Hadoop webhdfs 处于安全模式，用户未通过过滤器错误进行身份验证

Question

我已经使用 Kerberos (CDH4.3) 配置了启用安全性的 Hadoop HDFS。它工作正常，我可以使用命令行工具完成所有操作。还有一些其他非 hadoop 集群节点需要通过 REST API 访问 HDFS 文件系统，所以我启用了配置了 Kerberos 的 webhdfs。

我的安全 webhdfs 对于不涉及临时重定向的操作工作正常:即 LISTSTATUS、创建目录、删除目录、删除文件，所有工作正常。但是对于创建文件、打开文件等两步操作，它失败并显示以下信息:

[DEV][root@namenode ~]# curl -i --negotiate -u :   -X     
PUT "http: //namenode:50070/webhdfs/v1/tmp/hosts.txt?op=CREATE"

HTTP/1.1 401
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 0
Server: Jetty(6.1.26.cloudera.2)

HTTP/1.1 307 TEMPORARY_REDIRECT
Cache-Control: no-cache
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
Date: Fri, 25 Apr 2014 02:45:48 GMT
Pragma: no-cache
Set-Cookie:       
hadoop.auth="u=hdfs&p=hdfs@UNIXKRB&t=kerberos&e=1398429948267&s=XhEp/tfs5Pfp04Dp
9yy1moFPnEo=";Path=/
Location: http: //datanode.net:1006/webhdfs/v1/tmp/hosts.txt?      
op=CREATE&delegation=HgAEaGRmcwRoZGZzAIoBRZbGSGyKAUW60sxsWo4BAhTfG8IRyDJjJ2BSR7K
hveMEo3V4dxJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA&namenoderpcaddress=
namenode.net:8020&overwrite=false
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(6.1.26.cloudera.2)


[DEV][root@namenode~]# curl -i -X PUT -T hosts.txt     
http: //datanode.net:1006/webhdfs/v1/tmp/hosts.txt? 
op=CREATE&delegation=HgAEaGRmcwRoZGZzAIoBRZbGSGyKAUW60sxsWo4BAhTfG8IRyDJjJ2
BSR7KhveMEo3V4dxJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA&namenoderpcaddress=
namenode.net:8020&overwrite=false
[2] 19047
[3] 19048
[4] 19049
[DEV][root@namenode~]# HTTP/1.1 100 Continue

HTTP/1.1 401 Unauthorized
Cache-Control: no-cache
Expires: Fri, 25 Apr 2014 02:46:26 GMT
Date: Fri, 25 Apr 2014 02:46:26 GMT
Pragma: no-cache
Expires: Fri, 25 Apr 2014 02:46:26 GMT
Date: Fri, 25 Apr 2014 02:46:26 GMT
Pragma: no-cache
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26.cloudera.2)

{"RemoteException":       
{"exception":"SecurityException","javaClassName":"java.lang.SecurityException",
"message":"Failed to obtain user group information: java.io.IOException: 
 Security enabled but user not authenticated by filter"}}
[2]   Done                    curl -i -X PUT -T hosts.txt     
http://datanode.net:1006/webhdfs/v1/tmp/hosts.txt?op=CREATE
[3]   Done                        
delegation=HgAEaGRmcwRoZGZzAIoBRZbGSGyKAUW60sxsWo4BAhTfG8IRyDJjJ2BSR7KhveM
Eo3V4dxJXRUJIREZTIGRlbGVnYXRpb24NMC4wLjAuMDo1MDA3MA
[4]-  Done                    namenoderpcaddress=namenode.net:8020
[DEV][root@namenode ~]#

有没有人可以开灯？

非常感谢，

Answer 1

在我的例子中，它工作正常，如下所示:

$ curl -i --negotiate -u : -X PUT "http://nn30.my.dom:50070/webhdfs/v1/tmp/hosts.txt?op=CREATE"
HTTP/1.1 401
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 0
Server: Jetty(7.6.10.v20130312)


HTTP/1.1 307 Temporary Redirect
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
Cache-Control: no-cache
Date: Wed, 30 Apr 2014 12:39:24 GMT
Pragma: no-cache
Set-Cookie: hadoop.auth="u=gpadmin&p=gpadmin@MY.DOM&t=kerberos&e=1398897564735&s=p8haj7KXAsUqj5A2WzoE5VxJYA8=";Path=/
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Location: http ://dn33.my.dom:1006/webhdfs/v1/tmp/hosts.txt?op=CREATE&delegation=IgAHZ3BhZG1pbgdncGFkbWluAIoBRbKli0OKAUXWsg9DAxIUt8i6BG2xt5V4wB4vPpXSPm8MMVMSV0VCSERGUyBkZWxlZ2F0aW9uEzE5Mi4xNjguMTAuMzA6NTAwNzA&namenoderpcaddress=nn30.my.dom:8020&overwrite=false
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(7.6.10.v20130312)


$ curl -i -X PUT -T hosts.txt "http://dn33.my.dom:1006/webhdfs/v1/tmp/hosts.txt?op=CREATE&delegation=IgAHZ3BhZG1pbgdncGFkbWluAIoBRbKli0OKAUXWsg9DAxIUt8i6BG2xt5V4wB4vPpXSPm8MMVMSV0VCSERGUyBkZWxlZ2F0aW9uEzE5Mi4xNjguMTAuMzA6NTAwNzA&namenoderpcaddress=nn30.my.dom:8020&overwrite=false"
HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Expires: Wed, 30 Apr 2014 12:42:24 GMT
Date: Wed, 30 Apr 2014 12:42:24 GMT
Pragma: no-cache
Cache-Control: no-cache
Expires: Wed, 30 Apr 2014 12:42:24 GMT
Date: Wed, 30 Apr 2014 12:42:24 GMT
Pragma: no-cache
Location: webhdfs ://nn30.my.dom:50070/tmp/hosts.txt
Content-Type: application/octet-stream
Content-Length: 0
Server: Jetty(7.6.10.v20130312)

注意 - 如果您的屏幕截图未经过编辑/修改，您几乎没有语法问题 ("") 和拼写错误。我打赌你错过了第一个命令的 http 输入的 ""。在此处查看您的信息 - http://hadoop.apache.org/docs/r1.0.4/webhdfs.html

顺便说一句，我使用的是 Pivotal HD，不过应该没有太大区别。