java - 如何将 UserGroupInformation 与 Kerberos WebHDFS 一起使用

Question

以下是非 hadoop 系统上的客户端代码，用于在安全的远程 HDFS 上执行操作。

Configuration conf = new
        Configuration();
conf.set("hadoop.security.authentication", "kerberos");
conf.set("java.security.krb5.conf",krbPath);
conf.set("fs.defaultFS", "webhdfs://10.31.251.254:50070");
conf.set("fs.webhdfs.impl", org.apache.hadoop.hdfs.web.WebHdfsFileSystem.class.getName());
conf.set("com.sun.security.auth.module.Krb5LoginModule", "required");
conf.set("debug", "true");
conf.set("ticketCache", "DIR:/etc/");
System.out.print("Conf......");

UserGroupInformation.setConfiguration(conf);



UserGroupInformation.loginUserFromKeytab("Dummy@EXAMPLE.COM", keytab);
System.out.print("Obtained......");
URI uri = URI.create("webhdfs://Dummy:50070");
FileSystem fs = FileSystem.get(uri, conf);

if (fs.mkdirs(new Path("/testKerb2")))
    System.out.print("Directory created...");

我能够执行这些操作，但无法从 krb5.conf 中读取票证配置值。 conf 文件中提到的票证生命周期为 1m，但代码生成的票证生命周期为 1d。而且票证不会在配置的 ticketCache 中生成。

请帮助配置，以便代码从 krb5.conf 文件中读取并在配置的路径中生成票据。

在控制台中关注 o/p

*911 [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - hadoop login commit
912 [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - using kerberos user:hdfs/YYYY@EXAMPLE.COM
914 [main] INFO org.apache.hadoop.security.UserGroupInformation  - Login successful for user hdfs/YYYY@EXAMPLE.COM using keytab file wcnew.keytab
Obtained......998 [main] DEBUG org.apache.hadoop.io.retry.RetryUtils  - multipleLinearRandomRetry = null
1026 [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:hdfs/YYYY@EXAMPLE.COM (auth:KERBEROS) from:org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getHttpUrlConnection(WebHdfsFileSystem.java:456)
1027 [main] DEBUG org.apache.hadoop.hdfs.web.WebHdfsFileSystem  - open AuthenticatedURL connection
1051 [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - Found tgt Ticket (hex) =
0000: 61 82 01 42 30 82 01 3E   A0 03 02 01 05 A1 0D 1B  a..B0..>........
0010: 0B 45 58 41 4D 50 4C 45   2E 43 4F 4D A2 20 30 1E  .EXAMPLE.COM. 0.
0020: A0 03 02 01 02 A1 17 30   15 1B 06 6B 72 62 74 67  .......0...krbtg
0030: 74 1B 0B 45 58 41 4D 50   4C 45 2E 43 4F 4D A3 82  t..EXAMPLE.COM..
0040: 01 04 30 82 01 00 A0 03   02 01 12 A1 03 02 01 01  ..0.............
0050: A2 81 F3 04 81 F0 EC 1A   94 3A 38 70 90 14 04 B5  .........:8p....
0060: 23 A5 0A 68 78 9E 52 74   A8 2C C2 98 8D FA 6F AD  #..hx.Rt.,....o.
0070: B1 8F 4A 69 02 B1 13 A0   8B 45 B1 51 1F 48 A6 2B  ..Ji.....E.Q.H.+
0080: 22 23 26 63 05 12 7F 1A   38 A9 81 0B 5B EA FA CC  "#&c....8...[...
0090: A7 D3 BC 15 37 46 32 2F   94 D4 A3 A4 88 9C 01 C5  ....7F2/........
00A0: 40 A5 83 CE 46 6B 6E 83   9E CD 8D DE A8 60 7F 77  @...Fkn......`.w
00B0: 3A 1D F4 E4 FB 26 E9 1F   D8 54 1E 78 0E 7C 15 8C  :....&...T.x....
00C0: 46 54 11 D9 69 F7 FD 65   F5 72 AB 48 75 B3 6E C1  FT..i..e.r.Hu.n.
00D0: 38 80 8C 72 62 CB 8F 55   F0 0C 3B BA 28 3B 74 3B  8..rb..U..;.(;t;
00E0: C7 BB F4 8F 81 FF 16 EA   D6 E1 42 5B A0 EE E6 13  ..........B[....
00F0: 8E 16 A3 0F F0 CE 0B 83   6D 5C E9 36 25 0C DF 8A  ........m\.6%...
0100: 09 76 41 86 2A CB B0 B6   19 58 6D 38 85 AD 94 92  .vA.*....Xm8....
0110: DE B8 44 D3 94 EC BB B7   DE D2 D3 DB 7E 32 03 06  ..D..........2..
0120: C2 CE 8D F5 36 AA DE E6   84 C6 FB F5 6A A9 D6 CF  ....6.......j...
0130: B9 20 0C F0 AB 56 3E 1E   9D 9E B5 BD 24 CD C1 DA  . ...V>.....$...
0140: AB AB B7 71 35 B4                                  ...q5.

Client Principal = hdfs/YYYY@EXAMPLE.COM
Server Principal = krbtgt/EXAMPLE.COM@EXAMPLE.COM
Session Key = EncryptionKey: keyType=17 keyBytes (hex dump)=
0000: 79 80 FD 99 CF 82 F2 76   C3 DE 1C 01 8A 78 EC 89  y......v.....x..


Forwardable Ticket true
Forwarded Ticket false
Proxiable Ticket false
Proxy Ticket false
Postdated Ticket false
Renewable Ticket false
Initial Ticket false
Auth Time = Tue Oct 07 03:46:09 UTC 2014
Start Time = Tue Oct 07 03:46:09 UTC 2014
End Time = Wed Oct 08 03:46:09 UTC 2014
Renew Till = null
Client Addresses  Null
Found ticket for hdfs/YYYY@EXAMPLE.COM to go to krbtgt/EXAMPLE.COM@EXAMPLE.COM expiring on Wed Oct 08 03:46:09 UTC 2014
Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23 1 3.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
>>> KrbKdcReq send: kdc=wckdserver.krbnet UDP:88, timeout=30000, number of retries =3, #bytes=680
>>> KDCCommunication: kdc=wckdserver.krbnet UDP:88, timeout=30000,Attempt =1, #bytes=680
>>> KrbKdcReq send: #bytes read=672
>>> KdcAccessibility: remove wckdserver.krbnet
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 637586272
Created InitSecContextToken:
0000: 01 00 6E 82 02 53 30 82   02 4F A0 03 02 01 05 A1  ..n..S0..O......
0010: 03 02 01 0E A2 07 03 05   00 20 00 00 00 A3 82 01  ......... ......
0020: 62 61 82 01 5E 30 82 01   5A A0 03 02 01 05 A1 0D  ba..^0..Z.......
0030: 1B 0B 45 58 41 4D 50 4C   45 2E 43 4F 4D A2 27 30  ..EXAMPLE.COM.'0
0040: 25 A0 03 02 01 00 A1 1E   30 1C 1B 04 48 54 54 50  %.......0...HTTP
0050: 1B 14 70 69 76 68 64 73   6E 65 2E 6C 6F 63 61 6C  ..pivhdsne.local
0060: 64 6F 6D 61 69 6E A3 82   01 19 30 82 01 15 A0 03  domain....0.....
0070: 02 01 12 A1 03 02 01 01   A2 82 01 07 04 82 01 03  ................
0080: 0C E0 CC 1B 75 0D 75 26   7E FC 33 D6 37 7D EC 09  ....u.u&..3.7...
0090: DA CE BE 9D 48 25 89 E4   9E F3 D0 07 13 CE 3D 96  ....H%........=.
00A0: E8 C2 0F 6E 8E 28 C7 85   3A D4 9D B8 CF 96 DD 3F  ...n.(..:......?
00B0: 42 8F 93 E3 E8 AD DB 84   51 02 E4 C6 BC F2 5D C5  B.......Q.....].
00C0: 17 73 9A B8 EC 10 76 58   6F F5 25 8D 5A C6 48 6B  .s....vXo.%.Z.Hk
00D0: A8 5A 30 83 14 F4 7D E0   90 FF D8 A8 A7 17 51 00  .Z0...........Q.
00E0: 43 0C 1D B6 2A C1 49 66   FA B8 5E 47 67 4B B0 FA  C...*.If..^GgK..
00F0: 10 C2 0D 72 BC 01 C5 D8   FD 5A 1F 8D 53 CA D9 88  ...r.....Z..S...
0100: 6C 00 7C 73 66 88 3A 41   35 B2 45 CF F5 19 8C 28  l..sf.:A5.E....(
0110: 87 C5 FC 4A E5 37 51 BA   8B E0 FF ED 69 03 2D 4E  ...J.7Q.....i.-N
0120: 3A E8 56 0A 84 92 98 95   E7 5B 15 DC 35 11 35 CF  :.V......[..5.5.
0130: F3 3A 99 6F C1 4A F5 49   62 E1 DC 0B FD F2 82 37  .:.o.J.Ib......7
0140: EE BB B8 85 78 50 1B 3A   E3 41 7D 96 2B 63 30 2A  ....xP.:.A..+c0*
0150: 70 C4 C3 D4 EA FF 1F F0   6A 9E BB 60 A2 A4 4D 3D  p.......j..`..M=
0160: 8E 48 57 12 10 A4 96 49   C2 1B AC 30 F7 3E 5A 98  .HW....I...0.>Z.
0170: CB D5 A0 F0 2F FB A4 F3   6D 3C 00 C5 F2 CC 32 BC  ..../...m<....2.
0180: F0 B1 04 A4 81 D3 30 81   D0 A0 03 02 01 11 A2 81  ......0.........
0190: C8 04 81 C5 58 E6 68 49   27 EA D0 A2 9A FB EA 70  ....X.hI'......p
01A0: 61 10 FD 7E 66 B5 EF 02   F0 DA 5E 3E C0 3B 53 72  a...f.....^>.;Sr
01B0: 77 1B 4C 69 7D 49 96 19   58 11 E7 FB DC BE 6B 3D  w.Li.I..X.....k=
01C0: BD 47 24 49 E9 01 7D A3   AD 14 1C 92 94 8A BE 71  .G$I...........q
01D0: AE 60 FB 8B F9 29 26 6E   49 27 8F F9 BA EC ED 77  .`...)&nI'.....w
01E0: 4E F1 E2 E7 9C F6 79 57   9D 95 6C 6D 28 B5 43 F3  N.....yW..lm(.C.
01F0: A2 03 CE DF 3D 0F FE 2E   F8 63 B5 F5 C8 D9 A7 77  ....=....c.....w
0200: 79 53 80 90 DD B9 7C 50   06 F3 84 B5 CE 90 6F 8D  yS.....P......o.
0210: 71 3B EF A0 7A CC 8D 2E   7E 25 DE ED EE F8 1C D2  q;..z....%......
0220: 41 DD BE 05 26 A0 1B 19   BF 58 7B 8E 87 C4 AC EF  A...&....X......
0230: 8D 66 C6 AF C9 42 3B E8   A8 A1 8E 80 D4 3F E7 9D  .f...B;......?..
0240: 58 D8 F2 53 A7 62 C0 70   84 21 5E C2 85 BC 86 70  X..S.b.p.!^....p
0250: ED CC 78 0A 52 D2 F3 EB   B1                       ..x.R....

1566 [main] DEBUG org.apache.hadoop.security.authentication.client.KerberosAuthenticator  - Using fallback authenticator sequence.
Found ticket for hdfs/YYYY@EXAMPLE.COM to go to krbtgt/EXAMPLE.COM@EXAMPLE.COM expiring on Wed Oct 08 03:46:09 UTC 2014
Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23 1 3.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
>>> KrbKdcReq send: kdc=wckdserver.krbnet UDP:88, timeout=30000, number of retries =3, #bytes=680
>>> KDCCommunication: kdc=wckdserver.krbnet UDP:88, timeout=30000,Attempt =1, #bytes=680
>>> KrbKdcReq send: #bytes read=672
>>> KdcAccessibility: remove wckdserver.krbnet
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 464503906
Created InitSecContextToken:
0000: 01 00 6E 82 02 53 30 82   02 4F A0 03 02 01 05 A1  ..n..S0..O......
0010: 03 02 01 0E A2 07 03 05   00 20 00 00 00 A3 82 01  ......... ......
0020: 62 61 82 01 5E 30 82 01   5A A0 03 02 01 05 A1 0D  ba..^0..Z.......
0030: 1B 0B 45 58 41 4D 50 4C   45 2E 43 4F 4D A2 27 30  ..EXAMPLE.COM.'0
0040: 25 A0 03 02 01 00 A1 1E   30 1C 1B 04 48 54 54 50  %.......0...HTTP
0050: 1B 14 70 69 76 68 64 73   6E 65 2E 6C 6F 63 61 6C  ..pivhdsne.local
0060: 64 6F 6D 61 69 6E A3 82   01 19 30 82 01 15 A0 03  domain....0.....
0070: 02 01 12 A1 03 02 01 01   A2 82 01 07 04 82 01 03  ................
0080: EB 9C 24 60 E8 63 A6 EF   E8 9C B7 DB 4B 0B DB A0  ..$`.c......K...
0090: 47 01 B0 C3 DF 50 96 3E   76 D3 36 14 62 CC 14 3D  G....P.>v.6.b..=
00A0: 5D 06 07 2C F8 E7 79 09   A6 73 4A 2C D5 2D 6F 09  ]..,..y..sJ,.-o.
00B0: 08 C2 A0 85 B7 AF D3 3E   BE 83 F5 11 62 21 4F 5C  .......>....b!O\
00C0: 73 09 A2 8F 4A CD 3F 4C   31 46 6E C5 98 C6 5D EF  s...J.?L1Fn...].
00D0: 37 B3 50 C6 D4 18 82 62   65 6A 0D 0C 71 EA 96 16  7.P....bej..q...
00E0: E7 9A E8 4C CA 90 0A 3D   FD 03 C1 ED 3F 85 5B C7  ...L...=....?.[.
00F0: 3A 15 F7 52 95 58 D5 07   3D 0C 93 8B 63 C7 CA 19  :..R.X..=...c...
0100: 29 3D 68 BF 58 B6 5C 48   26 31 06 31 1B A1 AF 3C  )=h.X.\H&1.1...<
0110: FD 98 BE 75 46 42 06 70   C6 74 B2 1C DC CC 13 AE  ...uFB.p.t......
0120: C0 D2 BB 78 EF 36 21 25   7C 06 20 91 3A 59 99 D1  ...x.6!%.. .:Y..
0130: F0 D3 0C 5A 5F E6 27 98   C5 FD 56 98 83 22 94 4E  ...Z_.'...V..".N
0140: 32 1F 5E 55 C5 07 CA 27   AE C2 0B B9 8F 33 06 05  2.^U...'.....3..
0150: 6B 84 9C 16 9D 30 D1 8A   AB F7 79 7A 9F 7C 11 5E  k....0....yz...^
0160: 81 7F 63 FC C2 49 B0 2F   13 2C B9 00 24 A0 44 DD  ..c..I./.,..$.D.
0170: 06 56 3E BF 16 15 14 DD   C2 5B 63 8E DC F8 63 30  .V>......[c...c0
0180: 6A C0 E6 A4 81 D3 30 81   D0 A0 03 02 01 11 A2 81  j.....0.........
0190: C8 04 81 C5 8D 4D DD 54   F7 22 23 7F AC 89 E6 25  .....M.T."#....%
01A0: 1C E0 95 26 DB D0 FD 01   5F 0F C2 51 98 AC 0A FA  ...&...._..Q....
01B0: 74 56 BF 1E C0 A6 B6 1F   B8 7F E7 EC B1 54 1C DD  tV...........T..
01C0: CB BA 33 58 7D 13 86 84   0A 83 2D B1 5D 96 D8 B2  ..3X......-.]...
01D0: AF 01 CA 5D 94 BE 38 E9   D0 75 4E 2E C6 16 4C BD  ...]..8..uN...L.
01E0: C0 45 9C 85 A7 A7 35 6A   81 AC 10 8F EF F9 D1 A5  .E....5j........
01F0: 72 9E 76 93 F5 98 B6 25   E2 17 B8 59 7E 55 26 95  r.v....%...Y.U&.
0200: 36 EF 1D 2E 7F 6B 1C 26   46 BF DB 4D 48 31 86 4B  6....k.&F..MH1.K
0210: 9D CC 67 8B 71 D5 24 8E   C4 42 1D 99 0B C0 7B 6E  ..g.q.$..B.....n
0220: 44 ED 8E B0 1B BA D5 AE   41 E5 9A 2A 30 36 91 38  D.......A..*06.8
0230: 7D BA 47 FC 61 64 53 49   68 75 AC CA 13 DC B6 8B  ..G.adSIhu......
0240: 0E E4 84 3F 61 7B 6E 71   4E 5F B1 56 17 AA 70 61  ...?a.nqN_.V..pa
0250: 0F EF 8C C7 CB 45 BA 01   64                       .....E..d

1898 [main] WARN org.apache.hadoop.security.token.Token  - Cannot find class for token kind WEBHDFS delegation
1899 [main] DEBUG org.apache.hadoop.security.SecurityUtil  - Acquired token Kind: WEBHDFS delegation, Service: xxxx:50070, Ident: 00 04 68 64 66 73 04 68 64 66 73 00 8a 01 48 e8 b9 be 33 8a 01 49 0c c6 42 33 8d 04 d5 6c 8f 99
1904 [main] DEBUG org.apache.hadoop.hdfs.web.WebHdfsFileSystem  - Created new DT for xxxx:50070
1908 [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:hdfs/YYYY@EXAMPLE.COM (auth:KERBEROS) from:org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getHttpUrlConnection(WebHdfsFileSystem.java:456)
1908 [main] DEBUG org.apache.hadoop.hdfs.web.WebHdfsFileSystem  - open URL connection
Directory created...2921 [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:hdfs/YYYY@EXAMPLE.COM (auth:KERBEROS) from:org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getHttpUrlConnection(WebHdfsFileSystem.java:456)*

Answer 1

您必须在系统中设置 krb5.conf 文件的位置，而不是 Hadoop 配置，即替换您的行

conf.set("java.security.krb5.conf",krbPath);

与

System.setProperty( "java.security.krb5.conf", krbPath);

(但是你可能自己想出来了，因为这个问题已经有 5 个月了)