amazon-web-services - 如何在 intelliJ 中将 AWS 凭证 key 添加到 Hadoop 配置

Question

我试图通过 IntelliJ 与 AWS 对 Hadoop 执行操作，但遇到了 Hadoop 问题 - 无法访问我的凭证 key 。
我的 IntelliJ 帐户已经连接到 AWS，我可以使用这些服务，但是使用 Hadoop，我遇到了以下问题。

这是代码 -

public static void main(String[] args) throws Exception {

    String s3Path = "adoop-map-reduce-collocation-extraction/corpus-cardinality-by-decade";
    String output = "s3a://adoop-map-reduce-collocation-extraction/statistic-extraction";

    Configuration conf = new Configuration();
    conf.set("CorpusCardinalityFilePath", s3Path);

    Job job = Job.getInstance(conf, "Appearances Calculation");
    job.setJarByClass(AppearancesCalculation.class);
    job.setMapperClass(AC_Mapper.class);
    job.setCombinerClass(AC_Combiner.class);
    job.setReducerClass(AC_Reducer.class);
    job.setOutputKeyClass(nGramKey.class);
    job.setOutputValueClass(LongWritable.class);
    job.setMapOutputValueClass(LongWritable.class);
    job.setMapOutputKeyClass(nGramKey.class);
    job.setOutputValueClass(nGramValue.class);

    FileInputFormat.addInputPath(job, new Path(inPath));
    FileOutputFormat.setOutputPath(job, new Path(output));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
}

和所有依赖项 -


3.0.0

<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.11.531</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>amazon-kinesis-client</artifactId>
        <version>1.2.1</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.jetbrains</groupId>
        <artifactId>annotations-java5</artifactId>
        <version>17.0.0</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>${hadoop.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-aws</artifactId>
        <version>${hadoop.version}</version>
    </dependency>

</dependencies>

这是整个错误 -

Exception in thread "main" java.lang.RuntimeException: java.io.InterruptedIOException: doesBucketExist on adoop-map-reduce-collocation-extraction: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(FileOutputFormat.java:182) at AppearancesCalculation.main(AppearancesCalculation.java:153) Caused by: java.io.InterruptedIOException: doesBucketExist on adoop-map-reduce-collocation-extraction: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:141) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:332) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:275) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(FileOutputFormat.java:178) ... 1 more Caused by: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:151) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1225) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:801) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:751) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4914) at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5693) at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:5666) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4898) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4860) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1394) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1333) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:320) ... 9 more Caused by: com.amazonaws.SdkClientException: Unable to load credentials from service endpoint at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183) at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162) at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:172) at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:129) ... 26 more Caused by: java.net.SocketTimeoutException: connect timed out at java.base/java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) at java.base/java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:400) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:243) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:225) at java.base/java.net.PlainSocketImpl.connect(PlainSocketImpl.java:148) at java.base/java.net.Socket.connect(Socket.java:591) at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) at java.base/sun.net.www.http.HttpClient.(HttpClient.java:242) at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341) at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362) at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1224) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1203) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1057) at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:991) at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:54) at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:113) at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:82) at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:197) at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:122) ... 29 more Disconnected from the target VM, address: '127.0.0.1:60790', transport: 'socket'

Process finished with exit code 1

Answer 1

查看错误我会说您需要使用属性将文件 core-site.xml 添加到资源中 fs.s3a.access.key , fs.s3a.secret.key和 fs.s3a.endpoint