python - 在 SageMaker 中调用 fit 方法时如何防止出现 NoCredentialsError？

Question

我是 Python SageMaker 的新手(我的背景是 C#)。目前，我遇到了一个问题，因为最后一个方法调用(我的意思是 fit 方法)导致“NoCredentialsError”。我不明白这个。 AWS 凭证已设置，我确实使用它们与 AWS 进行通信，例如与 S3 进行通信。我怎样才能防止这个错误？

import io
import os
import gzip
import pickle
import urllib.request
import boto3
import sagemaker
import sagemaker.amazon.common as smac

DOWNLOADED_FILENAME = 'C:/Users/Daan/PycharmProjects/downloads/mnist.pkl.gz'
if not os.path.exists(DOWNLOADED_FILENAME):
    urllib.request.urlretrieve("http://deeplearning.net/data/mnist/mnist.pkl.gz", DOWNLOADED_FILENAME)

with gzip.open(DOWNLOADED_FILENAME, 'rb') as f:
    train_set, valid_set, test_set = pickle.load(f, encoding='latin1')
vectors = train_set[0].T
buf = io.BytesIO()
smac.write_numpy_to_dense_tensor(buf, vectors)
buf.seek(0)
key = 'recordio-pb-data'
bucket_name = 'SOMEKINDOFBUCKETNAME'
prefix = 'sagemaker/pca'
path = os.path.join(prefix, 'train', key)
print(path)

session = boto3.session.Session(aws_access_key_id='SECRET',aws_secret_access_key='SECRET',region_name='eu-west-1')
client = boto3.client('sagemaker',region_name='eu-west-1',aws_access_key_id='SECRET',aws_secret_access_key='SECRET')
region='eu-west-1'
sagemakerSession= sagemaker.Session(sagemaker_client=client,boto_session=session)
s3_resource=session.resource('s3')
bucket = s3_resource.Bucket(bucket_name)
current_bucket = bucket.Object(path)

train_data = 's3://{}/{}/train/{}'.format(bucket_name, prefix, key)
print('uploading training data location: {}'.format(train_data))
current_bucket.upload_fileobj(buf)

output_location = 's3://{}/{}/output'.format('SOMEBUCKETNAME', prefix)
print('training artifacts will be uploaded to: {}'.format(output_location))

region='eu-west-1'

containers = {'us-west-2': 'SOMELOCATION',
              'us-east-1': 'SOMELOCATION',
              'us-east-2': 'SOMELOCATION',
              'eu-west-1': 'SOMELOCATION'}
container = containers[region]

role='AmazonSageMaker-ExecutionRole-SOMEVALUE'
pca = sagemaker.estimator.Estimator(container,
                                    role,
                                    train_instance_count=1,
                                    train_instance_type='ml.c4.xlarge',
                                    output_path=output_location,
                                    sagemaker_session=sagemakerSession)


pca.set_hyperparameters(feature_dim=50000,
                        num_components=10,
                        subtract_mean=True,
                        algorithm_mode='randomized',
                        mini_batch_size=200)

pca.fit(inputs=train_data)

print('END')

Answer 1

我不确定您是否屏蔽了实际的访问 ID 和 key ，或者这就是您正在运行的内容。

session = boto3.session.Session(aws_access_key_id='SECRET',aws_secret_access_key='SECRET',region_name='eu-west-1')
client = boto3.client('sagemaker',region_name='eu-west-1',aws_access_key_id='SECRET',aws_secret_access_key='SECRET')

我希望您在上述代码行中提供实际的 aws_access_key_id 和 aws_secret_access_key。

在代码中指定相同而不是硬编码的另一种方法是在您的配置文件目录中创建一个凭据文件，即

在 Mac 中 ~/.aws/

在 Windows 中“%UserProfile%\.aws”

该文件是一个纯文本文件，名称为“credentials”(不带引号)。文件包含

[default]
aws_access_key_id=XXXXXXXXXXXXXX
aws_secret_access_key=YYYYYYYYYYYYYYYYYYYYYYYYYYY

AWS CLI 将从上述位置获取它并使用它。您还可以使用非默认配置文件并通过

传递配置文件

os.environ["AWS_PROFILE"] = "profile-name"

希望这有帮助。