gpt4 book ai didi

How to specify spot instance type for the `aws emr create-cluster` command?(如何为`aws emr create-cluster`命令指定Spot实例类型?)

转载 作者:bug小助手 更新时间:2023-10-25 21:57:26 27 4
gpt4 key购买 nike



I created a EMR cluster by selecting "Use Spot purchasing option" for the Task nodes:

我通过为任务节点选择“使用现货购买选项”创建了一个EMR集群:


enter image description here


Then I copied the CLI command by clicking "View command for cloning cluster":

然后我复制了CLI命令,点击[查看克隆集群命令]:


enter image description here


However, this command does not have any info for using the spot instance. If I create EMR cluster by this command, it will create Task nodes using on-demand instances:

但是,此命令没有任何有关使用Spot实例的信息。如果我使用此命令创建EMR集群,它将使用按需实例创建任务节点:


aws emr create-cluster \
--name hm-amazon-emr-cluster \
--release-label emr-6.12.0 \
--applications=Name Trino \
--instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","Name":"Primary","InstanceType":"r5.xlarge","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}]}},{"InstanceCount":1,"InstanceGroupType":"CORE","Name":"Core","InstanceType":"r5.xlarge","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}],"EbsOptimized":true}},{"InstanceCount":15,"InstanceGroupType":"TASK","Name":"Task","InstanceType":"r5.xlarge","EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"VolumeType":"gp2","SizeInGB":32},"VolumesPerInstance":2}]}}]' \
--log-uri s3n://hongbomiao-bucket/amazon-emr/logs/ \
--service-role arn:aws:iam::8888888888888:role/service-role/AmazonEMR-ServiceRole-xxxxxxxxxxxxxxx \
--scale-down-behavior TERMINATE_AT_TASK_COMPLETION \
--auto-termination-policy '{"IdleTimeout":7200}' \
--step-concurrency-level 10 \
--region us-west-2

Below is prettified version of --instance-groups:

下面是--Instance-Groups的美化版本:


[
{
"InstanceCount": 1,
"InstanceGroupType": "MASTER",
"Name": "Primary",
"InstanceType": "r5.xlarge",
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"VolumeType": "gp2",
"SizeInGB": 32
},
"VolumesPerInstance": 2
}
]
}
},
{
"InstanceCount": 1,
"InstanceGroupType": "CORE",
"Name": "Core",
"InstanceType": "r5.xlarge",
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"VolumeType": "gp2",
"SizeInGB": 32
},
"VolumesPerInstance": 2
}
],
"EbsOptimized": true
}
},
{
"InstanceCount": 15,
"InstanceGroupType": "TASK",
"Name": "Task",
"InstanceType": "r5.xlarge",
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"VolumeType": "gp2",
"SizeInGB": 32
},
"VolumesPerInstance": 2
}
]
}
}
]

I tried to create two clusters with "Use Spot purchasing option" for the Task nodes selected and unselected then copy out the command by clicking "View command for cloning cluster" button. And I got same contents.

我试着为选中和取消选中的任务节点创建两个集群,并使用“使用现货购买选项”,然后通过点击“查看克隆集群的命令”按钮复制出命令。我也得到了同样的内容。


How to specify spot instance type for the aws emr create-cluster command? Thanks!

如何为AWS EMR CREATE-CLUSTER命令指定Spot实例类型?谢谢!


更多回答
优秀答案推荐

Add the BidPrice property to the TASK instance group configuration. That is the maximum bidding price for a Spot Instance: it implicitly shows that you would like to use Spot Instances. Please see the documentation for command create-cluster for more information.

将BidPrice属性添加到任务实例组配置。这是Spot实例的最高竞价:它隐含地表明您希望使用Spot实例。有关详细信息,请参阅命令CREATE-CLUSTER的文档。



[BidPrice] - If specified, indicates that the instance group uses Spot Instances. This is the maximum price you are willing to pay for Spot Instances. Specify OnDemandPrice to set the amount equal to the On-Demand price, or specify an amount in USD.



Additionally, it seems like there is a bug in the export to CLI function. You can send a bug report in the AWS Management Console: click on the question mark icon on the page's header -> Send Feedback -> Set the type to "Report an issue".

此外,似乎在导出到CLI功能中存在错误。您可以在AWS管理控制台中发送错误报告:点击页面标题上的问号图标->发送反馈->将类型设置为“报告问题”。


更多回答

Thanks Leonid ! Just add some extra info here for future. There is also a BidPriceAsPercentageOfOnDemandPrice in the doc when using --instance-fleets I found which is useful so that no need check actual price.

谢谢莱昂尼德!只需在此处添加一些额外信息,以备将来使用。在使用我发现的--实例车队时,文档中还有一个BidPriceAsPercentageOfOnDemandPrice,这很有用,这样就不需要检查实际价格了。

27 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com