java - SQS 最大消息数

Question

我使用 Java 客户端应用程序查询 SQS 队列中的消息。队列有 12,000 条消息作为测试设置。我正在使用 openJDK 和最新的 aws-java-sdk (software.amazon.awssdk 2.10.62) pom.xml 进一步显示在下面。

我看到的问题是，尽管设置了 ma​​xNumberOfMessages(10) 我只得到 3。我知道这是最大值，不能保证消息数量，但没有动摇返回的消息数。始终是 3。

AWS Documentation: MaxNumberOfMessages The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned). Valid values: 1 to 10. Default: 1. Type: Integer Required: No

Consuming Messages Using Short Polling

When you consume messages from a queue using short polling, Amazon SQS samples a subset of its servers (based on a weighted random distribution) and returns messages from only those servers. Thus, a particular ReceiveMessage request might not return all of your messages. However, if you have fewer than 1,000 messages in your queue, a subsequent request will return your messages. If you keep consuming from your queues, Amazon SQS samples all of its servers, and you receive all of your messages.

因此，我们使用旧版 aws sdk 和新版 aws sdk 在 java 中测试了两个客户端，结果相同。总是只回复 3 条消息。

有趣的是，如果不是在外部(在我强大的桌面上)运行应用程序，而是将其作为 AWS Lambda 运行，您会收到 10 条消息。这个 lambda 测试是由一位同事使用 JavaScript 完成的。

所以问题仍然是为什么我们每个请求只收到 3 条消息，而在 lambda 中似乎可以收到 10 条消息。

鉴于每个请求的成本是基于亚马逊利润的加权随机分布 =))

SQS测试方法:

public void SQStart()
{
    AwsBasicCredentials awsCreds = AwsBasicCredentials.create("accessKeyID", "secretKeyID");
    AwsCredentialsProvider creds = StaticCredentialsProvider.create(awsCreds);
    SqsClient sqs = SqsClient.builder().credentialsProvider(creds).region(Region.EU_WEST_1).build();
    GetQueueUrlRequest getQueueRequest = GetQueueUrlRequest.builder()
            .queueName(QUEUE_NAME)
            .build();
    String queueUrl = sqs.getQueueUrl(getQueueRequest).queueUrl();

    for (int x =1; x < 100; x++) {
        ReceiveMessageRequest receiveMessageRequest = ReceiveMessageRequest.builder()
                .queueUrl(queueUrl)
                .maxNumberOfMessages(10)
                .build();


        List<Message> messages = sqs.receiveMessage(receiveMessageRequest).messages();
        if (messages.size() > 3 ) {
            System.out.println("YEY More than 3 Messages: "+ messages.size());
        }
    }
}

POM.XML:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>groupId</groupId>
    <artifactId>SQSTest</artifactId>
    <version>1.0-SNAPSHOT</version>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>


    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>2.10.62</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>

            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>sqs</artifactId>
        </dependency>

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.9</version>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.4.10</version>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>1.2</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.11.720</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.6.1</version>
        </dependency>
    </dependencies>
</project>

Answer 1

Given there is a cost per request is the weighted random distribution based on amazon profit =))

很明显，您的目标是降低成本，无论是通过向 SQS 发送更少的请求，还是强制 SQS 传递最大数量的可用消息。

正如您在问题中所述，SQS 没有义务传送最大数量的可用消息。不过，我想告诉您一件事，假设您还没有意识到。

<小时/>

长轮询

Developer Guide亚马逊的简单队列服务指出:

The process of consuming messages from a queue depends on whether you use short or long polling. By default, Amazon SQS uses short polling, querying only a subset of its servers (based on a weighted random distribution) to determine whether any messages are available for a response. You can use long polling to reduce your costs while allowing your consumers to receive messages as soon as they arrive in the queue.

您发送到 SQS 的消息可能全部存储在单独的服务器上。正如文档所述，如果您的队列设置为使用短轮询，则只能查询服务器的子集。我的猜测是您在调用 receiveMessage 时不走运，每次只返回 3 。

如果我们在同一文档页面上查看长轮询的好处，就会发现:

Long polling offers the following benefits:

• Eliminate empty responses by allowing Amazon SQS to wait until a message is available in a queue before sending a response. Unless the connection times out, the response to the ReceiveMessage request contains at least one of the available messages, up to the maximum number of messages specified in the ReceiveMessage action.

• Eliminate false empty responses by querying all—rather than a subset of—Amazon SQS servers.

第二点在这里非常重要。即使您没有看到空响应，服务器上可能还存在更多未查询的消息。如果启用长轮询，假设总共有超过 3 台服务器，您有望看到返回的消息量有所增加。

因此，我的建议是在队列上启用长轮询。为此，请参阅 Setting Up Long Polling页。

<小时/>

正如 DevilCode 在他的 comment 中提到的那样下面，他通过使用 FIFO 队列(而不是标准队列)并启用长轮询来解决他的问题。