java - WebFlux DataBufferLimitException : Part headers exceeded the memory usage limit of 8192 bytes

Question

在以下端点上传 csv 文件和 JSON 对象时

@PostMapping(value = "dataset/rows/query", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
Mono<List<Integer>> getRowsByQuery(@RequestPart("dataset") Mono<FilePart> file, 
                                   @RequestPart("query") QueryDTO query){
    return Mono.just(new ArrayList<>());
}

我收到以下错误:

2020-12-17 12:25:05.142 ERROR 195281 --- [or-http-epoll-3] a.w.r.e.AbstractErrorWebExceptionHandler : [d418565e-17]  500 Server Error for HTTP POST "/dataset/rows/query"

org.springframework.core.io.buffer.DataBufferLimitException: Part headers exceeded the memory usage limit of 8192 bytes
    at org.springframework.http.codec.multipart.MultipartParser$HeadersState.onNext(MultipartParser.java:360) ~[spring-web-5.3.1.jar:5.3.1]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ HTTP POST "/dataset/rows/query" [ExceptionHandlingWebHandler]
Stack trace:
        at org.springframework.http.codec.multipart.MultipartParser$HeadersState.onNext(MultipartParser.java:360) ~[spring-web-5.3.1.jar:5.3.1]
        at org.springframework.http.codec.multipart.MultipartParser.hookOnNext(MultipartParser.java:104) ~[spring-web-5.3.1.jar:5.3.1]
        at org.springframework.http.codec.multipart.MultipartParser.hookOnNext(MultipartParser.java:46) ~[spring-web-5.3.1.jar:5.3.1]
        at reactor.core.publisher.BaseSubscriber.onNext(BaseSubscriber.java:160) ~[reactor-core-3.4.0.jar:3.4.0]
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.0.jar:3.4.0]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:199) ~[reactor-core-3.4.0.jar:3.4.0]
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.0.jar:3.4.0]
        at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:265) ~[reactor-netty-core-1.0.1.jar:1.0.1]
        at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:371) ~[reactor-netty-core-1.0.1.jar:1.0.1]
        at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:381) ~[reactor-netty-core-1.0.1.jar:1.0.1]
        at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:535) ~[reactor-netty-http-1.0.1.jar:1.0.1]
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94) ~[reactor-netty-core-1.0.1.jar:1.0.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:229) ~[reactor-netty-http-1.0.1.jar:1.0.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar:4.1.54.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.54.Final.jar:4.1.54.Final]
        at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]

我尝试自定义 defaultCodex().maxinMemorySize()经过

@Component
public class ServerConfiguration implements WebFluxConfigurer {
    @Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024);
    }
}

和以下 application.yml

server:
  port: ${SERVER_PORT:8080}
  max-http-header-size: 900000000
spring:
  codec:
    max-in-memory-size: 900000000

但它似乎没有任何效果。
而且，奇怪的是，服务器端的错误似乎只发生在从 Angular 调用 API 时，而不是从 postman 调用。
来自 角我有以下标题:

POST /dataset/rows/query HTTP/1.1
Host: localhost:4200
Connection: keep-alive
Content-Length: 496570
Pragma: no-cache
Cache-Control: no-cache
Accept: */*
DNT: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarySBh6gJvnTeDzB43Y
Origin: http://localhost:4200
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: http://localhost:4200/app
Accept-Encoding: gzip, deflate, br
Accept-Language: it-IT,it;q=0.9,en-GB;q=0.8,en;q=0.7,ru-RU;q=0.6,ru;q=0.5,en-US;q=0.4

最后，这是端点对应的 OpenAPI yaml:

openapi: 3.0.1
info:
  title: OpenAPI definition
  version: v0
servers:
- url: http://localhost:8080
  description: Generated server url
paths:
  /dataset/rows/query:
    post:
      tags:
      - dataset-controller
      operationId: getRowsByQuery
      requestBody:
        content:
          multipart/form-data:
            schema:
              type: object
              properties:
                dataset:
                  type: string
                  format: binary
                query:
                  $ref: '#/components/schemas/QueryDTO'
      responses:
        "200":
          description: OK
          content:
            '*/*':
              schema:
                type: array
                items:
                  type: integer
                  format: int32
components:
  schemas:
    PredicateDTO:
      type: object
      properties:
        value:
          type: object
        key:
          type: string
        operator:
          type: string
          enum:
          - EQUAL
          - NOT_EQUAL
          - BELONGING
          - NOT_BELONGING
          - GREATER_THAN
          - GREATER_THAN_EQUAL
          - LESS_THAN
          - LESS_THAN_EQUAL
    QueryDTO:
      type: object
      properties:
        predicates:
          type: object
          additionalProperties:
            $ref: '#/components/schemas/PredicateDTO'

这是我的 dataset和我的 json object .
如何增加部分标题内存使用限制？

Answer 1

你走在正确的轨道上。
根据Spring docs ，您必须提供自己的MultipartHttpMessageReader如果您想自定义任何预定义的限制:

For Multipart parsing the maxInMemorySize property limits the size of non-file parts. For file parts, it determines the threshold at which the part is written to disk. For file parts written to disk, there is an additional maxDiskUsagePerPart property to limit the amount of disk space per part. There is also a maxParts property to limit the overall number of parts in a multipart request. To configure all three in WebFlux, you’ll need to supply a pre-configured instance of MultipartHttpMessageReader to ServerCodecConfigurer.

文档似乎没有提及堆栈跟踪中提到的部分标题大小限制。但是，对我来说，这是我需要从默认值增加以解决问题的唯一属性，但是您可以轻松自定义 DefaultPartHttpMessageReader 的任何其他默认限制.

@Configuration
public class CodecConfig implements WebFluxConfigurer {

    @Override
    public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
        DefaultPartHttpMessageReader partReader = new DefaultPartHttpMessageReader();
        partReader.setMaxHeadersSize(9216); // 9 KiB, default is 8 KiB
        partReader.setEnableLoggingRequestDetails(true);

        MultipartHttpMessageReader multipartReader = new MultipartHttpMessageReader(partReader);
        multipartReader.setEnableLoggingRequestDetails(true);

        configurer.defaultCodecs().multipartReader(multipartReader);
    }
}