apache-camel - Camel-Jetty http代理+大响应数据案例问题: Buffering capacity exceeded

Question

jetty 专家，通过 camel-jetty 作为 http 代理取回大数据时出错，欢迎任何经验/想法/建议，在此先感谢。 RouteBuilder代码:

from("servlet:balancer?matchOnUriPrefix=true")
.to("jetty:http://vm-a38e-8f19.nam.nsroot.net:19999/balancer/?" + "bridgeEndpoint=true"
                                            + "&disableStreamCache=true"
                                            + "&throwExceptionOnFailure=false" 
                                            + "&bufferSize=1024000000"
                                            + "&inboundBufferSize=1024000000"
                                            + "&outboundBufferSize=1024000000"
                                            + "&eagerCheckContentAvailable=true"
                                            + "&httpClient.responseBufferSize=1024000000&responseBufferSize=1024000000&chunked=false")

环境:

<jdk.version>1.8</jdk.version>
<camel.version>2.17.0</camel.version>
<spring.version>4.2.4.RELEASE</spring.version>
<spring.security.version>4.0.4.RELEASE</spring.security.version>

当基于 http 的 rest-service 返回 5000 条 json 数据记录时出错，我尝试了很多方法来调整响应缓冲区大小，不幸的是仍然没有用，它在服务端返回 <1000 条记录时起作用:

    [2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e
DirectByteBufferR@61e4dcc8[p=40262,l=47776,c=1024000000,r=7514]={umn12":"data4782"...4875","\r\n2000\r\n<<<column10":"data48...16":"data4892",>>>data3198","column...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        158] - Parsed false, remaining 0 HttpParser{s=CHUNK,2096474 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1232] - parseNext s=CHUNK DirectByteBuffer@26a1b8da[p=47776,l=47776,c=1024000000,r=0]={umn12":"data4782"...16":"data4892",<<<>>>data3198","column...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        158] - Parsed false, remaining 0 HttpParser{s=CHUNK,2096474 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint                         142] - filled 31280 SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,3/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        113] - Read 31280 bytes DirectByteBuffer@26a1b8da[p=0,l=31280,c=1024000000,r=31280]={<<<"column17":"data4...ata4964","colum>>>54","column16":"d...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} from SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,0/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1232] - parseNext s=CHUNK DirectByteBuffer@26a1b8da[p=0,l=31280,c=1024000000,r=31280]={<<<"column17":"data4...ata4964","colum>>>54","column16":"d...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e
DirectByteBufferR@5991d3e4[p=0,l=678,c=1024000000,r=678]={<<<"column17":"data4...a4894","column8>>>\r\n2000\r\n":"data48...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNK --> CHUNKED_CONTENT 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNKED_CONTENT --> CHUNK_SIZE 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNK_SIZE --> CHUNK 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e
DirectByteBufferR@2b0b1218[p=686,l=8878,c=1024000000,r=8192]={"column17":"data4...column8\r\n2000\r\n<<<":"data4894","col..."column6":"data>>>\r\n2000\r\n4913","co...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange                        228] - Failed HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497: req=false/rsp=true java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1555] - close HttpParser{s=CHUNK,2105344 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNK --> CLOSED 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        514] - Response failure HttpResponse[HTTP/1.1 200 OK]@54e3183e HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497 on HttpChannelOverHTTP@4b2502b4(exchange=HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]: java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        529] - Concurrent failure: response termination skipped, performed by helpers 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange                        259] - Aborted (true) while active HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497: java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        541] - State update failed: TRANSIENT -> CONTENT: FAILURE 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange                        210] - Terminated response for HttpExchange@2e196e3f req=TERMINATED/null@null res=TERMINATED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497, result: Result[HttpRequest[GET /balancer/restservice/largedata HTTP/1.1]@66fc8f65 > HttpResponse[HTTP/1.1 200 OK]@54e3183e] java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        442] - Response complete HttpResponse[HTTP/1.1 200 OK]@54e3183e 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpChannel                         89] - HttpExchange@2e196e3f req=TERMINATED/null@null res=TERMINATED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497 disassociated true from HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.ConnectionPool                      257] - Connection removed HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] - pooled: 0 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint                         75] - oshut SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.http.HttpConnectionOverHTTP         154] - HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] oshut 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractEndPoint                        108] - onClose SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,in,OSHUT,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint                         115] - close SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,in,OSHUT,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.SelectorManager                         756] - Destroyed SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,ISHUT,OSHUT,-,-,5/0,HttpConnectionOverHTTP}{io=0,kio=-1,kro=-1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractConnection                      210] - onClose HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractEndPoint                        108] - onClose SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,ISHUT,OSHUT,-,-,5/0,HttpConnectionOverHTTP}{io=0,kio=-1,kro=-1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.http.HttpConnectionOverHTTP         157] - HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] closed 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        450] - Request/Response failed: Result[HttpRequest[GET /balancer/restservice/largedata HTTP/1.1]@66fc8f65 > HttpResponse[HTTP/1.1 200 OK]@54e3183e] java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.apache.camel.processor.DefaultErrorHandler               153] - Failed delivery for (MessageId: ID-APACCNDLBIW2480-51289-1460370708686-0-7 on ExchangeId: ID-APACCNDLBIW2480-51289-1460370708686-0-8). On delivery attempt: 0 caught: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.apache.camel.processor.Pipeline                          78] - Message exchange has failed: so breaking out of pipeline for exchange: Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8] Exception: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded] 
[2016/04/11 18:32:09 CST] [ERROR] [org.apache.camel.processor.DefaultErrorHandler               204] - Failed delivery for (MessageId: ID-APACCNDLBIW2480-51289-1460370708686-0-7 on ExchangeId: ID-APACCNDLBIW2480-51289-1460370708686-0-8). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]

Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId              ProcessorId          Processor                                                                        Elapsed (ms)
[route1            ] [route1            ] [servlet:/balancer?matchOnUriPrefix=true                                       ] [      3135]
[route1            ] [policy1           ] [policy[null]                                                                  ] [      3136]
[route1            ] [to1               ] [jetty:http://vm-a38e-8f19.nam.nsroot.net:19999/balancer/?bridgeEndpoint=true&d] [      3133]

Stacktrace
--------------------------------------------------------------------------------------------------------------------------------------- 
org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]
    at org.apache.camel.component.jetty9.JettyContentExchange9.doTaskCompleted(JettyContentExchange9.java:155)
    at org.apache.camel.component.jetty9.JettyContentExchange9$2.onComplete(JettyContentExchange9.java:221)
    at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:193)
    at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:185)
    at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:453)
    at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:434)
    at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:360)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:254)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:38)
    at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1499)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1272)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:156)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:117)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
    at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:89)
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:122)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Buffering capacity exceeded
    at org.eclipse.jetty.client.util.BufferingResponseListener.onContent(BufferingResponseListener.java:114)
    at org.eclipse.jetty.client.api.Response$Listener$Adapter.onContent(Response.java:245)
    at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:124)
    at org.eclipse.jetty.client.ResponseNotifier.access$100(ResponseNotifier.java:35)
    at org.eclipse.jetty.client.ResponseNotifier$ContentCallback.process(ResponseNotifier.java:274)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
    at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:117)
    at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:322)
    ... 13 more
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        158] - Parsed true, remaining 22402 HttpParser{s=CLOSED,2105344 of -1} 

Answer 1

原因是:camel-jetty9-2.17.0.jar有限制，org.apache.camel.component.jetty9.JettyContentExchange9#send

BufferingResponseListener responseListener = new BufferingResponseListener() {

    @Override
    public void onComplete(Result result) {
        if (result.isFailed()) {
            doTaskCompleted(result.getFailure());
        } else {
            onResponseComplete(result, getContent(), getMediaType());
        }
    }
};

在 BufferingResponseListener 中，默认的 maxLength 是 2MB(2 * 1024 * 1024)。而单参数构造函数:BufferingResponseListener(int maxLength) 是调整 maxLength 的唯一方法。但是 camel 使用默认构造函数，所以所有大于 2MB 的响应都会出现这样的错误。