java - 使用curl命令将包含base 64编码文件(大小400Mb)的json上传到spring REST接口(interface)

Question

我正在尝试使用curl将包含大型base64编码文件(大小:400Mb)的json上传到Spring REST接口(interface)。但我收到 OutOfMemory 错误。据我所知，包含该文件的 json 超出了 JVM 的堆大小。什么方法可以很好地处理这么大的文件？任何示例处理代码将不胜感激。

外壳命令:

jsonstring='{"uuid":"111","type":"REPORT","userdata":"test defined 
data","time":3000,"wasted":120,"status":"PASS","message":"demo 
message","report":"'"$(base64 file.zip)"'"}'
curl -s --insecure -H "Content-Type: application/json;" --data "@-" 
https://localhost:443/api/v1/upload <<<"$jsonstring"

Spring 休息接口(interface):

@RequestMapping(value = "/api/v1/upload", method = RequestMethod.POST)
public String uploadFile(@RequestBody final UploadedFile uploadedFile, final 
HttpServletRequest request,
final HttpServletResponse response) {
byte[] decoded = 
DatatypeConverter.parseBase64Binary(uploadedFile.getReport());

错误:

Jul 17 10:03:38  bash[29824]: 2018-07-17 10:03:38.686 ERROR 29829 --- [.0-31443-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause
Jul 17 10:03:38 chld9004852 bash[29824]: java.lang.OutOfMemoryError: Java heap space
Jul 17 10:03:38 chld9004852 bash[29824]: at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68) ~[na:1.8.0_152]
Jul 17 10:03:38 chld9004852 bash[29824]: at java.lang.StringBuilder.<init>(StringBuilder.java:101) ~[na:1.8.0_152]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:394) ~[jackson-core-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2408) ~[jackson-core-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:269) ~[jackson-core-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:35) ~[jackson-databind-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10) ~[jackson-databind-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:127) ~[jackson-databind-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:369) ~[jackson-databind-2.9.6.jar!/:2.9.6]

Answer 1

您的主要问题是 Spring 使用 Jackson 映射器将整个内容加载到内存中。您只能通过使用较低级别的Streaming API来避免这种情况。 JSON 解析器——想想 XML SAX 或 StAX 与 DOM。

这个想法是让你的 Controller 期望一个InputStream而不是一个映射对象并手动解析它，one token at a time 。这样，您就只能创建处理该请求所需的对象或数据结构，而不是加载整个内容以及相当多的开销。

当然，另一个选择是只增加 JVM 的可用堆，并等待下一个大文件使其崩溃。 ;)