java - 在 spring boot 应用程序中上传 1+ GB 文件时出现 "java.lang.OutOfMemoryError: Java heap space"

Question

我完全按照 this 中的说明构建了 spring boot 应用程序上传单个文件的教程。我所做的更改如下所述。

我已将以下属性添加到 application.properties:

spring.http.multipart.max-file-size=2048MB
spring.http.multipart.max-request-size=2048MB

然后我将以下 jvm 参数添加到项目运行配置中，以将 jvm 堆大小限制为 2GB，如 this answer 中所述。 :

-Xmx2048m

接下来我还确保我的 spring boot 应用程序针对 64 位 java 运行 it seems that 32 位 java 需要“连续的”堆空间来预留。

但是当我试图上传大文件时，它仍然给我同样的错误。

首先我尝试上传 20MB 的文件。有效。接下来我继续尝试 ubuntu iso 值(value) 1GB，但它一直给我以下异常:

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_74]
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_74]
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_74]
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[na:1.8.0_74]
    at org.springframework.util.StreamUtils.copy(StreamUtils.java:128) ~[spring-core-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:109) ~[spring-core-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:156) ~[spring-core-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest$StandardMultipartFile.getBytes(StandardMultipartHttpServletRequest.java:291) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at com.digitate.ignio.spring_boot_hdfs_file_upload.controller.UploadController.singleFileUpload(UploadController.java:73) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_74]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_74]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_74]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_74]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:220) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:134) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.5.RELEASE.jar:4.3.5.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.6.jar:8.5.6]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[tomcat-embed-core-8.5.6.jar:8.5.6]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.5.RELEASE.jar:4.3.5.RELEASE]

目前我的机器上安装了 8GB RAM，其中 1.5GB 是免费的。我还尝试在有 2+ GB 可用 RAM 时运行 spring boot 应用程序。

在调试的过程中，我看到上传文件的大小确实是1+GB:

但是在执行第 73 行 file.getBytes() 时，它会抛出上述异常。

我在这里缺少什么？

Answer 1

对于大文件，最好使用 while 循环将 1024 字节读取到缓冲区，然后将缓冲区写入服务器。此代码已通过 5GB 文件上传进行测试。您需要相应地调整应用程序属性，即至少 5GB如果你想尝试一下

spring.http.multipart.max-file-size=6000MB
spring.http.multipart.max-request-size=6000MB

这是使用 spring 框架编写的代码片段。

@Override
public ResponseEntity<?> uploadFile(@RequestParam("file") MultipartFile multipartfile) {
    ResponseEntity<String> response;
    multipartfile.getOriginalFilename();
    byte [] bufferedbytes= new byte[1024];
    File file= new File("/home/david/Music/"+multipartfile.getOriginalFilename());
    FileOutputStream outStream = null;
    int count=0;
    try {
        BufferedInputStream fileInputStream= new BufferedInputStream(multipartfile.getInputStream());
        outStream=new FileOutputStream(file);
        while((count=fileInputStream.read(bufferedbytes))!=-1) {
         outStream.write(bufferedbytes,0,count);

        }
         outStream.close();
    } catch (IOException e) {
         response= new ResponseEntity<String>("File failed to upload"+multipartfile.getOriginalFilename(),HttpStatus.PAYLOAD_TOO_LARGE);

        return response;

    }

     response= new ResponseEntity<String>("File uploaded sucessifully"+multipartfile.getOriginalFilename(),HttpStatus.OK);

    return response;
}