java - GAE 将大文件从 URL 上传到 blobstore

Question

我想将 30MB 到 2GB 的大型视频文件上传到具有外部 URL 来访问该文件的 Blob 存储。

到目前为止，我尝试过类似的方法，但它不适用于大文件，我认为是因为 32MB 的限制:

    //InputStream is = new URL(getUrlThumbnailFullSize(wistiaResponse)).openStream();
    HttpURLConnection con = (HttpURLConnection) new URL("URL of the file here").openConnection();
    con.setConnectTimeout(0);
    con.connect();
    InputStream is = con.getInputStream();
    //create file service to upload the video to the blob store
    FileService fileService = FileServiceFactory.getFileService();
    //create the App Engine File; assign MIME type and name
    AppEngineFile file = fileService.createNewBlobFile("content type here", "file name here");
    //lock when uploading file
    boolean lock = true;
    //open channel
    FileWriteChannel writeChannel = fileService.openWriteChannel(file, lock);
    //upload file
    writeChannel.write(ByteBuffer.wrap(IOUtils.toByteArray(is)));
    writeChannel.closeFinally();
    //release the lock
    lock = false;
    //get blob key
    BlobKey blobKey = FileServiceFactory.getFileService().getBlobKey(file);

但我有以下堆栈跟踪:

SEVERE: com.google.appengine.api.urlfetch.ResponseTooLargeException: The response from url http://embed.wistia.com/deliveries/a0f8dc8ed210b53152af601eb0af40d76cd2929f.bin was too large. at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:57) at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:417) at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:296) at controller.CtrlWistia.download(CtrlWistia.java:149) at controller.CtrlWistia.ajax(CtrlWistia.java:89) at controller.CtrlAjax.main(CtrlAjax.java:66) at vidaao.AjaxServlet.doPost(AjaxServlet.java:37) at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176) at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:123) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:61) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.appengine.tools.development.BackendServersFilter.doFilter(BackendServersFilter.java:97) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:94) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:409) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326)

Answer 1

请使用blobstoreService.createUploadUrl用于大文件上传。

[更新]啊，我想我明白了。目前这是不可能的，因为 URLFetch 服务有 32MB 响应大小限制。作为解决方法，您可以从客户端 JavaScript 中的 URL 下载文件，然后使用 blobstore 服务创建的上传 URL 上传该文件吗？

[更新]好吧，你需要在服务器端实现这一点，从问题中我不清楚这一点。所以请忘记 createUploadUrl。

如果服务器支持 Range header ，您可以多次请求较小的数据 block ，并将它们存储为单独的 Blob 存储对象，以及所需的元数据(文件名、mimetype、Blob 键列表等)。

此外，如果您想将它们合并到一个文件中，您可以使用 Google 云存储。与Resumable upload option ，您可以多次将数据 block 发送到 Google Cloud Storage。