google-app-engine - 上传文件到谷歌云存储 : 500 Backend Error

Question

我们尝试先将文件上传到 Google Cloud Storage，然后再将它们移动到 BigQuery，但在某些上传过程中我们经常遇到“500 Internal Server Error”或“410 Gone”(下面的原始消息)。

我们使用的是官方 SDK，并添加了指数退避重试，但错误始终存在。您有什么建议吗？

这是我们上传 (scala) 的方式:

val credential = new GoogleCredential().setAccessToken(accessToken)

val requestInitializer = new HttpRequestInitializer() {
    def initialize(request: HttpRequest): Unit = {
        credential.initialize(request)

        // to avoid read timed out exception
        request.setConnectTimeout(200000)
        request.setReadTimeout(200000)

        request.setIOExceptionHandler(new 
            HttpBackOffIOExceptionHandler(new ExponentialBackOff()))
        request.setUnsuccessfulResponseHandler(new 
            HttpBackOffUnsuccessfulResponseHandler(new ExponentialBackOff()))
    }
}

val storage = new Storage.Builder(
    new NetHttpTransport,
    JacksonFactory.getDefaultInstance,
    requestInitializer
).setApplicationName("MyAppHere").build

val objectMetadata = new StorageObject()
    .setBucket(bucketName)
    .setName(distantFileName)

val isc = new InputStreamContent("binary/octet-stream", fis)
val length = isc.getLength
val insertObject = storage.objects().insert(bucketName, objectMetadata, isc)
// For small files, you may wish to call setDirectUploadEnabled(true), to
// reduce the number of HTTP requests made to the server.
if (length > 0 && length <= 2 * 1000 * 1000 /* 2MB */ ) {
    insertObject.getMediaHttpUploader.setDirectUploadEnabled(true)
}
insertObject.execute()

我们的 scala 依赖项:

"com.google.api-client" % "google-api-client" % "1.18.0-rc",
"com.google.api-client" % "google-api-client-jackson2" % "1.18.0-rc",
"com.google.apis" % "google-api-services-bigquery" % "v2-rev142-1.18.0-rc",
"com.google.apis" % "google-api-services-storage" % "v1-rev1-1.18.0-rc",
"com.google.http-client" % "google-http-client" % "1.18.0-rc",
"com.google.oauth-client" % "google-oauth-client" % "1.18.0-rc"

原始 SDK 错误响应:

500 Internal Server Error
    {
        "code" : 500,
        "errors" : [ {
        "domain" : "global",
        "message" : "Backend Error",
        "reason" : "backendError"
    } ],
        "message" : "Backend Error"
    }

410 Gone
    {
        "code" : 500,
        "errors" : [ {
        "domain" : "global",
        "message" : "Backend Error",
        "reason" : "backendError"
    } ],
        "message" : "Backend Error"
    }

Answer 1

每个后端错误都应该用指数重试来处理，因为可能会出现服务问题。

如果错误在假设 10 小时后仍然存在，那么您应该联系支持人员，以便为您提供一对一的帮助来解决您的问题。