Azure - 403 错误 - 将大量存储帐户 blob 复制到即用即付帐户上的另一个存储帐户

Question

我得到了

403 Forbidden

将大量 block blob 从一个存储帐户复制到另一个存储帐户(在不同区域作为备份)时出错。复制 100,000 多个内容后，我收到 403 Forbidden 错误。

我看到过有关配额的答案，但我相信这是针对免费帐户的。我有一个拥有 578,000 个文件的客户端，我将其从本地移动到 Azure，工作正常，但我无法将副本复制到我设置为用作备份的另一个存储帐户(主要是在删除的情况下)。

我正在使用 StartCopyAsync，然后检查 Copystate 状态以验证复制是否成功，并在我的代码中重试，但 StartCopyAsync 似乎失败。

复制工作正常，直到我复制了超过 100,000 个文件，然后出现错误。我不确定是什么原因造成的，因为相同的代码首先适用于这么多的 blob。我添加了一个日志文件，告诉我哪个文件失败，我可以在 Azure 资源管理器中打开该文件。

我可以发布代码，但现在，我想知道我是否遇到了某种我不知道的报价/带宽问题。

    namespace BackupCloudContainers
    {

    class Program
    {
        static string privateconnectionstring = ConfigurationManager.AppSettings["StorageConnectionString"];
        static string privatebackupconnectionstring = ConfigurationManager.AppSettings["BackupStorageConnectionString"];
        static DateTime testdate = new DateTime(2017, 8, 28, 0, 0, 0);
        static string destContainerName = "";
        static void Main(string[] args)
        {
            try
            {
                //Console.WriteLine("Starting Backup at " + DateTime.Now.ToString("hh:mm:ss.ffff"));
                Log("Starting Incremental Backup (everything since " + testdate.ToString("f") + ") at " + DateTime.Now.ToString("hh:mm:ss.ffff"));
                Backup().GetAwaiter().GetResult();
                // Console.WriteLine("Backup Created as " + destContainerName);
                Log("Backup Created as " + destContainerName);
                //Console.WriteLine("Backup ended at " + DateTime.Now.ToString("hh:mm:ss.ffff"));
                Log("Backup ended at " + DateTime.Now.ToString("hh:mm:ss.ffff"));
                Console.WriteLine("\n\nPress Enter to close. ");
                Console.ReadLine();
            }
            catch (Exception e)
            {
                //Console.WriteLine("Exception - " + e.Message);
                Log("Exception - " + e.Message);
                if (e.InnerException != null)
                {
                    //Console.WriteLine("Inner Exception - " + e.InnerException.Message);
                    Log("Inner Exception - " + e.InnerException.Message);
                }
            }
        }
        static async Task Backup()
        {

            CloudStorageAccount _storageAccount = CloudStorageAccount.Parse(privateconnectionstring);
            CloudStorageAccount _storageBackupAccount = CloudStorageAccount.Parse(privatebackupconnectionstring);

            CloudBlobClient blobClient = _storageAccount.CreateCloudBlobClient();
            CloudBlobClient blobBackupClient = _storageBackupAccount.CreateCloudBlobClient();

            foreach (var srcContainer in blobClient.ListContainers())
            {
                // skip any containers with a backup name
                if (srcContainer.Name.IndexOf("-backup-") > -1)
                {
                    continue;
                }
                var backupTimeInTicks = DateTime.UtcNow.Ticks;
                //var destContainerName = srcContainer.Name + "-" + backupTimeInTicks;
                var backupDateTime = DateTime.UtcNow.ToString("yyyyMMdd-hhmmssfff");
                destContainerName = srcContainer.Name + "-backup-" + backupDateTime;

                var destContainer = blobBackupClient.GetContainerReference(destContainerName);
//                var destContainer = blobClient.GetContainerReference(destContainerName);

                // assume it does not exist already,
                // as that wouldn't make sense.
                await destContainer.CreateAsync();

                // ensure that the container is not accessible
                // to the outside world,
                // as we want all the backups to be internal.
                BlobContainerPermissions destContainerPermissions = destContainer.GetPermissions();
                if (destContainerPermissions.PublicAccess != BlobContainerPublicAccessType.Off)
                {
                    destContainerPermissions.PublicAccess = BlobContainerPublicAccessType.Off;
                    await destContainer.SetPermissionsAsync(destContainerPermissions);
                }

                // copy src container to dest container,
                // note that this is synchronous operation in reality,
                // as I want to only add real metadata to container
                // once all the blobs have been copied successfully.
                await CopyContainers(srcContainer, destContainer);
                await EnsureCopySucceeded(destContainer);

                // ensure we have some metadata for the container
                // as this will helps us to delete older containers
                // on a later date.
                await destContainer.FetchAttributesAsync();

                var destContainerMetadata = destContainer.Metadata;
                if (!destContainerMetadata.ContainsKey("BackupOf"))
                {
                    string cname = srcContainer.Name.ToLowerInvariant();
                    destContainerMetadata.Add("BackupOf", cname);
                    destContainerMetadata.Add("CreatedAt", backupTimeInTicks.ToString());
                    destContainerMetadata.Add("CreatedDate", backupDateTime);
                    await destContainer.SetMetadataAsync();
                    //destContainer.SetMetadata();
                }
            }

            // let's purge the older containers,
            // if we already have multiple newer backups of them.
            // why keep them around.
            // just asking for trouble.
            //var blobGroupedContainers = blobBackupClient.ListContainers()
            //    .Where(container => container.Metadata.ContainsKey("Backup-Of"))
            //    .Select(container => new
            //    {
            //        Container = container,
            //        BackupOf = container.Metadata["Backup-Of"],
            //        CreatedAt = new DateTime(long.Parse(container.Metadata["Created-At"]))
            //    }).GroupBy(arg => arg.BackupOf);

            var blobGroupedContainers = blobClient.ListContainers()
                .Where(container => container.Metadata.ContainsKey("BackupOf"))
                .Select(container => new
                {
                    Container = container,
                    BackupOf = container.Metadata["BackupOf"],
                    CreatedAt = new DateTime(long.Parse(container.Metadata["CreatedAt"]))
                }).GroupBy(arg => arg.BackupOf);

            // Remove the Delete for now
      //      foreach (var blobGroupedContainer in blobGroupedContainers)
      //      {
      //          var containersToDelete = blobGroupedContainer.Select(arg => new
      //          {
      //              Container = arg.Container,
      //              CreatedAt = new DateTime(arg.CreatedAt.Year, arg.CreatedAt.Month, arg.CreatedAt.Day)
      //          })
      //              .GroupBy(arg => arg.CreatedAt)
      //              .OrderByDescending(grouping => grouping.Key)
      //              .Skip(7) /* skip last 7 days worth of data */
      //              .SelectMany(grouping => grouping)
      //              .Select(arg => arg.Container);

      //// Remove the Delete for now
      //          //foreach (var containerToDelete in containersToDelete)
      //          //{
      //          //    await containerToDelete.DeleteIfExistsAsync();
      //          //}
      //      }
        }

        static async Task EnsureCopySucceeded(CloudBlobContainer destContainer)
        {
            bool pendingCopy = true;
            var retryCountLookup = new Dictionary<string, int>();

            while (pendingCopy)
            {
                pendingCopy = false;

                var destBlobList = destContainer.ListBlobs(null, true, BlobListingDetails.Copy);

                foreach (var dest in destBlobList)
                {
                    var destBlob = dest as CloudBlob;
                    if (destBlob == null)
                    {
                        continue;
                    }

                    var blobIdentifier = destBlob.Name;

                    if (destBlob.CopyState.Status == CopyStatus.Aborted ||
                        destBlob.CopyState.Status == CopyStatus.Failed)
                    {
                        int retryCount;
                        if (retryCountLookup.TryGetValue(blobIdentifier, out retryCount))
                        {
                            if (retryCount > 4)
                            {
                                throw new Exception("[CRITICAL] Failed to copy '"
                                                        + destBlob.CopyState.Source.AbsolutePath + "' to '"
                                                        + destBlob.StorageUri + "' due to reason of: " +
                                                        destBlob.CopyState.StatusDescription);
                            }

                            retryCountLookup[blobIdentifier] = retryCount + 1;
                        }
                        else
                        {
                            retryCountLookup[blobIdentifier] = 1;
                        }

                        pendingCopy = true;

                        // restart the copy process for src and dest blobs.
                        // note we also have retry count protection,
                        // so if any of the blobs fail too much,
                        // we'll give up.
                        await destBlob.StartCopyAsync(destBlob.CopyState.Source);
                    }
                    else if (destBlob.CopyState.Status == CopyStatus.Pending)
                    {
                        pendingCopy = true;
                    }
                }

                Thread.Sleep(1000);
            }
        }

        static async Task CopyContainers(
                CloudBlobContainer srcContainer,
                CloudBlobContainer destContainer)
        {
            // get the SAS token to use for all blobs
            string blobToken = srcContainer.GetSharedAccessSignature(new SharedAccessBlobPolicy()
            {
                Permissions = SharedAccessBlobPermissions.Read,
                SharedAccessStartTime = DateTime.Now.AddMinutes(-5),
                SharedAccessExpiryTime = DateTime.Now.AddHours(3)
            });
            int ii = 0;
            int cntr = 0;
            int waitcntr = 0;
            string sourceuri = "";
            int datecntr = 0;
            try
            {

                //Console.WriteLine("  container contains " + srcContainer.ListBlobs(null, true).Count().ToString());
                Log("  container contains " + srcContainer.ListBlobs(null, true).Count().ToString());
                foreach (var srcBlob in srcContainer.ListBlobs(null, true))
                {
                    ii++;

                    //THIS IS FOR COUNTING Blobs that would be on the Incremental Backup
                    CloudBlob blob = (CloudBlob)srcBlob;
                    if (blob.Properties.LastModified > testdate)
                    {
                        datecntr++;
                    }
                    else
                    {
                        // We are only doing an Incremental Backup this time - so skip all other files 
                        continue;
                    }


                    //if (ii > 2000)
                    //{
                    //    //Console.WriteLine("   test run ended ");
                    //    Log("   test run ended ");
                    //    break;
                    //}


                    cntr++;
                    if (cntr > 999)
                    {
                        //Console.WriteLine("    " + ii.ToString() + " processed at " + DateTime.Now.ToString("hh:mm:ss"));
                        Log("    " + ii.ToString() + " processed at " + DateTime.Now.ToString("hh:mm:ss"));

                        //Log("   EnsureCopySucceeded - finished at " + DateTime.Now.ToString("hh:mm:ss"));
                        //await EnsureCopySucceeded(destContainer);
                        //Log("   EnsureCopySucceeded - finished at " + DateTime.Now.ToString("hh:mm:ss"));

                        cntr = 0;

                    }

                    waitcntr++;
                    if (waitcntr > 29999)
                    {
                        Log("   EnsureCopySucceeded (ii=" + ii.ToString() + "- started at " + DateTime.Now.ToString("hh:mm:ss"));
                        await EnsureCopySucceeded(destContainer);
                        Log("   EnsureCopySucceeded - finished at " + DateTime.Now.ToString("hh:mm:ss"));
                        waitcntr = 0;
                    }


                    var srcCloudBlob = srcBlob as CloudBlob;
                    if (srcCloudBlob == null)
                    {
                        continue;
                    }

                    CloudBlob destCloudBlob;

                    if (srcCloudBlob.Properties.BlobType == BlobType.BlockBlob)
                    {
                        destCloudBlob = destContainer.GetBlockBlobReference(srcCloudBlob.Name);
                    }
                    else
                    {
                        destCloudBlob = destContainer.GetPageBlobReference(srcCloudBlob.Name);
                    }
                    sourceuri = srcCloudBlob.Uri.AbsoluteUri + blobToken;

                    try
                    {
                        await destCloudBlob.StartCopyAsync(new Uri(srcCloudBlob.Uri.AbsoluteUri + blobToken));
                    }
                    catch (Exception e)
                    {
                        Log("Error at item " + ii.ToString() + "      Source = " + sourceuri + "      Message = " + e.Message + "      Time = " + DateTime.Now.ToString("F") + "\r\n");
                    }
                }
                Log("Total Items checked = " + ii.ToString() + "    backed up files = " + datecntr.ToString());
                Log("TestDate = " + testdate.ToString("F") + "     datecntr = " + datecntr.ToString());
            }
            catch (Exception e)
            {
                Log("Error at item " + ii.ToString());
                Log("      Source = " + sourceuri);
                Log("      Message = " + e.Message);
                Log("      Time = " + DateTime.Now.ToString("F") + "\r\n");
                //throw e; 
            }
        }


        static void Log(string logdata)
        {
            Console.WriteLine(logdata);
            File.AppendAllText("c:\\junk\\dwlog.txt", logdata + "\r\n");
        }
    }
    }

Answer 1

您提到您的代码在 3 小时后开始失败。好吧，以下几行代码是罪魁祸首:

    string blobToken = srcContainer.GetSharedAccessSignature(new SharedAccessBlobPolicy()
    {
        Permissions = SharedAccessBlobPermissions.Read,
        SharedAccessStartTime = DateTime.Now.AddMinutes(-5),
        SharedAccessExpiryTime = DateTime.Now.AddHours(3)
    });

如果您注意到，您正在创建一个有效期为 3 小时的共享访问签名 (SAS)，并且您正在将此 SAS 用于所有 Blob。只要 SAS 有效(即未过期)，您的代码就可以使用。一旦 SAS 过期，由于现在 SAS token 无权执行该操作，您将开始收到 403(未授权) 错误。

我的建议是创建一个有效期较长的 SAS token 。我建议使用有效期为 15 天的 SAS token ，因为这是 Azure 存储尝试将 blob 从一个帐户复制到另一个帐户的最长时间。