python - 在Python中捕获403禁止访问

Question

我具有以下功能来运行BigQuery数据提取(请参见下文)。当我发送太多请求时，我收到错误消息:

google.api_core.exceptions.Forbidden: 403 Exceeded rate limits: too many concurrent queries for this project_and_region. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors

我想知道为什么我的代码无法捕获禁止错误，因为我明确编写了捕获403的函数？

from google.cloud import bigquery
from google.api_core.exceptions import Forbidden, InternalServerError, ServiceUnavailable

def run_job(query, query_params, attempt_nb=1):

    # Configure
    job_config = bigquery.QueryJobConfig()
    job_config.query_parameters = query_params
    query_job = client.query(
        query,
        # Location must match that of the dataset(s) referenced in the query.
        location='US',
        job_config=job_config)  # API request - starts the query

    # Try to run and transform to DataFrame()
    try:
        df = query_job.to_dataframe()
        assert query_job.state == 'DONE'
        return df

    except Forbidden:
        # Exception mapping a ``403 Forbidden`` response."""
        return retry_job(query, query_params, attempt_nb)

    except InternalServerError:
        # Exception mapping a ``500 Internal Server Error`` response. or a :attr:`grpc.StatusCode.INTERNAL` error."""
        return retry_job(query, query_params, attempt_nb)

    except ServiceUnavailable:
        # Exception mapping a ``503 Service Unavailable`` response or a :attr:`grpc.StatusCode.UNAVAILABLE` error."""
        return retry_job(query, query_params, attempt_nb)


def retry_job(query, query_params, attempt_nb):
    # If the error is a rate limit or connection error, wait and
    # try again.
    # 403: Forbidden: Both access denied and rate limits.
    # 408: Timeout
    # 500: Internal Service Error
    # 503: Service Unavailable
    # Old way: if err.resp.status in [403, 408, 500, 503]:
    if attempt_nb < 3:
        print(' ! New BigQuery error. Retrying in 10s')
        time.sleep(10)
        return run_job(query, query_params, attempt_nb + 1)
    else:
        raise Exception('BigQuery error. Failed 3 times', query)

Answer 1

该异常很可能由以下行而不是在try块内引发。如果重试发生并且在递归过程中再次引发异常，则to_dataframe()可能是罪魁祸首。

query_job = client.query(
    query,
    # Location must match that of the dataset(s) referenced in the query.
    location='US',
    job_config=job_config)  # API request - starts the query

查看 the source for this library， query()方法调用 POST创建一个作业，在该作业中，根据Google在 Troubleshooting errors上的页面检查了rateLimitExceededed:

This error returns if your project exceeds the concurrent rate limit or the API requests limit by sending too many requests too quickly.

您可以通过在调用周围添加日志记录和/或将 query()调用放入 try块中以查看是否解决了问题来进一步测试。