python - 由于 AKI DirName 扩展 : Python [SSL: CERTIFICATE_VERIFY_FAILED] . 在 openSSL 中找到根本原因 ..:无法获取本地颁发者证书

Question

简而言之
我得到:[SSL: CERTIFICATE_VERIFY_FAILED] 证书验证失败:无法获取本地颁发者证书 (_ssl.c:1129)
什么是比 [SSL: CERTIFICATE_VERIFY_FAILED] 证书验证失败更糟糕的错误:证书链中的自签名证书 (_ssl.c:1129)
而且我怀疑是python还是我自己的代码，但是纯openSSL也会出现同样的错误:

openssl s_client -connect my-domain.com:443 -CAfile root.pem -verify 2

   ......

    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
......

如果 openSSL 不起作用，python 也将不起作用。确保这是针对 TLS 握手时的任何错误。 ;-)
新旧证书对比，终于在两个新增的 Authority Key Identifier 中找到了根本原因条目 DirName和 serial .他们以前不存在的地方:

openssl x509 -in my-domain.pem -text

The only diff and root cause was Authority Key Identifier with two new entries:

X509v3 Authority Key Identifier:
    keyid:1D:E8:38:95:85:65:A2:D9:44:99:96:30:D1:81:D5:5B:F7:38:CC:8C
    DirName:/C=DE/O=My Company/OU=My OU/CN=My Sub-CA
    serial:02

新的也是最后一个问题:为什么 openSSL 不适用于 DirName和 serial在 AKI 内?

我保留了所有最初的东西，因为这可能有助于其他人构建灵活的 python 代码.....
现在，我下面的所有方法都可以正常工作，正如预期的那样。
如果 CA 不在信任库中，我会收到预期的 python 错误:
[SSL: CERTIFICATE_VERIFY_FAILED] 证书验证失败:证书链中的自签名证书 (_ssl.c:1129)
在上下文、 session 或请求级别设置信任库，取你想要的，它对我有用。
过去2年的情况
在过去(2 年前)我将我的私有(private) CA(根和子 CA)导入 Win 10 本地计算机信任库
并使用了这个 code snippet创建我自己的 urllib3 上下文以使用 context.load_default_certs() 加载 Windows 商店
这一直有效，直到本周我不得不更新我的 CA 和服务器证书。 :-(
我像往常一样删除了旧的私有(private) CA 包并将新 CA 导入 Windows 本地计算机信任库和 firefox 信任库。
所有浏览器都可以在新的信任库中正常工作!
更新我的私有(private) CA 后的新问题
使用 python 请求，我现在再次遇到这些烦人的错误:
步骤代码中的 SSLError - 消息:HTTPSConnectionPool(host='my-domain.com', port=443): Max retries exceeded with url:/(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] 证书验证失败:无法获得本地颁发者证书 (_ssl.c:1129)')))
我的代码方法
我当前的代码尝试了不同的方法来解决这个问题，但没有任何效果。
我确定(或希望)错误在我身边，但我不再看到它了；-)

import certifi
import json
import os
import requests

    class MyHttpAdapter(HTTPAdapter):
        def init_poolmanager(self, *args, **kwargs):
            # Google: "SSL failure on Windows using python requests" -- usage of Windows Truststore
            # https://stackoverflow.com/questions/42981429/ssl-failure-on-windows-using-python-requests
            context = create_urllib3_context()

我尝试了这 3 个设置来加载不同的 ca 包(每次尝试只有一个代码行处于事件状态)

            ## --------------------
            # variant 1: usage of Windows Truststore where I imported my root-ca and sub-ca into local computer storage
            # load_default_certs() --> this loads the OS defaults of Windows Truststore!!!!
            context.load_default_certs()
            
            # variant 2: my special ca bundle contains only my private CA only with 3 entries: root-ca, sub-ca and server cert
            # load_verify_locations() --> this loads just a specific CA bundle !!!!
            context.load_verify_locations(cafile="C:/Users/...../my_ca_bundle.pem")
            
            # variant 3: I added my private CA with root-ca, sub-ca and server cert to certifi cacert.pem file as a hack
            # certifi.where(): C:\Users\........\Python39\Lib\site-packages\certifi\cacert.pem
            # load_verify_locations() --> this loads just a specific CA bundle !!!!
            context.load_verify_locations(cafile=certifi.where())
            ## --------------------

为了验证我的私有(private) CA 是否在所选存储中，我执行 assert在我的 CA 的 CommonName 上，因此如果找不到它会很快失败。如果没问题，稍后会将上下文添加到 session.mount 中:

            # get and print all CAs from loaded list to verify that my CA is within the list
            # notice the server certificate of pem file is not loaded into the list, don't no why, maybe because it's no CA
            json_ca_certs = context.get_ca_certs()
            print("All current context CA certs: {}".format(json.dumps(json_ca_certs, indent=1, sort_keys=True)))
            length=len(json_ca_certs)
            print("Number of certificates in bundle: {}".format(length))

            # fail fast:
            assert "MY CommonName" in str(json_ca_certs), "private CA not found in used bundle"

            kwargs['ssl_context'] = context
            return super().init_poolmanager(*args, **kwargs)

我没有设置那些环境变量，它们是 None

    def __init__(self, **kwargs):

        print("REQUESTS_CA_BUNDLE: '{}'".format(os.environ.get('REQUESTS_CA_BUNDLE')))
        print("CURL_CA_BUNDLE: '{}'".format(os.environ.get('CURL_CA_BUNDLE')))

我创建了一个 session ，并且由于使用我的信任库创建上下文不再起作用，我尝试了 3 种不同的方式来使用 session.verify(每次尝试仅激活一个代码行):

        self.session = requests.Session()

        # variant 1: default not working anymore:
        self.session.verify = True

        # variant 2: usage of my special ca bundle does not work
        self.session.verify = "C:/Users/...../my_ca_bundle.pem"

        # variant 3: certifi cacert.pem file contains my private CA and does not work
        # certifi.where(): C:\Users\........\Python39\Lib\site-packages\certifi\cacert.pem
        self.session.verify = certifi.where()

在这里，我终于将我的适配器安装到 session :

        # mount with Schema only as prefix to use them for all my calls (this worked in past and should not be a new problem)
        adapter = self.MyHttpAdapter()
        self.session.mount("https://", adapter)

现在我做我的 self.session.get()来电。我再次尝试了这些设置，因为 context cafile也不是 session.verify作品，直接在 requests session 内:

    _api_url = 'https://my-domain.com/api/function'
    
    ###

    # variant 1: does not longer work, no setting above for session.verify nor context.load...():
    response = self.session.get(_api_url)

    # variant 2: verify = my special ca bundle work around does not work:
    response = self.session.get(_api_url, verify="C:/Users/...../my_ca_bundle.pem")
    
    # variant 3: verify = certifi cacert.pem work around does not work:
    response = self.session.get(_api_url, verify=certifi.where())

我在 requests/sessions.py 中添加了一些打印调试输出，以查看使用了什么
sessions request()使用的最终设置:

        settings = self.merge_environment_settings(
            prep.url, proxies, stream, verify, cert
        )
        print(f"DEBUG: request settings: {settings}")

sessions merge_environment_settings()就在 return () 之前

        print(f"DEBUG: merge_environment_settings() before return: verify={verify}")
        return {'verify': verify, 'proxies': proxies, 'stream': stream,
                'cert': cert}

只是看看，我的设置被使用:

DEBUG: merge_environment_settings() before return: verify=path_to_ca_bundle.pem
DEBUG: request settings: {'verify': 'path_to_ca_bundle.pem', 'proxies': OrderedDict(), 'stream': False, 'cert': None}

Python 和安装的模块是最新的:
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21)
证书==2021.10.8
请求==2.27.1
请求工具带==0.9.1

Answer 1

带有简短问题和解决方案的初始条目已经太长了。由于它现在可以与新的 CA 和服务器证书一起使用，我将把最新的问题放在这个答案上。
在 rfc3280 4.2.1.1授权 key 标识符

The authority key identifier extension provides a means of
identifying the public key corresponding to the private key used to
sign a certificate.  This extension is used where an issuer has
multiple signing keys (either due to multiple concurrent key pairs or
due to changeover).  The identification MAY be based on either the
key identifier (the subject key identifier in the issuer's
certificate) or on the issuer name and serial number.

The keyIdentifier field of the authorityKeyIdentifier extension MUST
be included in all certificates generated by conforming CAs to
facilitate certification path construction......

...
id-ce-authorityKeyIdentifier OBJECT IDENTIFIER ::=  { id-ce 35 }

AuthorityKeyIdentifier ::= SEQUENCE {
  keyIdentifier             [0] KeyIdentifier           OPTIONAL,
  authorityCertIssuer       [1] GeneralNames            OPTIONAL,
  authorityCertSerialNumber [2] CertificateSerialNumber OPTIONAL  }

注意:所有三个 AuthorityKeyIdentifier 属性都是可选的，并且没有链接在一起。
但在前一段可以读到:
识别可能基于 要么 1. key 标识符 (颁发者证书中的主题 key 标识符) 或 2. 发行人名称和序列号 .
相比之下，我们在 rfc4158 3.5.12 中看到了注释。 “匹配 key 标识符 (KID)”

...
NOTE:  Although required to be present by [RFC3280], it is extremely
important that KIDs be used only as sorting criteria or as hints
during certification path building.  KIDs are not required to match
during certification path validation and cannot be used to eliminate
certificates.  This is of critical importance for interoperating
across domains and multi-vendor implementations where the KIDs may
not be calculated in the same fashion.

它只是一个排序标准，而不是验证标准。
最后在 openSSL FAQ 15 “为什么 OpenSSL 会错误地设置授权 key 标识符 (AKID) 扩展？”

It doesn’t: this extension is often the cause of confusion.
Consider a certificate chain A->B->C so that A signs B and B signs C. Suppose certificate C contains AKID.
The purpose of this extension is to identify the authority certificate B. This can be done either by including the subject key identifier of B or its issuer name and serial number.
In this latter case because it is identifying certificate B it must contain the issuer name and serial number of B.
It is often wrongly assumed that it should contain the subject name of B. If it did this would be redundant information because it would duplicate the issuer name of C.

所以，他们是这个意思？？？
这可以通过包括

B 的主题 key 标识符
--> 或 <--

其发行人名称和 序列号。
在后一种情况下，因为它正在识别证书 B
它必须包含发行者名称和序列号 B .

这周我终于迷茫了……