python - 如何递归字典并动态更新值以便以后可以引用它们？

Question

我正在尝试编写一个配置变量引擎，该引擎接受 YAML 文件(包含 AWS 配置变量)作为输入并转换为 JSON，以便可以将其上传到 HTTP k/v API(例如 Consul)。我感到困惑的一个功能是允许开发人员在后续 key 中“包含” key 集(用下划线标识，在最终有效负载中被省略)。示例如下:

# Region
us-east-1:
  # Any key preceded by an underscore (_) is considered a "tag group" and will not be uploaded to Consul KV unless explicitly included.
  _taggroup1:
    key1: value1
    key2: value2
    key3: value3
  _taggroup2:
    key4: value1
    key5: value2
    key6: value3

  dev:
    _include: us-east-1/_taggroup1
  qa:
    _include:
      - us-east-1/_taggroup1
      - us-east-1/_taggroup2
    key6: baz
  prod:
    _include:
      - us-east-1/_taggroup1
      - us-east-1/_taggroup2

us-west-1:
  _taggroup1:
    key1: value1
    key2: value2
    key3: value3
  _taggroup2:
    key4: value1
    key5: value2
    key6: value3

  dev:
    _include:
      - us-west-1/_taggroup1
  qa:
    _include:
      - us-west-1/_taggroup1
      - us-west-1/_taggroup2
    key2: foo
  prod:
    _include:
      - us-west-1/_taggroup1
      - us-west-1/_taggroup2
    key4: foo
    key5: bar
    key1: undef

  us-west-1a:
    qa:
      _include: us-west-1/qa
    prod:
      _include: us-west-1/prod

  us-west-1b:
    _include: us-west-1/us-west-1a

如您所见，我正在尝试构建一个配置文件，该文件允许开发人员对变量进行分组，并随后根据需要包含/覆盖它们。

到目前为止，我为此实验编写的代码本质上是标准递归函数，并添加了特定于该应用程序的内容:

# parse_input is a separate function that converts a YAML stream into
# an OrderedDict
original_dict = parse_input(stream1)

def print_dict(input_dict):

    new_dict = collections.OrderedDict()

    for key, value in input_dict.iteritems():
        if key.startswith('_'):
            if key == '_include':
                if isinstance(value, list):
                    for item in value:
                        x = dpath.util.get(original_dict, item)
                        for k, v in x.iteritems():
                            new_dict[k] = v
                else:
                    x = dpath.util.get(original_dict, value)
                    for k, v in x.iteritems():
                        new_dict[k] = v
            else:
                continue
            continue
        elif isinstance(value, dict):
            new_dict[key] = print_dict(value)
        else:
            new_dict[key] = value
    return new_dict

到目前为止我所实现的输出是这样的:

{
    "us-east-1": {
        "dev": {
            "key1": "value1", 
            "key2": "value2", 
            "key3": "value3"
        }, 
        "qa": {
            "key1": "value1", 
            "key2": "value2", 
            "key3": "value3", 
            "key4": "value1", 
            "key5": "value2", 
            "key6": "baz"
        }, 
        "prod": {
            "key1": "value1", 
            "key2": "value2", 
            "key3": "value3", 
            "key4": "value1", 
            "key5": "value2", 
            "key6": "value3"
        }
    }, 
    "us-west-1": {
        "dev": {
            "key1": "value1", 
            "key2": "value2", 
            "key3": "value3"
        }, 
        "qa": {
            "key1": "value1", 
            "key2": "foo", 
            "key3": "value3", 
            "key4": "value1", 
            "key5": "value2", 
            "key6": "value3"
        }, 
        "prod": {
            "key1": "undef", 
            "key2": "value2", 
            "key3": "value3", 
            "key4": "foo", 
            "key5": "bar", 
            "key6": "value3"
        }, 
        "us-west-1a": {
            "qa": {
                "_include": [
                    "us-west-1/_taggroup1", 
                    "us-west-1/_taggroup2"
                ], 
                "key2": "foo"
            }, 
            "prod": {
                "_include": [
                    "us-west-1/_taggroup1", 
                    "us-west-1/_taggroup2"
                ], 
                "key4": "foo", 
                "key5": "bar", 
                "key1": "undef"
            }
        }, 
        "us-west-1b": {
            "qa": {
                "_include": "us-west-1/qa"
            }, 
            "prod": {
                "_include": "us-west-1/prod"
            }
        }
    }
}

正如你所看到的，我似乎已经成功了一半。我的问题是，在最初的实验中，在引用包含集(使用 dpath 返回键)时，通过引用函数中的original_dict 变量，我获得了良好的结果。随着函数递归得更深(即本例中的 AZ 特定变量)，这很快就会变成一个问题，因为我不知道如何动态更新原始字典中的键，或者以其他方式跟踪更改，因此该函数将注入(inject)使用 _include 键设置的键，但无法重新评估它们。

如何消除依赖于引用原始字典，而是动态跟踪更改，以便在树的更深处正确评估 _include 键？

Answer 1

我认为这段代码解决了您面临的问题。关键的更改是使用 dpath 的结果递归到 print_dict。我还折叠了一些多余的代码。

代码:

import yaml
import collections
import json
import dpath

with open('data.yml', 'rb') as f:
    original_dict = yaml.load(f)

def print_dict(input_dict):

    new_dict = collections.OrderedDict()

    for key, value in input_dict.iteritems():
        if key.startswith('_'):
            if key == '_include':
                if not isinstance(value, list):
                    value = [value]
                for item in value:
                    x = print_dict(dpath.util.get(original_dict, item))
                    for k, v in x.iteritems():
                        new_dict[k] = v
        elif isinstance(value, dict):
            new_dict[key] = print_dict(value)
        else:
            new_dict[key] = value
    return new_dict

print(json.dumps(print_dict(original_dict), indent=2))

输出:

{
  "us-east-1": {
    "qa": {
      "key3": "value3", 
      "key2": "value2", 
      "key1": "value1", 
      "key6": "baz", 
      "key5": "value2", 
      "key4": "value1"
    }, 
    "prod": {
      "key3": "value3", 
      "key2": "value2", 
      "key1": "value1", 
      "key6": "value3", 
      "key5": "value2", 
      "key4": "value1"
    }, 
    "dev": {
      "key3": "value3", 
      "key2": "value2", 
      "key1": "value1"
    }
  }, 
  "us-west-1": {
    "qa": {
      "key2": "value2", 
      "key3": "value3", 
      "key1": "value1", 
      "key6": "value3", 
      "key5": "value2", 
      "key4": "value1"
    }, 
    "us-west-1b": {
      "qa": {
        "key2": "value2", 
        "key3": "value3", 
        "key1": "value1", 
        "key6": "value3", 
        "key5": "value2", 
        "key4": "value1"
      }, 
      "prod": {
        "key1": "value1", 
        "key3": "value3", 
        "key2": "value2", 
        "key6": "value3", 
        "key5": "bar", 
        "key4": "foo"
      }
    }, 
    "prod": {
      "key1": "value1", 
      "key3": "value3", 
      "key2": "value2", 
      "key6": "value3", 
      "key5": "bar", 
      "key4": "foo"
    }, 
    "us-west-1a": {
      "qa": {
        "key2": "value2", 
        "key3": "value3", 
        "key1": "value1", 
        "key6": "value3", 
        "key5": "value2", 
        "key4": "value1"
      }, 
      "prod": {
        "key1": "value1", 
        "key3": "value3", 
        "key2": "value2", 
        "key6": "value3", 
        "key5": "bar", 
        "key4": "foo"
      }
    }, 
    "dev": {
      "key3": "value3", 
      "key2": "value2", 
      "key1": "value1"
    }
  }
}