json - Postgres - 大型 jsonb 列的性能

Question

我们在我们的一个数据库表中使用 Postgres jsonb 类型。表结构如下:

CREATE TABLE T (
  id UUID NOT NULL PRIMARY KEY,
  payload JSONB
);

CREATE INDEX ON T USING gin (payload jsonb_path_ops);

Payload 是一个复杂的 json 字符串。下面是一个例子:

{
    "business": {
        "taxId": "626642071",
        "legalName": "Jon's Deli",
        "phoneNumbers": [
            {
                "phoneType": "Business",
                "telephoneNumber": "8384407555"
            },
            {
                "phoneType": "Work",
                "telephoneNumber": "6032255248"
            }
        ],
        "addresses": [
            {
                "city": "San Francisco",
                "state": "CA",
                "postalCode": "94101",
                "countryCode": "USA",
                "addressLine1": "123 Market St"
            }
        ]
    },
    "stakeholders": [
        {
            "person": {
                "taxId": "540646815",
                "firstName": "GdXFouh",
                "lastName": "IlUAcgCGz",
                "dateOfBirth": "1980-12-11",
                "emailAddress": "jywxsijgix@qaqmlz.com",
                "phoneNumbers": [
                    {
                        "phoneType": "Mobile",
                        "telephoneNumber": "4901371573"
                    }
                ],
                "addresses": [
                    {
                        "city": "San Francisco",
                        "state": "CA",
                        "postalCode": "94101",
                        "countryCode": "USA",
                        "addressLine1": "123 Market St"
                    }
                ]
            }
        }
    ]
}

请注意，phoneNumbers、addresses 和stakeholders 是数组，这意味着数组中可以有多个元素。

我尝试向表中插入一百万行。 payload 的每个字段都是随机生成的。最初测试程序运行得非常快。但是在插入大约800,000行之后，每1000行就卡住——插入1000行，然后测试程序挂了2分钟，然后回来再插入1000行，...

我们怀疑这是由大量的 jsonb 索引更新引起的。因为对于单行，索引中有很多字段需要更新。我们只是想确认是否有人遇到过同样的问题。

---

实际上我们不需要索引整个payload列。仅需要某些字段:business->taxId、business->phoneNumbers->telephoneNumber、stakeholders->person->taxId 和 stakeholders->person->emailAddress.

我尝试了以下两个指标:

CREATE INDEX ON T USING gin ((payload->'business'->'taxId') jsonb_path_ops);
CREATE INDEX ON T USING gin ((payload ->'stakeholders'->'person'->'taxId') jsonb_path_ops);

然后运行两条语句:

explain select * from T where payload->'business'->'taxId' @> '"123456789"'; (1)
explain select * from T where payload->'stakeholders'->'person'->'taxId' @> '"123456789"'; (2)

第一条语句是使用索引。但是第二个是全表扫描，速度很慢。这就是我们转而索引整个 payload 列的原因。

---

欢迎提出任何建议。

顺便说一句，我们使用的是 Postgres 9.5.4。

Answer 1

您的查询:

select * from T where payload->'stakeholders'->'person'->'taxId' @> '"123456789"';

不起作用。这是因为“利益相关者”是数组。有效的查询是:

select * from T where payload->'stakeholders' @> '[{"person": {"taxId": "54"}}]'::jsonb

但在这种情况下，postgres 可以对整个利益相关者使用索引。

                                                       QUERY PLAN                                                       
------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on t  (cost=1388.08..1425.90 rows=10 width=36) (actual time=1.959..1.959 rows=1 loops=1)
   Recheck Cond: ((payload -> 'stakeholders'::text) @> '[{"person": {"taxId": "54"}}]'::jsonb)
   Heap Blocks: exact=1
   ->  Bitmap Index Scan on t_expr_idx3  (cost=0.00..1388.08 rows=10 width=0) (actual time=1.946..1.946 rows=1 loops=1)
         Index Cond: ((payload -> 'stakeholders'::text) @> '[{"person": {"taxId": "54"}}]'::jsonb)
 Planning time: 0.071 ms
 Execution time: 1.978 ms

为了使用更具体的索引，我使用修改后的方法:How do you create a Postgresql JSONB array in array index?

CREATE OR REPLACE FUNCTION extract_taxids(a_json jsonb).
RETURNS jsonb AS $BODY$
     SELECT jsonb_agg(j) FROM (SELECT jsonb_array_elements(a_json->'stakeholders')->'person'->'taxId' AS j) AS j
$BODY$ LANGUAGE sql IMMUTABLE;
CREATE INDEX ON T USING gin (extract_taxids(payload));

瞧:

EXPLAIN ANALYZE select * from T where extract_taxids(payload) @> '["54"]';

                                                           QUERY PLAN                                                           
--------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on t  (cost=12.08..52.38 rows=10 width=36) (actual time=0.101..0.102 rows=1 loops=1)
   Recheck Cond: (extract_taxids(payload) @> '["54"]'::jsonb)
   Heap Blocks: exact=1
   ->  Bitmap Index Scan on t_extract_taxids_idx  (cost=0.00..12.07 rows=10 width=0) (actual time=0.008..0.008 rows=1 loops=1)
         Index Cond: (extract_taxids(payload) @> '["54"]'::jsonb)
 Planning time: 0.128 ms
 Execution time: 0.117 ms