MySQL查询优化，大表，使用临时文件排序

Question

我需要您帮助优化查询。一个表是一个日志表，其中有数百万个条目，我尝试将查询时间缩短到 < 1 秒。我的查询应该给出总体概述，因此应该很快。我确信我可以在脚本的帮助下对列表进行多个简单的查询。但是脚本能做什么，我想 mysql 也能做——至少我希望如此。也许并非所有部分都最好使用，但我陷入了一个生成临时表和文件排序的查询(我发现这非常糟糕)。在阅读周围的内容时，我发现可以使用一些整洁且位置良好的索引，但现在我陷入了一个特定的点。

让我向您展示我的最终查询及其结果:

SELECT
    ps.SERVER_ID,
    ps.FULLNAME,
    SUM(CASE WHEN pml.ID_TYPE = 3 THEN 1 ELSE 0 END) 'amount_warning',
    SUM(CASE WHEN pml.ID_TYPE = 4 THEN 1 ELSE 0 END) 'amount_error',
    SUM(CASE WHEN pml.ID_TYPE = 5 THEN 1 ELSE 0 END) 'amount_alert',
    SUM(CASE WHEN pml.ID_TYPE = 7 THEN 1 ELSE 0 END) 'amount_critical'
FROM
    PAR_SERVER ps
INNER JOIN
    PAR_MONITORINGv2_LOG pml ON ps.SERVER_ID = pml.SERVER_ID
WHERE
    pml.CREATED_DATE > date_sub( NOW( ) , INTERVAL 7 DAY )
GROUP BY
    ps.SERVER_ID;

这是我得到的:

mysql> [thequeryabove]
[...]
59 rows in set (11.69 sec)

mysql> explain [thequeryabove]
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type   | possible_keys               | key     | key_len | ref                       | rows    | Extra                                        |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
|  1 | SIMPLE      | pml   | ALL    | SERVER_ID,SERVER_ID-ID_TYPE | NULL    | NULL    | NULL                      | 4014447 | Using where; Using temporary; Using filesort |
|  1 | SIMPLE      | ps    | eq_ref | PRIMARY                     | PRIMARY | 4       | database.pml.SERVER_ID |       1 |                                              |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
2 rows in set (0.00 sec)

这是我当前的表格设置:

mysql> describe PAR_SERVER;
+----------------+--------------+------+-----+---------+----------------+
| Field          | Type         | Null | Key | Default | Extra          |
+----------------+--------------+------+-----+---------+----------------+
| SERVER_ID      | int(255)     | NO   | PRI | NULL    | auto_increment |
| FULLNAME       | varchar(255) | YES  |     | NULL    |                |
| SHORTNAME      | varchar(255) | YES  | MUL | NULL    |                |
+----------------+--------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)

mysql> show indexes from PAR_SERVER;
+------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table      | Non_unique | Key_name  | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| PAR_SERVER |          0 | PRIMARY   |            1 | SERVER_ID   | A         |         142 |     NULL | NULL   |      | BTREE      |         |
| PAR_SERVER |          1 | shortname |            1 | SHORTNAME   | A         |         142 |     NULL | NULL   | YES  | BTREE      |         |
+------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
2 rows in set (0.00 sec)

mysql> select count(*) from PAR_SERVER;
+----------+
| count(*) |
+----------+
|      142 |
+----------+
1 row in set (0.00 sec)

mysql> describe PAR_MONITORINGv2_LOG;
+--------------+----------+------+-----+---------+----------------+
| Field        | Type     | Null | Key | Default | Extra          |
+--------------+----------+------+-----+---------+----------------+
| ID           | int(11)  | NO   | PRI | NULL    | auto_increment |
| ID_TYPE      | int(11)  | NO   | MUL | NULL    |                |
| ID_SERVICE   | int(11)  | NO   | MUL | NULL    |                |
| SERVER_ID    | int(11)  | NO   | MUL | NULL    |                |
| MESSAGE      | tinytext | NO   |     | NULL    |                |
| CREATED_DATE | datetime | NO   |     | NULL    |                |
+--------------+----------+------+-----+---------+----------------+
6 rows in set (0.00 sec)

mysql> show indexes from PAR_MONITORINGv2_LOG;
+----------------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table                | Non_unique | Key_name          | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| PAR_MONITORINGv2_LOG |          0 | PRIMARY           |            1 | ID          | A         |     3998188 |     NULL | NULL   |      | BTREE      |         |
| PAR_MONITORINGv2_LOG |          1 | ID_TYPE           |            1 | ID_TYPE     | A         |           7 |     NULL | NULL   |      | BTREE      |         |
| PAR_MONITORINGv2_LOG |          1 | ID_SERVICE        |            1 | ID_SERVICE  | A         |           5 |     NULL | NULL   |      | BTREE      |         |
| PAR_MONITORINGv2_LOG |          1 | SERVER_ID         |            1 | SERVER_ID   | A         |          66 |     NULL | NULL   |      | BTREE      |         |
| PAR_MONITORINGv2_LOG |          1 | SERVER_ID-ID_TYPE |            1 | SERVER_ID   | A         |          66 |     NULL | NULL   |      | BTREE      |         |
| PAR_MONITORINGv2_LOG |          1 | SERVER_ID-ID_TYPE |            2 | ID_TYPE     | A         |         258 |     NULL | NULL   |      | BTREE      |         |
+----------------------+------------+-------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
6 rows in set (0.00 sec)

mysql> select count(*) from PAR_MONITORINGv2_LOG;
+----------+
| count(*) |
+----------+
|  3998386 |
+----------+
1 row in set (0.00 sec)

以下是逐步打破我的查询的时间结果。在花了这么长时间修复每个部分后，我可能会一步一步地进行。但目前只有运行时间为 2.30 秒的查询对这个问题感兴趣。

mysql> SELECT ps.SERVER_ID, ps.FULLNAME FROM PAR_SERVER ps INNER JOIN PAR_MONITORINGv2_LOG pml ON ps.SERVER_ID = pml.SERVER_ID WHERE pml.CREATED_DATE > date_sub( NOW( ) , INTERVAL 7 DAY ) GROUP BY ps.SERVER_ID;
[...]
59 rows in set (6.41 sec)

mysql> explain [thequeryabove]
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type   | possible_keys               | key     | key_len | ref                       | rows    | Extra                                        |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
|  1 | SIMPLE      | pml   | ALL    | SERVER_ID,SERVER_ID-ID_TYPE | NULL    | NULL    | NULL                      | 4014788 | Using where; Using temporary; Using filesort |
|  1 | SIMPLE      | ps    | eq_ref | PRIMARY                     | PRIMARY | 4       | database.pml.SERVER_ID |       1 |                                              |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+---------+----------------------------------------------+
2 rows in set (0.00 sec)

mysql> SELECT ps.SERVER_ID, ps.FULLNAME FROM PAR_SERVER ps INNER JOIN PAR_MONITORINGv2_LOG pml ON ps.SERVER_ID = pml.SERVER_ID GROUP BY ps.SERVER_ID;
[...]
59 rows in set (2.30 sec)

mysql> explain [thequeryabove]
+----+-------------+-------+--------+-----------------------------+-----------+---------+---------------------------+---------+----------------------------------------------+
| id | select_type | table | type   | possible_keys               | key       | key_len | ref                       | rows    | Extra                                        |
+----+-------------+-------+--------+-----------------------------+-----------+---------+---------------------------+---------+----------------------------------------------+
|  1 | SIMPLE      | pml   | index  | SERVER_ID,SERVER_ID-ID_TYPE | SERVER_ID | 4       | NULL                      | 4015694 | Using index; Using temporary; Using filesort |
|  1 | SIMPLE      | ps    | eq_ref | PRIMARY                     | PRIMARY   | 4       | database.pml.SERVER_ID |       1 |                                              |
+----+-------------+-------+--------+-----------------------------+-----------+---------+---------------------------+---------+----------------------------------------------+
2 rows in set (0.00 sec)

mysql> SELECT pml.SERVER_ID FROM PAR_MONITORINGv2_LOG pml GROUP BY pml.SERVER_ID;
[...]
65 rows in set (0.00 sec)

mysql> explain [thequeryabove]
+----+-------------+-------+-------+---------------+-----------+---------+------+------+--------------------------+
| id | select_type | table | type  | possible_keys | key       | key_len | ref  | rows | Extra                    |
+----+-------------+-------+-------+---------------+-----------+---------+------+------+--------------------------+
|  1 | SIMPLE      | pml   | range | NULL          | SERVER_ID | 4       | NULL |   67 | Using index for group-by |
+----+-------------+-------+-------+---------------+-----------+---------+------+------+--------------------------+
1 row in set (0.00 sec)

通过为 (SERVER_ID, ID_TYPE) 定义索引，我能够极大地改进查询，正如我的以下示例查询所证实的那样:

mysql> SELECT count(*) 'count_warnings' FROM PAR_MONITORINGv2_LOG pml WHERE pml.SERVER_ID = 191 AND pml.ID_TYPE = 3 GROUP BY pml.SERVER_ID;
[...]
1 row in set (0.01 sec)

mysql> explain [thequeryabove]
+----+-------------+-------+------+-------------------------------------+-------------------+---------+-------------+-------+-------------+
| id | select_type | table | type | possible_keys                       | key               | key_len | ref         | rows  | Extra       |
+----+-------------+-------+------+-------------------------------------+-------------------+---------+-------------+-------+-------------+
|  1 | SIMPLE      | pml   | ref  | ID_TYPE,SERVER_ID,SERVER_ID-ID_TYPE | SERVER_ID-ID_TYPE | 8       | const,const | 10254 | Using index |
+----+-------------+-------+------+-------------------------------------+-------------------+---------+-------------+-------+-------------+
1 row in set (0.00 sec)

我现在陷入了最崩溃的查询，执行时间长达 2.30 秒。我不知道如何在没有任何 where 子句的查询中使用索引。

Answer 1

在 PAR_MONITORINGv2_LOG(CREATED_DATE, SERVER_ID,ID_TYPE) 上添加复合索引肯定会让您的查询受益匪浅。不过，我建议即使是 CREATED_DATE 上的简单索引也会大大提高性能。