linux - Apache 和其他服务挂起

Question

我在 Gentoo 上有一个生产 LAMP 服务器，它一直在运行一个个人 Intranet 网站。从昨天下午开始，它在任何连接后不久就挂起；相对较小的登录页面加载良好，但登录后涉及大量 MySQL 查询和数据，它会无限期挂起。

SSH 仍然能够连接，但奇怪的是，在传输了几页的字符后，连接也突然挂起。因此，要获取以下信息，我必须继续重新登录。我尝试了/etc/init.d/apache2 restart 和/etc/init.d/mysql restart ，然后完全重新启动系统；但可惜问题仍然存在。详细信息如下。

顶部:

top - 12:23:52 up  1:34,  2 users,  load average: 0.16, 0.09, 0.06 Tasks:  81 total,   1 running,  80 sleeping,   0 stopped,   0 zombie Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st Mem:   3920788k total,   123476k used,  3797312k free,     4676k buffers Swap:  1227772k total,        0k used,  1227772k free,   48524k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND   447 root      20   0     0    0    0 S    0  0.0   0:00.16 khubd
    1 root      20   0  2020  640  568 S    0  0.0   0:00.51 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0
    5 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/u:0
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1
    9 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1
   10 root      20   0     0    0    0 S    0  0.0   0:00.69 kworker/0:1

Apache 日志显示常见的黑客尝试:

# tail -50 /var/log/apache2/error_log
[Mon Mar 17 19:03:48 2014] [error] [client 116.58.240.169] File does not exist: /var/www/mysite/pma
[Mon Mar 17 19:03:48 2014] [error] [client 116.58.240.169] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 05:58:42 2014] [error] [client 202.53.8.82] File does not exist: /var/www/mysite/admin.cgi
[Tue Mar 18 07:19:42 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/phpTest
[Tue Mar 18 07:19:43 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/phpMyAdmin
[Tue Mar 18 07:19:43 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/pma
[Tue Mar 18 07:19:44 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 08:24:16 2014] [error] [client 222.5.204.73] invalid request-URI \xcc\\\xa4/\x83\x8f\x90:\x84\x90\x0f\xc4\x8dfe\xecb\x94v\x1f[\xd7Z\x95$X\xaby\x13k\x88\xf2\xeb\xf7\x1b\xfc\xe8a\xff
[Tue Mar 18 08:29:49 2014] [error] [client 76.3.191.245] invalid request-URI
[Tue Mar 18 08:38:00 2014] [error] [client 35.2.240.149] invalid request-URI
[Tue Mar 18 08:50:52 2014] [error] [client 173.26.148.34] invalid request-URI
[Tue Mar 18 10:57:48 2014] [error] [client 110.175.79.216] invalid request-URI
[Tue Mar 18 10:57:53 2014] [error] [client 110.248.140.59] invalid request-URI D\xe8\x91a\xbc\xe5WZ\xd0C]\x9f~\xb5\x89\bd\x9e"[w,\xc6\xd9\xde\x8b]#JJ\xbf\x12
[Tue Mar 18 14:24:54 2014] [error] [client 108.14.2.113] invalid request-URI
[Tue Mar 18 14:40:08 2014] [error] [client 86.217.136.41] invalid request-URI \x94FI-\x02;4JVOV\x0f\xba\b
[Tue Mar 18 14:45:42 2014] [error] [client 98.119.127.76] invalid request-URI
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-120x120-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-120x120.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon.png
[Tue Mar 18 16:20:45 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/phpTest
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/phpMyAdmin
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/pma
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 16:40:58 2014] [error] [client 122.170.93.35] invalid request-URI
[Tue Mar 18 16:57:54 2014] [error] [client 124.107.151.190] invalid request-URI
[Tue Mar 18 17:36:17 2014] [error] [client 68.147.250.90] invalid request-URI \x1d\x1e;&\x9e\xd2\xa8\xc2GNQ\\
[Tue Mar 18 23:38:20 2014] [error] [client 92.240.68.153] request failed: error reading the headers
[Wed Mar 19 02:52:43 2014] [error] [client 162.213.24.36] File does not exist: /var/www/mysite/CFIDE
[Wed Mar 19 06:26:06 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:07 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:07 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:09 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:15 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 07:48:28 2014] [error] [client 201.161.37.93] File does not exist: /var/www/crownware/manager
[Wed Mar 19 09:27:08 2014] [error] [client 113.184.228.73] invalid request-URI \xad_X\xdf\x9aIM6x\x01ti\xf6Ko\xebi
[Wed Mar 19 09:36:06 2014] [error] [client 162.213.24.36] File does not exist: /var/www/crownware/CFIDE
[Wed Mar 19 10:28:15 2014] [notice] caught SIGTERM, shutting down
[Wed Mar 19 10:28:17 2014] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0j PHP/5.4.6--pl0-gentoo configured -- resuming normal operations
[Wed Mar 19 10:43:31 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:31 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:35 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:35 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:36 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:47:16 2014] [notice] caught SIGTERM, shutting down
[Wed Mar 19 10:49:32 2014] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0j PHP/5.4.6--pl0-gentoo configured -- resuming normal operations
[Wed Mar 19 10:53:45 2014] [error] [client 65.60.209.141] Invalid URI in request \x13\xe0\x94\xc4\xa4o\xd1\xd3*\xe0\xe7\x1a\xce\xd9\xe8\t\xca\xc3k\x9f\xb0\x06\x13\xbcE\x17\xbb\x02\x9c:\xffD\x8d\x1f\x85Wv\x14\xfd\x8f\xe3k\xc6\xfe\xf7\x1bu
[Wed Mar 19 12:20:07 2014] [error] [client 173.24.52.209] invalid request-URI

来自/var/log/mysql/mysqld.err 的最后一条感兴趣的消息(5 天前):

140314  9:56:02  InnoDB: ERROR: the age of the last checkpoint is 9448765,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.

版本:

# uname -a
Linux myhost 3.3.8-gentoo #1 SMP Fri Sep 28 09:34:42 MYT 2012 i686 Intel(R) Xeon(R) CPU E31220 @ 3.10GHz GenuineIntel GNU/Linux

# mysqld -V
140319 12:37:13 [Warning] '--default-character-set' is deprecated and will be removed in a future release. Please use '--character-set-server' instead.
140319 12:37:13 [Warning] '--default-collation' is deprecated and will be removed in a future release. Please use '--collation-server' instead.
mysqld  Ver 5.1.62-log for pc-linux-gnu on i686 (Gentoo Linux mysql-5.1.62-r1)

# apache2 -V
Server version: Apache/2.2.23 (Unix)
Server built:   Oct 27 2012 19:17:52
Server's Module Magic Number: 20051115:31
Server loaded:  APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/var/run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
 -D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"

# php -v
PHP 5.4.6--pl0-gentoo (cli) (built: Oct 27 2012 18:42:24)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies

磁盘似乎还有足够的空间:

# df
Filesystem     1K-blocks     Used Available Use% Mounted on
rootfs         960125048 84604800 826748732  10% /
udev               10240        0     10240   0% /dev
/dev/sda3      960125048 84604800 826748732  10% /
tmpfs            1960392      220   1960172   1% /run
rc-svcdir           1024       64       960   7% /lib/rc/init.d
cgroup_root        10240        0     10240   0% /sys/fs/cgroup
shm              1960392        0   1960392   0% /dev/shm

Apache 进程:

# ps -ef|grep -i apache
root      2060     1  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2062  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2066  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2067  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2068  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2069  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2070  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2123  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2124  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2125  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2148  2060  0 10:50 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2149  2060  0 10:50 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start

跟踪父(根)进程反复显示此信息，不确定是否正常:

# strace -p 2060
Process 2060 attached
select(0, NULL, NULL, NULL, {0, 669445}) = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)

事实上，SSH 在几千个字节后也挂起，这表明我应该比 Apache 更广泛地寻找。下一步诊断是什么？

Answer 1

从 Apache 日志中可以明显看出，您的站点是典型漏洞利用脚本的目标，这些脚本只是尝试通过对已知应用程序的请求来轰炸服务器以查找漏洞。

这可能导致了妥协 - 这部分很难判断，因为您没有详细说明您的计算机上正在运行哪些其他脚本。

我建议在您的服务器上运行 rootkit 分析器或类似工具。

此外，此类问题更适合 serverfault.com，因为它与编程(stackoverflow 的内容)无关，而是与系统管理/服务器管理相关。

为了防止此类请求到达您的服务器，建议使用 WAF(Web 应用程序防火墙)或其他代理，它们会在此类请求到达您的计算机之前对其进行限制和阻止。

naxsi是 nginx 的一个模块，提供开源 WAF。