python - Apache 和 Python 线程处理奇怪的结果

Question

我正在编写一些代码来查询 SteamAPI 以获取服务器上的数据，然后编译更大的列表并输出 JSON 数据。

它通过以下方式做到这一点:

• 请求所有服务器 Ip 的列表(ip，端口)
• 然后向 ip/port 发送请求
• 转换结果，
• 将其添加到主列表中
• 使用 json.dumps() 格式化最终列表

它是用 python 2.7.6 编写的

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# gets all the miscreated servers

import Queue
import json
import unicodedata
from threading import Thread

import pyfscache
import valve.source.a2s
import valve.source.master_server

cache_it = pyfscache.FSCache('./cache/', days=0, hours=4, minutes=30)

# The main queue object
q = Queue.LifoQueue()
# this is the list we append to
final_servers_list = []


def print_headers():
    print "Content-type: application/json\n"
    print ""


def get_master_server_list():
    msq = valve.source.master_server.MasterServerQuerier()
    servers = msq.find(appid="299740")
    return servers


def normalize(data):
    if type(data) is unicode:
        return unicodedata.normalize('NFKD', data).encode('ascii', 'ignore')
    else:
        return data


def get_single_server_data():
    while not q.empty():  # check that the queue isn't empty
        try:
            server_address = q.get()
            _server = valve.source.a2s.ServerQuerier(server_address)
            info = _server.get_info()

            try:
                server_time = info['server_tags'].split(';')[0][-5:]
            except:
                server_time = '00:00'
            try:
                players = info['server_tags'].split(';')[1]
            except:
                players = '0'
            try:
                whitelisted = info['server_tags'].split(';')[2]
            except:
                whitelisted = '0'

            final_servers_list.append({'name': normalize(info['server_name']),
                                       'mapName': normalize(info['map']),
                                       'ip': normalize(server_address[0]),
                                       'port': normalize(server_address[1]),
                                       'time': normalize(server_time),
                                       'players': normalize(players),
                                       'whiteListed': normalize(whitelisted),
                                       'maxPlayers': normalize(info['max_players']),
                                       'version': normalize(info['version'])})
            q.task_done()
        except:
            q.task_done()


@cache_it
def get_server_list():
    master_server_list = get_master_server_list()
    for server in master_server_list:
        q.put(server)
    for i in range(200):
        t1 = Thread(target=get_single_server_data)  # target is the above function
        t1.start()  # start the thread
    q.join()
    return final_servers_list


if __name__ == '__main__':
    print_headers()
    print json.dumps(get_server_list())

现在这段代码在我运行 scotchbox vagrant lamp stack 的本地机器上运行良好 python 与服务器/开发机器上的版本相同。

我在我的机器上得到了我期望的结果我得到了大约 500 台服务器并且所有数据都完全符合我的期望。

然而，当我在运行 apache2 的网络服务器上运行它时，它会向我吐出一个 10,000 范围内的列表，其中许多结果在列表中出现 10-20 次。即使我尝试过滤结果，它也好像某些数据略有不同(因为它可能是一秒钟后或第二次被请求？并且服务器标签已更改？)

我认为这与线程和 apache 与 python 有关，并且由于某种原因它没有某种锁定文件，但我终究无法弄清楚这一点。我以为我已经解决了这个问题

来源/etc/apache2/envvars

然后为 ssh 终端运行脚本

它开始工作，但下次缓存过期并且运行代码时，它返回相同的结果给我。

任何建议将不胜感激，因为我在这里用头撞墙。

作为我运行 apache2 -V 时的旁注

它吐出一个错误:

[Wed Feb 01 05:35:59.192112 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOCK_DIR} is not defined
[Wed Feb 01 05:35:59.192531 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_PID_FILE} is not defined
[Wed Feb 01 05:35:59.193300 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.214298 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_RUN_DIR} is not defined
[Wed Feb 01 05:35:59.215112 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.215499 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.215708 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.216057 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.216272 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.216595 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.217080 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.217475 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.217812 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.218115 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.218369 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.218657 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.218885 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.219117 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.219348 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.219631 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Feb 01 05:35:59.219845 2017] [core:warn] [pid 30832] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
AH00526: Syntax error on line 75 of /etc/apache2/apache2.conf:
Invalid Mutex directory in argument file:${APACHE_LOCK_DIR}

它在开发机器和生产机器上都这样做，所以我认为这不是什么大事。

最后是站点的 apache 配置。

<VirtualHost *:80>
        ServerName "servers.miscreatedgame.com"
        ServerAdmin "csprance@entradainteractive.com"
        DocumentRoot "/var/www/servers.com/_build"

        # serverpanel appollo
        <Directory "/var/www/servers.com/_build">
                AddHandler cgi-script .py
                Options +ExecCGI
        </Directory>



        ErrorLog ${APACHE_LOG_DIR}/servers-error.log
        CustomLog ${APACHE_LOG_DIR}/servers-access.log combined

</VirtualHost>

Answer 1

正如@SergGr 所说，这绝对像是多个请求之间的线程竞争条件。我建议尝试使您的代码可重入。我会将实际创建线程和构建整个服务器列表的代码放在一个单独的进程中，然后将其返回给使用 ipc 处理用户请求的进程。