python: Mac OS X.malloc 错误。未分配正在释放的指针。中止陷阱 6

Question

我正在运行一个多线程 python 脚本。它的作用是爬网并插入/更新到 mysql 中。这是我的代码

我的线程.py

import threading
import time

class MyThread (threading.Thread):
    def __init__(self, threadID, threadname, q):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.threadname = threadname
        self.queue = q
        self.__exitFlag = False
        self.__signal_lock = threading.Lock()

    def run(self):
        print "Starting " + self.threadname
        self.process_data()
        print "Exiting " + self.threadname

    def stop(self):
        with self.__signal_lock:
            self.__exitFlag = True

    def process_data(self):
        while not self.__exitFlag:
            if not self.queue.empty():
                data = self.queue.get()
                # crawl data from the web...
                # update to mysql
                # assuming we have already connected mysql:
                # db = MySQLDb()
                # db.connect
                query = ""
                db.query(query)

mysql_db.py

class MySQLDb:
    conn = None

    def connect(self):
        self.conn = MySQLdb.connect(
            host="127.0.0.1",
            user = "root",
            passwd = "password",
            db = "moviestats")

        self.cursor = self.conn.cursor(MySQLdb.cursors.DictCursor)

    def query(self, sql):
        try:
            self.cursor.execute(sql)
            self.conn.commit()
        except (AttributeError, MySQLdb.OperationalError):
            # solution to: MySQL server has gone away
            self.cursor.close()
            self.connect()
            self.cursor = self.conn.cursor(MySQLdb.cursors.DictCursor)
            self.cursor.execute(sql)
            self.conn.commit()

错误日志如下:

Process:         Python [905]
Path:            /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Identifier:      Python
Version:         2.7.7 (2.7.7)
Code Type:       X86-64 (Native)
Parent Process:  bash [751]
Responsible:     Terminal [410]
User ID:         501

Date/Time:       2014-07-09 22:31:43.221 +0800
OS Version:      Mac OS X 10.9.3 (13D65)
Report Version:  11
....

....
Crashed Thread:  5

Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000

Application Specific Information:
abort() called
*** error for object 0x100a4b600: pointer being freed was not allocated
......
Thread 5 Crashed:
0   libsystem_kernel.dylib          0x00007fff83153866 __pthread_kill + 10
1   libsystem_pthread.dylib         0x00007fff8de8735c pthread_kill + 92
2   libsystem_c.dylib               0x00007fff8ef88b1a abort + 125
3   libsystem_malloc.dylib          0x00007fff8220707f free + 411
4   libmysqlclient.18.dylib         0x0000000101027302 vio_delete + 44
5   libmysqlclient.18.dylib         0x000000010100709a end_server + 48
6   libmysqlclient.18.dylib         0x0000000101006f81 cli_safe_read + 49
7   libmysqlclient.18.dylib         0x000000010100b469 cli_read_query_result + 26
8   libmysqlclient.18.dylib         0x000000010100a648 mysql_real_query + 83
9   _mysql.so                       0x0000000100533be8 _mysql_ConnectionObject_query + 85
10  org.python.python               0x00000001000c2fad PyEval_EvalFrameEx + 21405
11  org.python.python               0x00000001000c3bfa PyEval_EvalFrameEx + 24554
12  org.python.python               0x00000001000c3bfa PyEval_EvalFrameEx + 24554
13  org.python.python               0x00000001000c4fb3 PyEval_EvalCodeEx + 2115
14  org.python.python               0x00000001000c33f0 PyEval_EvalFrameEx + 22496
15  org.python.python               0x00000001000c3bfa PyEval_EvalFrameEx + 24554
16  org.python.python               0x00000001000c3bfa PyEval_EvalFrameEx + 24554
17  org.python.python               0x00000001000c4fb3 PyEval_EvalCodeEx + 2115
18  org.python.python               0x00000001000c33f0 PyEval_EvalFrameEx + 22496
19  org.python.python               0x00000001000c3bfa PyEval_EvalFrameEx + 24554
20  org.python.python               0x00000001000c3bfa PyEval_EvalFrameEx + 24554
21  org.python.python               0x00000001000c3bfa PyEval_EvalFrameEx + 24554
22  org.python.python               0x00000001000c4fb3 PyEval_EvalCodeEx + 2115
23  org.python.python               0x000000010003eac0 function_call + 176
24  org.python.python               0x000000010000ceb2 PyObject_Call + 98
25  org.python.python               0x000000010001f56d instancemethod_call + 365
26  org.python.python               0x000000010000ceb2 PyObject_Call + 98
27  org.python.python               0x00000001000bc957 PyEval_CallObjectWithKeywords + 87
28  org.python.python               0x0000000100102f27 t_bootstrap + 71
29  libsystem_pthread.dylib         0x00007fff8de86899 _pthread_body + 138
30  libsystem_pthread.dylib         0x00007fff8de8672a _pthread_start + 137
31  libsystem_pthread.dylib         0x00007fff8de8afc9 thread_start + 13

我用 50 个线程运行脚本。错误发生是间歇性的，但它是可重复的。我缩小了问题范围，这是由于对 mysql 的插入/更新。我读到这可能是由于并发问题，但我该如何解决？

Answer 1

我在 OSX 上使用 MySQLdb 时遇到了同样的 malloc 错误。导致我出错的原因是在线程之间共享 MySQLdb 连接。使用每个线程的连接为我修复了它。

来自文档 http://mysql-python.sourceforge.net/MySQLdb.html :

The MySQL protocol can not handle multiple threads using the same connection at once. Some earlier versions of MySQLdb utilized locking to achieve a threadsafety of 2. While this is not terribly hard to accomplish using the standard Cursor class (which uses mysql_store_result()), it is complicated by SSCursor (which uses mysql_use_result(); with the latter you must ensure all the rows have been read before another query can be executed. It is further complicated by the addition of transactions, since transactions start when a cursor execute a query, but end when COMMIT or ROLLBACK is executed by the Connection object. Two threads simply cannot share a connection while a transaction is in progress, in addition to not being able to share it during query execution. This excessively complicated the code to the point where it just isn't worth it.

The general upshot of this is: Don't share connections between threads. It's really not worth your effort or mine, and in the end, will probably hurt performance, since the MySQL server runs a separate thread for each connection. You can certainly do things like cache connections in a pool, and give those connections to one thread at a time. If you let two threads use a connection simultaneously, the MySQL client library will probably upchuck and die. You have been warned.