python 和 twisted proxy，如何动态 gunzip？

Question

在使用 twistedmatrix ProxyClient 时，我如何压缩和处理响应部分？

我需要检查文本或 javascript 和 ajax 查询/答案。我应该使用 handleResponseEnd 吗？

我认为它在 handleResponsePart 内部，但看起来我误解了一点或其他什么，这是我的框架代码:

from twisted.python import log
from twisted.web import http, proxy

class ProxyClient(proxy.ProxyClient):
    """Mange returned header, content here.

    Use `self.father` methods to modify request directly.
    """
    def handleHeader(self, key, value):
            # change response header here
            log.msg("Header: %s: %s" % (key, value))
            proxy.ProxyClient.handleHeader(self, key, value)

    def handleResponsePart(self, buffer):
    # this part below do not work, 
    # looks like @ this moment i do not have 'Content-Encoding' or 'Content-Type'
    # what am i misunderstading?
            cEncoding = self.father.getAllHeaders().get('Content-Encoding', '')
            cType = self.father.getAllHeaders().get('Content-Type', '')
            print >> sys.stderr, 'Content-Encoding', cEncoding
            print >> sys.stderr, 'Content-Type', cType
            if ('text' in cType.lower() or 'javascript' in cType.lower()) and 'gzip' in cEncoding.lower():

                buf = StringIO(buffer)
                s = gzip.GzipFile(mode="rb", fileobj=buf)
                content = s.read(len(buffer))

                # here process content as it should be gunziped

    proxy.ProxyClient.handleResponsePart(self, buffer)

class ProxyClientFactory(proxy.ProxyClientFactory):
    protocol = ProxyClient

class ProxyRequest(proxy.ProxyRequest):
    protocols = dict(http=ProxyClientFactory)

class Proxy(proxy.Proxy):
    requestFactory = ProxyRequest

class ProxyFactory(http.HTTPFactory):
    protocol = Proxy

从我的记录中我有:

2013-06-11 14:07:33+0200 [ProxyClient,client] Header: Date: Tue, 11 Jun 2013 12:07:25 GMT
2013-06-11 14:07:33+0200 [ProxyClient,client] Header: Server: Apache
...
2013-06-11 14:07:33+0200 [ProxyClient,client] Header: Content-Type: text/html;charset=ISO-8859-1
...
2013-06-11 14:07:33+0200 [ProxyClient,client] Header: Content-Encoding: gzip
...
2013-06-11 14:07:33+0200 [ProxyClient,client] Header: Connection: close

所以我应该具备这两个条件吧!请问我错过了什么？

即使我对第二种方式不感兴趣，也就是删除对请求的接受，像这样，是否可以这样做:(顺便说一句，它看起来不起作用，或者经过测试的网络服务器不关心我们不想接收 gzip 压缩内容这一事实)

class ProxyRequest(proxy.ProxyRequest):
    protocols = dict(http=ProxyClientFactory)

    def process(self):
        # removing the accept so that we do not tell "i'm ok with gzip encoded content" and should receive only not gzip-ed
        self.requestHeaders.removeHeader('accept')
        self.requestHeaders.removeHeader('accept-encoding')

Answer 1

您必须在 handleResponsePart 中将数据 block 收集到 StringIO 缓冲区中，然后在 handleResponseEnd 中使用 GzipFile 进行解码。