java - 使用 Netty 构建在同一端口上处理 HTTP 和 HTTPS 的 MITM 代理时出现的问题

Question

我正在尝试学习如何使用 Netty 构建 MITM 代理。我的目标是让代理在同一端口上处理 HTTP 和 HTTPS。为简单起见，代理只会对任何入站消息响应 hello-world 消息。我做了很多学习，但我仍然很难实现我的目标。

下面是我写的相关代码。

// HTTP/HTTPS proxy service handler
public class InboundFrontHandler extends ByteToMessageDecoder {
    private static final HttpResponse CONNECT_RESPONSE = new DefaultHttpResponse( HttpVersion.HTTP_1_1,
            new HttpResponseStatus( 200, "Connection established") );

    @Override
    protected void decode( ChannelHandlerContext context, ByteBuf in, List<Object> out ) {
        log.info("Decoding message.");
        // Will use the first five bytes to detect a protocol.
        if (in.readableBytes() < 5) {
            return;
        }

        log.info("Received message: {}", in.toString(StandardCharsets.UTF_8));

        ChannelPipeline pipeline = context.pipeline();

        if (SslHandler.isEncrypted( in ) ) {
            log.info("message is encrypted.");
            if( pipeline.get( SSL_HANDLER ) == null ) {   // SSL_HANDLER: String
                log.info("no ssl handler available.");
                if( pipeline.get( HTTP_CODEC_HANDLER ) == null ) { // HTTP_CODEC_HANDLER: String
                    log.info("no http codec available");
                    pipeline.addLast(SSL_HANDLER, getSslHandler());
                    pipeline.addLast( HTTP_CODEC_HANDLER, new HttpServerCodec());
                    pipeline.addLast( DEFLATER_HANDLER, new HttpContentCompressor()); // DEFLATER_HANDLER: String
                } else {
                    log.info("http codec available");
                    pipeline.addBefore( HTTP_CODEC_HANDLER, SSL_HANDLER, getSslHandler());
                }
            }
        } else {
            log.info("message is not encrypted");
            if( pipeline.get( HTTP_CODEC_HANDLER ) == null ) {
                pipeline.addLast( HTTP_CODEC_HANDLER, new HttpServerCodec());
                pipeline.addLast( DEFLATER_HANDLER, new HttpContentCompressor());
            }
            if( isConnect( in ) ) {
                context.write( CONNECT_RESPONSE );
                log.info("responded CONNECT method with {}", CONNECT_RESPONSE);
                return;
            }
        }

        if( pipeline.get(HW_HANDLER) == null) {  // HW_HANDLER: String
            pipeline.addLast( HW_HANDLER, new HttpHandler());
        }
    }

    private boolean isConnect(ByteBuf in) {
        int magic1 = in.getUnsignedByte(in.readerIndex());
        int magic2 = in.getUnsignedByte(in.readerIndex() + 1);
        return magic1 == 'C' && magic2 == 'O';
    }

    private SslHandler getSslHandler() { // get a SslHandler with a self signed certificate in keystore
        SSLEngine engine = null;
        try {
            engine = SslContextFactory.createServerSslContext().createSSLEngine();
            engine.setUseClientMode(false);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return new SslHandler(engine);
    }
}

// Dummy hello-world response generator
public class HttpHandler extends ChannelHandlerAdapter {
    private static final Logger log = LoggerFactory.getLogger( HttpHandler.class );

    @Override
    public void channelReadComplete( ChannelHandlerContext context ) {
        context.flush();
    }

    @Override
    public void channelRead( ChannelHandlerContext context, Object message ) {
        if( message instanceof HttpRequest ) {
            HttpRequest request = (HttpRequest) message;
            log.info(request.toString());

            HttpResponse response = helloWorldResponse();
            log.info( response.toString() );

            context.write( response );
        }
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
        cause.printStackTrace();
        ctx.close();
    }

    private HttpResponse helloWorldResponse() {
        byte[] content = "Hello, World".getBytes(StandardCharsets.UTF_8);
        FullHttpResponse response = new DefaultFullHttpResponse(HttpVersion.HTTP_1_1,
                HttpResponseStatus.OK, Unpooled.wrappedBuffer( content ));
        response.headers().set( HttpHeaders.Names.CONTENT_TYPE, "text/plain");
        response.headers().set( HttpHeaders.Names.CONTENT_LENGTH, response.content().readableBytes());

        response.headers().set(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.KEEP_ALIVE);

        return response;
    }
}

// Server bootstrap
public class HttpServer {
    final int port = 1119;
    public void run() throws InterruptedException {
        EventLoopGroup bossGroup = new NioEventLoopGroup();
        EventLoopGroup workerGroup = new NioEventLoopGroup();

        try {
            ServerBootstrap b = new ServerBootstrap();
            b.option(ChannelOption.SO_BACKLOG, 1024);
            b.group(bossGroup, workerGroup)
                    .channel(NioServerSocketChannel.class)
                    .childHandler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        public void initChannel(SocketChannel ch) throws Exception {
                            ch.pipeline().addLast(new InboundFrontHandler());
                        }
                    });

            Channel ch = b.bind(port).sync().channel();
            ch.closeFuture().sync();
        } finally {
            bossGroup.shutdownGracefully();
            workerGroup.shutdownGracefully();
        }
    }
}

当我运行代理并在 Chrome 中输入网址 https://www.google.com 时，我收到以下日志:

2014-10-19/22:01:25.139 [nioEventLoopGroup-3-1] DEBUG i.n.u.ResourceLeakDetector.debug(): -Dio.netty.leakDetectionLevel: simple
    2014-10-19/22:01:25.147 [nioEventLoopGroup-3-1] INFO  c.t.n.p.InboundFrontHandler.decode(): Decoding message.
    2014-10-19/22:01:25.149 [nioEventLoopGroup-3-1] INFO  c.t.n.p.InboundFrontHandler.decode(): Received message: CONNECT www.google.com:443 HTTP/1.1
    Host: www.google.com
    Proxy-Connection: keep-alive
    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36


    2014-10-19/22:01:25.151 [nioEventLoopGroup-3-1] INFO  c.t.n.p.InboundFrontHandler.decode(): message is not encrypted
    2014-10-19/22:01:25.168 [nioEventLoopGroup-3-1] DEBUG i.n.u.i.JavassistTypeParameterMatcherGenerator.debug(): Generated: io.netty.util.internal.__matchers__.io.netty.handler.codec.http.HttpRequestMatcher
    2014-10-19/22:01:25.169 [nioEventLoopGroup-3-1] DEBUG i.n.u.i.JavassistTypeParameterMatcherGenerator.debug(): Generated: io.netty.util.internal.__matchers__.io.netty.handler.codec.http.HttpObjectMatcher
    2014-10-19/22:01:25.171 [nioEventLoopGroup-3-1] INFO  c.t.n.p.InboundFrontHandler.decode(): responded CONNECT method with DefaultHttpResponse(decodeResult: success)
    HTTP/1.1 200 Connection established
    2014-10-19/22:01:55.129 [nioEventLoopGroup-3-1] INFO  c.t.n.p.InboundFrontHandler.decode(): Decoding message.
    2014-10-19/22:01:55.129 [nioEventLoopGroup-3-1] INFO  c.t.n.p.InboundFrontHandler.decode(): Received message: CONNECT www.google.com:443 HTTP/1.1
    Host: www.google.com
    Proxy-Connection: keep-alive
    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36


    2014-10-19/22:01:55.129 [nioEventLoopGroup-3-1] INFO  c.t.n.p.InboundFrontHandler.decode(): message is not encrypted
    2014-10-19/22:01:55.134 [nioEventLoopGroup-3-1] WARN  i.n.c.DefaultChannelPipeline.warn(): An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
    io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: Duplicate handler name: deflater
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:258) ~[netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:191) ~[netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelInactiveNow(ChannelHandlerInvokerUtil.java:46) ~[netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelInactive(DefaultChannelHandlerInvoker.java:77) [netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.channel.DefaultChannelHandlerContext.fireChannelInactive(DefaultChannelHandlerContext.java:299) [netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:827) [netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$5.run(AbstractChannel.java:544) [netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:318) [netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353) [netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:794) [netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
    Caused by: java.lang.IllegalArgumentException: Duplicate handler name: deflater
        at io.netty.channel.DefaultChannelPipeline.checkDuplicateName(DefaultChannelPipeline.java:949) [netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.channel.DefaultChannelPipeline.addLast(DefaultChannelPipeline.java:141) [netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at io.netty.channel.DefaultChannelPipeline.addLast(DefaultChannelPipeline.java:130) [netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        at com.tao.netty.proxy.InboundFrontHandler.decode(InboundFrontHandler.java:66) ~[main/:na]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:227) ~[netty-all-5.0.0.Alpha1.jar:5.0.0.Alpha1]
        ... 10 common frames omitted

有几件事我不明白。首先，从日志中可以看到，浏览器重试了 CONNECT 请求，这表明它没有将代理发回的 CONNECT_RESPONSE 视为上次 CONNECT 请求的成功。其次，代码给出了这个异常消息:

Caused by: java.lang.IllegalArgumentException: Duplicate handler name: deflater

从以下 block 中抛出:

        if( pipeline.get( HTTP_CODEC_HANDLER ) == null ) {
            pipeline.addLast( HTTP_CODEC_HANDLER, new HttpServerCodec());
            pipeline.addLast( DEFLATER_HANDLER, new HttpContentCompressor());  // <<< throws duplicate name exception after second CONNECT request
        }

看起来HTTP_CODEC_HANDLER不知何故变成了空，但DEFLATER_HANDLER在第二次CONNECT请求时却没有，这让我很困惑。

第三，当 Chrome 重试时，其地址栏显示以下字符串:data:,。在我看来，Chrome 期望从 CONNECT 请求的响应中获取一些数据。为什么？

我知道这篇文章需要做很多工作。所以，非常感谢您的阅读。

更新
下面是 Chrome 中的 data:, 内容的快照:

Answer 1

实际上，您正在将其前面的其他处理程序写入管道。所以如果你在InboundFrontHandler中添加这段代码

log.info(context.pipeline().toMap());
context.writeAndFlush(CONNECT_RESPONSE).addListener(new ChannelFutureListener() {
        @Override
        public void operationComplete(ChannelFuture future) throws Exception {
               log.info("WriteandFlush : " + future.isSuccess());
        }
});

the output would be 
INFO  test.InboundFrontHandler  - {InboundFrontHandler#0=test.InboundFrontHandler@1f060469, HttpRequestDecoder#0=io.netty.handler.codec.http.HttpRequestDecoder@1c3c34f6, HttpResponseEncoder#0=io.netty.handler.codec.http.HttpResponseEncoder@7333cf2a, DEFLATER_HANDLER=io.netty.handler.codec.http.HttpContentCompressor@3d7fa45b}
INFO  test.InboundFrontHandler  - WriteandFlush : false

我建议阅读示例:https://github.com/netty/netty/tree/master/example/src/main/java/io/netty/example

因为关于管道、处理程序的使用方式以及写入的完成方式存在很多错误。例如，如果没有刷新，数据将不会被传输。

Chrome 发送了另一个 Connect 请求，因为它没有收到 http 200 响应。另一个建议是使用 curl 来准确了解正在发送和接收的数据，从而使调试变得更加容易样本 :curl -x 127.0.0.1:1119“https://www.google.com/”--trace -