ios - iOS 中的点对点视频

Question

我正在尝试将一部 iPhone 上的实时视频缓冲区传输到另一部 iPhone(称为客户端 iPhone)以进行预览显示，并接受来自客户端 iPhone 的命令。我正在考虑实现这一目标的标准方法。我找到的最接近的东西是 Github 上的 AVCaptureMultipeerVideoDataOutput .

然而，它仍然使用 Multipeer 连接框架，我认为它仍然需要在两部 iPhone 上进行一些设置。我想要的是理想情况下两部 iPhone 都不需要设置，只要两部 iPhone 都启用了 Wifi(或者如果可能的话，蓝牙)，对等点应该在应用程序中相互识别并提示用户发现设备。实现此目的的标准方法是什么以及示例代码的任何链接？

编辑:我从头开始编写代码后通过 Multipeer 连接让它工作。截至目前，我通过将数据缩小和压缩为 jpeg 将像素缓冲区发送到对等设备。在远程设备上，我设置了 UIImage，我在其中每帧显示一次数据。但是我认为 UIKit 可能不是显示数据的最佳方式，即使图像很小。如何使用 OpenGLES 显示这些数据？在 Opengles 中可以直接解码 jpeg 吗？

Answer 1

评论:

As of now, I am sending the pixel buffers to peer device by downscaling & compressing the data as jpeg. On the remote device, I have UIImage setup where I display the data every frame time. However I think UIKit may not be the best way to display data, even though images are small.

事实证明，这是通过 Multipeer Connectivity 框架传输图像的最佳方式。我已经尝试了所有替代方案:

1. 我使用 VideoToolbox 压缩了帧。太慢了。
2. 我使用 Compression 压缩了帧。太慢了，但更好。

让我为#2 提供一些代码:

在传输图像数据的iOS设备上:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer,0);
    __block uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
    dispatch_async(self.compressionQueue, ^{
        uint8_t *compressed = malloc(sizeof(uint8_t) * 1228808);
        size_t compressedSize = compression_encode_buffer(compressed, 1228808, baseAddress, 1228808, NULL, COMPRESSION_ZLIB);
        NSData *data = [NSData dataWithBytes:compressed length:compressedSize];
        NSLog(@"Sending size: %lu", [data length]);
        dispatch_async(dispatch_get_main_queue(), ^{
            __autoreleasing NSError *err;
            [((ViewController *)self.parentViewController).session sendData:data toPeers:((ViewController *)self.parentViewController).session.connectedPeers withMode:MCSessionSendDataReliable error:&err];
        });
    });
    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}

在显示图像数据的 iOS 设备上:

typedef struct {
    size_t length;
    void *data;
} ImageCacheDataStruct;

- (void)session:(nonnull MCSession *)session didReceiveData:(nonnull NSData *)data fromPeer:(nonnull MCPeerID *)peerID
{
    NSLog(@"Receiving size: %lu", [data length]);
    uint8_t *original = malloc(sizeof(uint8_t) * 1228808);
    size_t originalSize = compression_decode_buffer(original, 1228808, [data bytes], [data length], NULL, COMPRESSION_ZLIB);

    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    CGContextRef newContext = CGBitmapContextCreate(original, 640, 480, 8, 2560, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
    CGImageRef newImage = CGBitmapContextCreateImage(newContext);

    UIImage *image = [[UIImage alloc] initWithCGImage:newImage scale:1 orientation:UIImageOrientationUp];

    CGContextRelease(newContext);
    CGColorSpaceRelease(colorSpace);
    CGImageRelease(newImage);

    if (image) {
        dispatch_async(dispatch_get_main_queue(), ^{
            [((ViewerViewController *)self.childViewControllers.lastObject).view.layer setContents:(__bridge id)image.CGImage];
        });
    }
}

虽然此代码在接收端生成原始质量的图像，但您会发现这对于实时播放来说太慢了。

最好的方法是:

在发送图像数据的 iOS 设备上:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

    CVPixelBufferLockBaseAddress(imageBuffer,0);
    uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
    size_t width = CVPixelBufferGetWidth(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer);

    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
    CGImageRef newImage = CGBitmapContextCreateImage(newContext);


    UIImage *image = [[UIImage alloc] initWithCGImage:newImage scale:1 orientation:UIImageOrientationUp];
    CGImageRelease(newImage);
    CGContextRelease(newContext);
    CGColorSpaceRelease(colorSpace);
    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);

    if (image) {
        NSData *data = UIImageJPEGRepresentation(image, 0.7);
        NSError *err;
        [((ViewController *)self.parentViewController).session sendData:data toPeers:((ViewController *)self.parentViewController).session.connectedPeers withMode:MCSessionSendDataReliable error:&err];
    }
}

在接收图像数据的iOS设备上:

- (void)session:(nonnull MCSession *)session didReceiveData:(nonnull NSData *)data fromPeer:(nonnull MCPeerID *)peerID
{
  dispatch_async(self.imageCacheDataQueue, ^{
        dispatch_semaphore_wait(self.semaphore, DISPATCH_TIME_FOREVER);
        const void *dataBuffer = [data bytes];
        size_t dataLength = [data length];
        ImageCacheDataStruct *imageCacheDataStruct = calloc(1, sizeof(imageCacheDataStruct));
        imageCacheDataStruct->data = (void*)dataBuffer;
        imageCacheDataStruct->length = dataLength;

        __block const void * kMyKey;
        dispatch_queue_set_specific(self.imageDisplayQueue, &kMyKey, (void *)imageCacheDataStruct, NULL);

        dispatch_sync(self.imageDisplayQueue, ^{
            ImageCacheDataStruct *imageCacheDataStruct = calloc(1, sizeof(imageCacheDataStruct));
            imageCacheDataStruct = dispatch_queue_get_specific(self.imageDisplayQueue, &kMyKey);
            const void *dataBytes = imageCacheDataStruct->data;
            size_t length = imageCacheDataStruct->length;
            NSData *imageData = [NSData dataWithBytes:dataBytes length:length];
            UIImage *image = [UIImage imageWithData:imageData];
            if (image) {
                dispatch_async(dispatch_get_main_queue(), ^{
                    [((ViewerViewController *)self.childViewControllers.lastObject).view.layer setContents:(__bridge id)image.CGImage];
                    dispatch_semaphore_signal(self.semaphore);
                });
            }
        });
    });
}

信号量和单独的 GCD 队列的原因很简单:您希望帧以相等的时间间隔显示。否则，视频起初似乎有时会放慢速度，然后为了 catch 速度而加快速度超过正常速度。我的方案确保每一帧都以相同的速度一个接一个地播放，而不管网络带宽瓶颈。