ios - 将 GPUImage 与 AVVideoCompositing 结合使用

Question

我正在尝试结合使用 GPUImage 和 AVVideoCompositing 在两个视频之间实现实时色度键过滤器。天真地使用 CIImage imageFromCVPixelBuffer into CGImage into GPUImage out to CGImage to CIImage 到 CVPixelBuffer，效率非常低，并且会导致内存问题。

我注意到 GPUImage 框架中有纹理对象、渲染目标和帧缓冲区。我希望能够在 iOS 中利用 CVOpenGLESTextureCacheCreateTextureFromImage 将所有内容保存在 GPU 上。

我认为我不太了解框架的内部工作原理，因为我假设我可以在 GPUImageTextureInput 对象上设置一个过滤器链，然后获取过滤器的 renderTarget，这是一个 CVPixelBufferRef。下面的 renderTarget 始终为 nil，调用 imageFromCurrentFrameBuffer 会产生一个灰色框，这不是我的图像。

请注意，下面的示例不是色度键，而是单个视频上的简单亮度过滤器，以尝试证明这一概念。

@implementation MyCustomCompositor : NSObject <AVVideoCompositing>

- (instancetype)init
{
    self = [super init];
    if (self) {
        CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, [GPUImageContext sharedImageProcessingContext].context, NULL, &_textureCache);
    }
    return self;
}

- (NSDictionary<NSString *,id> *)requiredPixelBufferAttributesForRenderContext
{
    return @{(NSString *)kCVPixelBufferPixelFormatTypeKey : @[@(kCVPixelFormatType_32BGRA)],
             (NSString *)kCVPixelBufferOpenGLCompatibilityKey : @YES};
}

- (NSDictionary<NSString *,id> *)sourcePixelBufferAttributes
{
    return @{(NSString *)kCVPixelBufferPixelFormatTypeKey : @[@(kCVPixelFormatType_32BGRA)],
             (NSString *)kCVPixelBufferOpenGLCompatibilityKey : @YES};
}

- (void)startVideoCompositionRequest:(AVAsynchronousVideoCompositionRequest *)asyncVideoCompositionRequest
{
    @autoreleasepool {
        CVPixelBufferRef mePixelBuffer = [asyncVideoCompositionRequest sourceFrameByTrackID:200];
        CVPixelBufferLockBaseAddress(mePixelBuffer, kCVPixelBufferLock_ReadOnly);

        CVOpenGLESTextureRef meTextureRef = NULL;
        size_t width = CVPixelBufferGetWidth(mePixelBuffer);
        size_t height = CVPixelBufferGetHeight(mePixelBuffer);
        CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, _textureCache, mePixelBuffer, NULL, GL_TEXTURE_2D, GL_BGRA, (int)width, (int)height, GL_BGRA, GL_UNSIGNED_BYTE, 0, &meTextureRef);

        GPUImageTextureInput *meTextureInput = [[GPUImageTextureInput alloc] initWithTexture:CVOpenGLESTextureGetName(meTextureRef) size:CGSizeMake(width, height)];

        GPUImageBrightnessFilter *filter = [[GPUImageBrightnessFilter alloc] init];
        filter.brightness = 0.5;
        [meTextureInput addTarget:filter];

        [filter setFrameProcessingCompletionBlock:^(GPUImageOutput *imageOutput, CMTime time) {
            [asyncVideoCompositionRequest finishWithComposedVideoFrame:((GPUImageBrightnessFilter *)imageOutput).renderTarget];
        }];

        [meTextureInput processTextureWithFrameTime:kCMTimeZero];

        CFRelease(meTextureRef);
        CVOpenGLESTextureCacheFlush(_textureCache, 0);

        CVPixelBufferUnlockBaseAddress(mePixelBuffer, kCVPixelBufferLock_ReadOnly);
    }
}

我没有在 GPUImage 中使用 GPUMovieWriter 或视频 API，因为我需要对合成进行更精细的控制。组合可以由多个色度键指令组成，这些指令在不同的时间范围内引用不​​同的绿色视频叠加层，在我看来，GPUImage 中的电影 API 仅限于过滤整个视频文件.我还需要合成的能力来操纵轨道和混音。

我尝试过全神贯注地使用自定义着色器在 GL 中完成所有这些工作，但我想我会利用现有的框架，这些框架似乎可以完成我想做的事情。

Answer 1

我写了一个名为 GPUImageFrameInput 的类，它是从 GPUImageMovie 修改而来的。CVPixelBufferRef 是它的输入。它是这样的:

1. 将 sourcePixelBuffer 包装到 GPUImageFrameOutput
2. 创建包裹 destinationPixelBuffer 的 GPUImageFrameInput
3. 完成过滤器链
4. 完成

这里是关键代码。

// output 
// wrap the sourcePixelBuffer from the request
// it's modified from GPUImageVideoCamera
@interface GPUImageFrameOutput() {

}
-(void)processSourcePixelBuffer:(CVPixelBufferRef)pixelBuffer withSampleTime:(CMTime)currentSampleTime;
@end

@implement GPUImageFrameOutput()

-(void)processSourcePixelBuffer:(CVPixelBufferRef)pixelBuffer withSampleTime:(CMTime)currentSampleTime {
    runSynchronouslyOnVideoProcessingQueue(^{
        [GPUImageContext useImageProcessingContext];

        int bufferHeight = (int) CVPixelBufferGetHeight(movieFrame);
        int bufferWidth = (int) CVPixelBufferGetWidth(movieFrame);

        if (bufferHeight == 0 || bufferWidth == 0) {
            return;
        }

        // almost same as 
        // [GPUImageVideoCamera processVideoSampleBuffer:]
        // 
}

@end

// input 
// wrap the destinationPixelBuffer 
@interface GPUImageFrameInput() {
    CVPixelBufferRef targetBuffer;
    // ... others
}
@end

- (void)setPixelBuffer:(CVPixelBufferRef)buffer{
    targetBuffer = buffer;
}

- (CVOpenGLESTextureRef)createDataFBO {
    if (!movieFramebuffer) {
       glActiveTexture(GL_TEXTURE1);
       glGenFramebuffers(1, &movieFramebuffer);
       glBindFramebuffer(GL_FRAMEBUFFER, movieFramebuffer);
    }

    glBindFramebuffer(GL_FRAMEBUFFER, movieFramebuffer);
    glViewport(0, 0, (int)_videoSize.width, (int)_videoSize.height);

    CVOpenGLESTextureRef renderTexture = nil;

    if ([GPUImageContext supportsFastTextureUpload]) {
        CVBufferSetAttachment(targetBuffer, kCVImageBufferColorPrimariesKey, kCVImageBufferColorPrimaries_ITU_R_709_2, kCVAttachmentMode_ShouldPropagate);
        CVBufferSetAttachment(targetBuffer, kCVImageBufferYCbCrMatrixKey, kCVImageBufferYCbCrMatrix_ITU_R_601_4, kCVAttachmentMode_ShouldPropagate);
        CVBufferSetAttachment(targetBuffer, kCVImageBufferTransferFunctionKey, kCVImageBufferTransferFunction_ITU_R_709_2, kCVAttachmentMode_ShouldPropagate);

        CVOpenGLESTextureCacheCreateTextureFromImage (kCFAllocatorDefault, [[GPUImageContext sharedImageProcessingContext] coreVideoTextureCache],
                                                  targetBuffer,
                                                  NULL, // texture attributes
                                                  GL_TEXTURE_2D,
                                                  GL_RGBA, // opengl format
                                                  (int)CVPixelBufferGetWidth(targetBuffer),
                                                  (int)CVPixelBufferGetHeight(targetBuffer),
                                                  GL_BGRA, // native iOS format
                                                  GL_UNSIGNED_BYTE,
                                                  0,
                                                  &renderTexture);

        glBindTexture(CVOpenGLESTextureGetTarget(renderTexture), CVOpenGLESTextureGetName(renderTexture));
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

        glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, CVOpenGLESTextureGetName(renderTexture), 0);
    }
    else
    {
     //...
    }
    GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
    NSAssert(status == GL_FRAMEBUFFER_COMPLETE, @"Incomplete filter FBO: %d", status);

    return renderTexture;
}

然后你可以像其他人一样创建gpuimage链

[frameInput setPixelBuffer:destinationPixelBuffer];

for (...) {
    GPUImageMovieFrameOutput *output ...
    [output addTarget:filter atTextureLocation:index];
}
[filter addTarget:frameInput];

希望对您有所帮助!