ios - 为什么 AVSampleBufferDisplayLayer 停止显示从 AVCaptureVideoDataOutput 的委托(delegate)中获取的 CMSampleBuffers？

Question

我想用 AVSampleBufferDisplayLayer 显示一些 CMSampleBuffer，但它在显示第一个示例后卡住。

我从 AVCaptureVideoDataOutputSampleBuffer 委托(delegate)中获取样本缓冲区:

-(void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CFRetain(sampleBuffer);
    [self imageToBuffer:sampleBuffer];
    CFRelease(sampleBuffer);
}

将它们放入向量中

-(void) imageToBuffer: (CMSampleBufferRef )source{
//buffers is defined as: std::vector<CMSampleBufferRef> buffers;
        CMSampleBufferRef newRef;
        CMSampleBufferCreateCopy(kCFAllocatorDefault, source, &newRef);
        buffers.push_back(newRef);
}

然后尝试通过 AVSampleBufferDisplayLayer(在另一个 ViewController 中)显示它们

AVSampleBufferDisplayLayer * displayLayer = [[AVSampleBufferDisplayLayer alloc] init];

    displayLayer.bounds = self.view.bounds;
    displayLayer.position = CGPointMake(CGRectGetMidX(self.displayOnMe.bounds), CGRectGetMidY(self.displayOnMe.bounds));
    displayLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;
    displayLayer.backgroundColor = [[UIColor greenColor] CGColor];

    [self.view.layer addSublayer:displayLayer];
    self.view.autoresizingMask = UIViewAutoresizingFlexibleWidth | UIViewAutoresizingFlexibleHeight;

    dispatch_queue_t queue = dispatch_queue_create("My queue", DISPATCH_QUEUE_SERIAL);
    [displayLayer setNeedsDisplay];
    [displayLayer requestMediaDataWhenReadyOnQueue:queue
                                        usingBlock:^{
                                            while ([displayLayer isReadyForMoreMediaData]) {

                                                if (samplesKey < buffers.size()) {
                                                    CMSampleBufferRef buf = buffers[samplesKey];
                                                    [displayLayer enqueueSampleBuffer:buffers[samplesKey]];
                                                    samplesKey++;

                                                }else
                                                {
                                                    [displayLayer stopRequestingMediaData];
                                                    break;
                                                }
                                            }

                                        }];

但它显示第一个样本然后卡住，什么也不做。

而我的视频数据输出设置如下:

//set up our output
self.videoDataOutput = [[AVCaptureVideoDataOutput alloc] init];
dispatch_queue_t queue = dispatch_queue_create("VideoQueue", DISPATCH_QUEUE_SERIAL);
[_videoDataOutput setSampleBufferDelegate:self queue:queue];
[_videoDataOutput setVideoSettings:[NSDictionary dictionaryWithObjectsAndKeys:
                                                [NSNumber numberWithInt:kCVPixelFormatType_32BGRA],(id)kCVPixelBufferPixelFormatTypeKey,
                                                nil]]; 

Answer 1

我在相同的上下文中遇到过这个问题，试图从 AVCaptureVideoDataOutput 获取输出并将其显示在 AVSampleDisplay 层中。

如果您的帧按显示顺序显示，则修复非常简单，只需在 CMSampleBufferRef 上设置立即显示标志即可。

获取委托(delegate)返回的样本缓冲区，然后...

CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, YES);
CFMutableDictionaryRef dict = (CFMutableDictionaryRef)CFArrayGetValueAtIndex(attachments, 0);

CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanTrue);

如果您的帧按编码器顺序(而非显示顺序)输出，则 CMSampleBuffer 上的时间戳需要零偏差并重新标记，以便第一帧时间戳等于时间 0。

 double pts = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer));

 // ptsStart is equal to the first frames presentationTimeStamp so playback starts from time 0.
 CMTime presentationTimeStamp = CMTimeMake((pts-ptsStart)*1000000,1000000);

 CMSampleBufferSetOutputPresentationTimeStamp(sampleBuffer, presentationTimeStamp);

更新:

当我使用零偏差方法时，我遇到了一些视频仍然播放不流畅的情况，我进一步调查了。正确答案似乎是使用您打算播放的第一帧的 PTS。

我的答案在这里，但我也会在这里发布。

Set rate at which AVSampleBufferDisplayLayer renders sample buffers

Timebase 需要设置为您要解码的第一帧的呈现时间戳 (pts)。我通过从所有后续点中减去初始点并将时基设置为 0 将第一帧的点索引为 0。无论出于何种原因，这不适用于某些视频。

你想要这样的东西(在解码调用之前调用):

CMTimebaseRef controlTimebase;
CMTimebaseCreateWithMasterClock( CFAllocatorGetDefault(), CMClockGetHostTimeClock(), &controlTimebase );

displayLayer.controlTimebase = controlTimebase;

// Set the timebase to the initial pts here
CMTimebaseSetTime(displayLayer.controlTimebase, CMTimeMake(ptsInitial, 1));
CMTimebaseSetRate(displayLayer.controlTimebase, 1.0);

为 CMSampleBuffer 设置 PTS...

CMSampleBufferSetOutputPresentationTimeStamp(sampleBuffer, presentationTimeStamp);

也许确保没有设置立即显示....

CFDictionarySetValue(dict, kCMSampleAttachmentKey_DisplayImmediately, kCFBooleanFalse);

这在 WWDC 2014 session 513 中有非常简短的介绍。