ios - 如何为YCbCr图像创建vImage_CGImageFormat？

Question

我想创建一个库，该库可以将使用 AVCaptureDevice 捕获的任何视频帧转换为目标像素格式。因此，它必须支持kCVPixelFormatType_32BGRA，kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange，kCVPixelFormatType_420YpCbCr8BiPlanarFullRange和可选的kCVPixelFormatType_420YpCbCr8Planar。

为了提高性能，我想使用Accelerate框架，因为它包含丰富的conversion functions集。由于目标像素格式可能有所不同，并且是由库用户设置的，因此不妨使用通用vImageConvert_AnyToAny函数:

您可以使用vImage的vImageConvert_AnyToAny(:::: _ :)函数来
在任意的Core Video或Core Graphics图像数据之间转换
颜色空间和位深度。源图像和目标图像是
由一个或多个缓冲区描述。例如，Y'CbCr图像可能是
由一个包含亮度信息的缓冲区和一个缓冲区组成
包含色度信息。

要使用此功能，我必须创建vImageConverter来定义图像之间的转换。此类的构造方法需要使用vImage_CGImageFormat形式的源图像和目标图像格式描述:

vImage_CGImageFormat in_format = ...;
vImage_CGImageFormat out_format = ...; 
vImageConverterRef converter = vImageConverter_CreateWithCGImageFormat(&in_format, &out_format, NULL, kvImagePrintDiagnosticsToConsole, &err);

if( err == kvImageNoError )
{
    vImage_Buffer *src_planes = ...;
    vImage_Buffer *dst_planes = ...;

    err = vImageConvert_AnyToAny(converter, src_planes, dst_planes, NULL,  kvImagePrintDiagnosticsToConsole);
}

该代码基于此Apple文章: Building a Basic Conversion Workflow

对于 kCVPixelFormatType_32BGRA，这样的vImage_CGImageFormat很简单，并在 vImage_Utilities.h 中进行了描述:

/*!
 * @struct vImage_CGImageFormat
 * @abstract A pixel format
 * @discussion A vImage_CGImageFormat describes the ordering of the color channels, how many there are,
 * the size and type of the data in the color channels and whether the data is premultiplied by alpha or not.
 * This format mirrors the image format descriptors used by CoreGraphics to create things like CGImageRef and
 * CGBitmapContextRef.
 *
 * This vImage_CGImageFormat:
 *
 *  <pre>@textblock
 *      vImage_CGImageFormat format = {
 *          .bitsPerComponent = 8,
 *          .bitsPerPixel = 32,
 *          .colorSpace = CGColorSpaceCreateDeviceRGB(),                                    // don't forget to release this!
 *          .bitmapInfo = kCGImageAlphaPremultipliedFirst | kCGBitmapByteOrder32Little,
 *          .version = 0,                                                                   // must be 0
 *          .decode = NULL,
 *          .renderingIntent = kCGRenderingIntentDefault
 *      };
 *  @/textblock</pre>
 *
 * codes for a little endian ARGB8888 pixel, or what is called in the rest of vImage, BGRA8888. Note: for 16-
 * and 32-bits per component formats (int16_t, uint16_t, half-float, float) most vImage image filters assume
 * the data is in host-endian format. (The APIs in this header do not.) Host-endian is little endian for Intel
 * and ARM, big endian for PowerPC. If the data is not in host-endian format, then you may use
 * vImagePermuteChannels_ARGB8888 or vImageByteSwap_Planar16U to swap the image data byte ordering.
 *
 * Some examples:
 *  <pre>@textblock
 *      ARGB8888     ->  {8, 32, NULL, alpha first, 0, NULL, kCGRenderingIntentDefault}     alpha first = { kCGImageAlphaFirst, kCGImageAlphaPremultipliedFirst, kCGImageAlphaNoneSkipFirst }
 *      RGBA8888     ->  {8, 32, NULL, alpha last,  0, NULL, kCGRenderingIntentDefault}     alpha last  = { kCGImageAlphaLast,  kCGImageAlphaPremultipliedLast,  kCGImageAlphaNoneSkipLast }
 *      BGRA8888     ->  {8, 32, NULL, alpha first | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault}
 *      RGB888       ->  {8, 24, NULL, kCGImageAlphaNone | kCGBitmapByteOrderDefault, 0, NULL, kCGRenderingIntentDefault}
 *      RGB565       ->  {5, 16, NULL, kCGImageAlphaNone | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
 *      ARGB1555     ->  {5, 16, NULL, alpha first | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
 *      RGBA16F      ->  {16, 64, NULL, alpha last | kCGBitmapFloatComponents | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault }
 *      CMYK8888     ->  {8, 32, CGColorSpaceCreateDeviceCMYK(), kCGImageAlphaNone, 0, NULL, kCGRenderingIntentDefault  }
 *      ARGBFFFF premultiplied    ->  {32, 128, NULL, kCGImageAlphaPremultipliedFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
 *      ARGBFFFF not-premultiplied -> {32, 128, NULL, kCGImageAlphaFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
 *      ARGBFFFF, alpha = 1 ->        {32, 128, NULL, kCGImageAlphaNoneSkipFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
 *  @/textblock</pre>
 *
 *  Note that some of these formats, particularly RGB565 and 16F formats are supported by vImage but
 *  not necessarily CoreGraphics. They will be converted to a higher precision format as necessary by
 *  vImage in vImageCreateCGImageFromBuffer().
 *
 *  By C rules, uninitialized struct parameters are set to zero. The last three parameters are usually zero, so can usually be omitted.
 *
 *  <pre>@textblock
 *      vImage_CGImageFormat srgb888 = (vImage_CGImageFormat){
 *          .bitsPerComponent = 8,
 *          .bitsPerPixel = 24,
 *          .colorSpace = NULL,
 *          .bitmapInfo = kCGImageAlphaNone | kCGBitmapByteOrderDefault };
 *  @/textblock</pre>
 *
 * To understand how these various parameters relate to one another, we can look at the process of converting from
 * one vImage_CGImageFormat format to another:
 *
 *  1) transform endianness of src format given by bitmapInfo to host endian  (except 8 bitPerComponent content)
 *  2) remove decode array transformation, and up convert to a higher range format as necessary to preserve precision / range
 *  3) convert src colorspace to reference XYZ colorspace (may cause upconvert to preserve range / precision)
 *  4) convert XYZ to destination colorspace + rendering intent
 *  5) convert to destination precision (given by bitsPerComponent)
 *  6) deal with any alpha changes (given by bitmapInfo) or flattening that needs to occur
 *  7) Apply any channel reordering requested, if it didn't happen at an earlier step. (As indicated by src and dest bitmapInfo)
 *  8) Apply destination decode array
 *  9) Apply endianness transform given by dest bitmapInfo
 *
 * Clearly, for most common transformations not all steps need to occur and multiple steps can be collapsed into a compound operation.
 *
 *  @field  bitsPerComponent    The number of bits needed to represent one channel of data in one pixel. For ARGB8888, this would be 8. Expected values: {1, 2, 4, 5, 8, 10, 12, 16, 32}
 *  @field  bitsPerPixel        The number of bits needed to represent one pixel. For ARGB8888, this would be 32.
 *                              It is possible that bitsPerPixel > bitsPerComponent * number of components, but in practice this is rare.
 *                              The number of color components is given by the colorspace and the number of alpha components (0 or 1) is given by
 *                              by the bitmapInfo.
 *  @field  colorSpace          A description of how the pixel data in the image is positioned relative to a reference XYZ color space.
 *                                  See CoreGraphics/CGColorSpace.h.  Pass NULL as a shorthand for sRGB. The vImage_CGImageFormat is not
 *                                  capable of managing the memory held by the colorSpace. If you created the colorspace, you must
 *                                  be sure to release it before all references to it disappear from scope.
 *  @field  bitmapInfo          The CGBitmapInfo describing the color channels. See CoreGraphics/CGImage.h.
 *                                  ARGB8888 is kCGImageAlphaFirst | kCGBitmapByteOrderDefault
 *                                  BGRA8888 is kCGImageAlphaFirst | kCGBitmapByteOrder32Little
 *  @field  version             The struct is versioned for future expansion.  Pass 0 here.
 *  @field  decode              Prior to transformations caused by the colorspace, color channels are subject to a linear transformation.
 *                              This allows for a different range than the typical [0,1.0]. NULL indicates default behavior of [0,1.0]
 *                              range, and is what you should use if you don't understand this parameter. See description of CGImageCreate()
 *                              for a discussion of decode arrays. See also Decode Arrays section of Chapter 4.8 of the PDF specification.
 *                              The vImage_CGImageFormat is not capable of managing the memory held by the decode array. If you created a
 *                              decode array on the heap, you must be sure to release it before all references to it disappear from scope.
 *
 *  @field renderingIntent      See CGColorSpace.h. kCGRenderingIntentDefault is typical here. By convention, rendering intent changes that
 *                              are not accompanied by a colorspace change are ignored.
 */

我不明白如何为YCbCr像素格式创建vImage_CGImageFormat。

首先，我认为完全不支持此格式，但该格式有一些特殊功能: Understanding YpCbCr Image Formats和vImageConverter对多平面图像具有特殊功能，例如 vImageConverter_GetNumberOfSourceBuffers和 vImageConverter_GetSourceBufferOrder。在最后一个函数中，有许多 vImage Buffer Type Codes甚至 kvImageBufferTypeCode_CVPixelBuffer_YCbCr。

因此，似乎可以为YCbCr创建vImageConverter，并且我需要帮助以了解如何执行此操作。

Answer 1

vImageConverter_CreateWithCGImageFormat创建一个转换器，用于在两种Core Graphics格式之间进行转换。

在这种情况下，您不应该使用vImageConverter_CreateForCVToCGImageFormat将CVPixelBuffer(带有可以使用vImageCVImageFormat生成的vImageCVImageFormat_CreateWithCVPixelBuffer)转换为CGImage吗？ vImage_CGImageFormat是您的目标格式，因此您可以定义其属性。