swift - 无需预乘即可将 CGImage 转换为 MTLTexture

Question

我有一个之前从 png 文件创建的 UIImage:

let strokeUIImage = UIImage(data: pngData)

我想将 strokeImage(具有不透明度)转换为 MTLTexture 以便在 MTKView 中显示，但进行转换似乎执行了不需要的预乘，这会使所有内容变暗半透明边缘。

我的混合设置如下:

pipelineDescriptor.colorAttachments[0].isBlendingEnabled = true
pipelineDescriptor.colorAttachments[0].rgbBlendOperation = .add
pipelineDescriptor.colorAttachments[0].alphaBlendOperation = .add
pipelineDescriptor.colorAttachments[0].sourceRGBBlendFactor = .one
pipelineDescriptor.colorAttachments[0].sourceAlphaBlendFactor = .one
pipelineDescriptor.colorAttachments[0].destinationRGBBlendFactor = .oneMinusSourceAlpha
pipelineDescriptor.colorAttachments[0].destinationAlphaBlendFactor = .oneMinusSourceAlpha

我试过两种转换方法:

let stampTexture = try! MTKTextureLoader(device: self.device!).newTexture(cgImage: strokeUIImage.cgImage!, options: nil)

以及更精细的 dataProvider 驱动方法:

let image = strokeUIImage.cgImage!
    let imageWidth = image.width
    let imageHeight = image.height
    let bytesPerPixel:Int! = 4
    let rowBytes = imageWidth * bytesPerPixel        

    let texDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm_srgb,
                                                              width: imageWidth,
                                                              height: imageHeight,
                                                              mipmapped: false)

    guard let stampTexture = device!.makeTexture(descriptor: texDescriptor) else { return }

    let srcData: CFData! = image.dataProvider?.data
    let pixelData = CFDataGetBytePtr(srcData)

    let region = MTLRegionMake2D(0, 0, imageWidth, imageHeight)
    stampTexture.replace(region: region, mipmapLevel: 0, withBytes: pixelData!, bytesPerRow: Int(rowBytes))

两者都会产生相同的不需要的预乘结果。

我尝试了后者，因为有一些帖子表明旧的 swift3 方法 CGDataProviderCopyData() 从未预乘的图像中提取原始像素数据。可悲的是，等价物:

let srcData: CFData! = image.dataProvider?.data

似乎并不能解决问题。我错过了什么吗？

如有任何指点，我们将不胜感激。

Answer 1

经过多次试验，我找到了解决 CoreGraphics 图像中固有的预乘问题的解决方案。感谢 Warren 关于使用 Accelerate 函数(特别是 vImageUnpremultiplyData_ARGB8888)的提示，我想，为什么不使用 vImage_CGImageFormat 构建 CGImage，这将允许我使用 bitmapInfo 设置指定如何解释 alpha...结果并不完美，如下图附件所示:

不知何故，在翻译中，alpha 值被稍微提高了，(可能是 rgb，但不是很明显)。顺便说一下，我应该指出 png 像素格式是 sRGB，我使用的 MTKView 设置为 MTLPixelFormat.rgba16Float(应用要求)

下面是我实现的完整 metalDrawStrokeUIImage 例程。特别值得注意的是这一行:

bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue)

这实际上是在不调用 vImageUnpremultiplyData_ARGB8888 的情况下取消关联 alpha(我认为)。查看生成的图像肯定看起来像未预乘的图像...

最后，为了在 MTKView 端取回预乘纹理，我让片段着色器处理预乘:

fragment float4 premult_fragment(VertexOut interpolated [[stage_in]],
                                 texture2d<float> texture [[texture(0)]],
                                 sampler sampler2D [[sampler(0)]]) {
  float4 sampled = texture.sample(sampler2D, interpolated.texCoord);

  // this fragment shader premultiplies incoming rgb with texture's alpha

  return float4(sampled.r * sampled.a,
                sampled.g * sampled.a,
                sampled.b * sampled.a,
                sampled.a );


} // end of premult_fragment

结果非常接近输入源，但图像可能比传入的 png 不透明 5%。同样，png 像素格式是 sRGB，我用来显示的 MTKView 设置为 MTLPixelFormat.rgba16Float 。所以，我确定某处有些东西变得糊涂了。如果有人有任何指示，我将不胜感激。

下面是相关代码的其余部分:

func metalDrawStrokeUIImage (strokeUIImage: UIImage, strokeBbox: CGRect) {

self.metalSetupRenderPipeline(compStyle: compMode.strokeCopy) // needed so stampTexture is not modified by fragmentFunction

let bytesPerPixel = 4
let bitsPerComponent = 8

let width = Int(strokeUIImage.size.width)
let height = Int(strokeUIImage.size.height)

let rowBytes = width * bytesPerPixel
//
let texDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm_srgb,
                                                             width: width,
                                                             height: height,
                                                             mipmapped: false)
guard let stampTexture = device!.makeTexture(descriptor: texDescriptor) else { return }

//let cgImage: CGImage = strokeUIImage.cgImage!
//let sourceColorSpace = cgImage.colorSpace else {
guard
  let cgImage = strokeUIImage.cgImage,
  let sourceColorSpace = cgImage.colorSpace else {
    print("Unable to initialize cgImage or colorSpace.")
    return
}

var format = vImage_CGImageFormat(
  bitsPerComponent: UInt32(cgImage.bitsPerComponent),
  bitsPerPixel: UInt32(cgImage.bitsPerPixel),
  colorSpace: Unmanaged.passRetained(sourceColorSpace),
  bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue),
  version: 0, decode: nil,
  renderingIntent: CGColorRenderingIntent.defaultIntent)
var sourceBuffer = vImage_Buffer()

defer {
  free(sourceBuffer.data)
}

var error = vImageBuffer_InitWithCGImage(&sourceBuffer, &format, nil, cgImage, numericCast(kvImageNoFlags))

guard error == kvImageNoError else {
  print ("[MetalBrushStrokeView]: can't vImageBuffer_InitWithCGImage")
  return

}

//vImagePremultiplyData_RGBA8888(&sourceBuffer, &sourceBuffer, numericCast(kvImageNoFlags))


// create a CGImage from vImage_Buffer
var destCGImage = vImageCreateCGImageFromBuffer(&sourceBuffer, &format, nil, nil, numericCast(kvImageNoFlags), &error)?.takeRetainedValue()


guard error == kvImageNoError else {
  print ("[MetalBrushStrokeView]: can't vImageCreateCGImageFromBuffer")
  return
}

let dstData: CFData = (destCGImage!.dataProvider!.data)!
let pixelData = CFDataGetBytePtr(dstData)

destCGImage = nil

let region = MTLRegionMake2D(0, 0, Int(width), Int(height))
stampTexture.replace(region: region, mipmapLevel: 0, withBytes: pixelData!, bytesPerRow: Int(rowBytes))
let stampColor = UIColor.white
let stampCorners = self.stampSetVerticesFromBbox(bbox: strokeBbox)
self.stampAppendToVertexBuffer(stampLayer: stampLayerMode.stampLayerFG, stampCorners: stampCorners, stampColor: stampColor)
self.metalRenderStampSingle(stampTexture: stampTexture)
self.initializeStampArray() // clears out the stamp array so we always draw 1 stamp at a time


} // end of func metalDrawStrokeUIImage (strokeUIImage: UIImage, strokeBbox: CGRect)