android - ARCore Android Studio 向相机 View 添加过滤器

Question

我正在尝试向 Android Studio SDK 上的 ARCore 添加过滤器，例如 sobel 边缘检测。

目前我正在使用

GLES20.glReadPixels(x, y, w, h, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, intBuffer);

每帧从 glsurfaceview 读取图像并使用 GPU-Image Android 添加过滤器。

但它会导致性能问题，例如丢帧并最终导致崩溃。在 ARCore 中访问图像数据的最佳方式是什么？

我看到 ARCore 的 BackgroundRenderer 正在渲染相机 View ，但我看不到它从哪里获取相机纹理。如何将过滤后的图像放回 ARCore 相机预览？

Answer 1

目前还没有真正好的方法来做到这一点。 They are aware of the limitation ，我个人希望他们尽快添加框架访问功能。

在此期间，this SO post可以帮助您进行帧捕获。我已经实现了 PBO 写入，并且我可以以大约 15 fps 的速度捕获帧，而没有太多可见的延迟。

For the time being, your best bet for accessing image data is probably drawing the texture to a renderbuffer and using glReadPixels into a persistent-mapped pixel unpack buffer. Use a fence sync to detect when the glReadPixels is complete.

Another option is to use a compute shader and write directly to a persistent-mapped SSBO. (Disregard persistent-mapped suggestion. I thought EXT_buffer_storage had broader support)

The later is possibly fewer copies (the renderbuffer pixels may still hit DRAM even if you invalidate it after the glReadPixels), but it's also a less-common code path and incurs render/compute changeovers so I don't have intuition about which approach would be more efficient.