java - OpenCV + OpenGL : copy gl texture, 用 OpenCV 修改并渲染

Question

我正在编写一个使用 CameraX 将图像从相机下载到 SurfaceTexture 的 Android 应用程序。 ，然后将纹理绑定(bind)到 OpenGL 的纹理以在 GLSurfaceView 上执行进一步渲染.我想对 onDrawFrame 中的帧进行“注入(inject)”一些修改使用 OpenCV。

我考虑将 GL 像素缓冲区复制到 cv Mat，然后对其进行转换，然后使用 glTexSubImage2D 将其复制回纹理.问题是，我的代码不起作用，但会抛出 cv::error(): OpenCV(4.2.0) Error: Assertion failed (!_src.empty()) in cvtColor , 在 device/generic/goldfish-opengl/system/GLESv2_enc/GL2Encoder.cpp:s_glReadPixels:4413 GL error 0x500 之后已打印，因此我怀疑将像素复制到缓冲区的过程有问题。

有人可以告诉我，我在哪里弄乱了缓冲区？另外，如果有人有一个想法，如果有更好、更性能友好的方法来做到这一点，你能指出来吗？

我的代码:

@Override
public void onDrawFrame(GL10 gl) {
    GLES20.glClear( GLES20.GL_COLOR_BUFFER_BIT );

    surfaceTexture.updateTexImage();

    GLES20.glUseProgram(hProgram);

    int ph = GLES20.glGetAttribLocation(hProgram, "vPosition");
    int tch = GLES20.glGetAttribLocation ( hProgram, "vTexCoord" );
    int th = GLES20.glGetUniformLocation ( hProgram, "sTexture" );

    GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, hTex[0]);
    GLES20.glUniform1i(th, 0);

    GLES20.glVertexAttribPointer(ph, 2, GLES20.GL_FLOAT, false, 4*2, pVertex);
    GLES20.glVertexAttribPointer(tch, 2, GLES20.GL_FLOAT, false, 4*2, pTexCoord );
    GLES20.glEnableVertexAttribArray(ph);
    GLES20.glEnableVertexAttribArray(tch);

    Mat img = new Mat(this.bufferHeight, this.bufferWidth, CV_8UC3);

    GLES20.glPixelStorei(GL_PACK_ALIGNMENT, (img.step1() & 3) == 0 ? 1 : 4);

    GLES20.glPixelStorei(GL_PACK_ROW_LENGTH, (int)img.step1()/(int)img.elemSize());

    byte rarr[] = new byte[this.bufferWidth * this.bufferHeight];
    ByteBuffer buff = ByteBuffer.wrap(rarr);

    GLES20.glReadPixels(0, 0, img.cols(), img.rows(), GL_BGRA, GL_UNSIGNED_BYTE, buff);

    img.get(0, 0, buff.array());

        Imgproc.cvtColor(img, img, Imgproc.COLOR_BGR2GRAY);

    img.put(0, 0, buff.array());

    GLES20.glTexSubImage2D(hTex[0], 0, 0, 0, img.width(), img.height(),GL_BGRA,GL_UNSIGNED_BYTE, buff);

    GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, 4);
    GLES20.glFlush();
}

崩溃日志:

2020-02-01 23:05:48.690 15840-15912/com.myapp.app I/ReactNativeJS: Running "appApp" with {"rootTag":1}
2020-02-01 23:05:48.756 15840-15840/com.myapp.app D/appView: Camera permission granted
2020-02-01 23:05:48.879 15840-15929/com.myapp.app D/eglCodecCommon: setVertexArrayObject: set vao to 0 (0) 0 0
2020-02-01 23:05:48.879 15840-15929/com.myapp.app D/EGL_emulation: eglCreateContext: 0xd751b800: maj 2 min 0 rcv 2
2020-02-01 23:05:48.924 15840-15929/com.myapp.app D/EGL_emulation: eglMakeCurrent: 0xd751b800: ver 2 0 (tinfo 0xb4c801a0)
2020-02-01 23:05:48.926 15840-15929/com.myapp.app E/EGL_emulation: eglQueryContext 32c0  EGL_BAD_ATTRIBUTE
2020-02-01 23:05:48.926 15840-15929/com.myapp.app E/EGL_emulation: tid 15929: eglQueryContext(1902): error 0x3004 (EGL_BAD_ATTRIBUTE)
2020-02-01 23:05:48.926 15840-15929/com.myapp.app D/appView: appRenderer fired appRendererInitialized callback
2020-02-01 23:05:48.929 15840-15929/com.myapp.app W/appap: Core platform API violation: Ljava/nio/Buffer;->position:I from Landroid/opengl/GLES20; using JNI
2020-02-01 23:05:48.930 15840-15929/com.myapp.app W/appap: Core platform API violation: Ljava/nio/Buffer;->limit:I from Landroid/opengl/GLES20; using JNI
2020-02-01 23:05:48.930 15840-15929/com.myapp.app W/appap: Core platform API violation: Ljava/nio/Buffer;->_elementSizeShift:I from Landroid/opengl/GLES20; using JNI
2020-02-01 23:05:48.930 15840-15929/com.myapp.app W/appap: Core platform API violation: Ljava/nio/Buffer;->address:J from Landroid/opengl/GLES20; using JNI
2020-02-01 23:05:48.941 15840-15929/com.myapp.app E/emuglGLESv2_enc: device/generic/goldfish-opengl/system/GLESv2_enc/GL2Encoder.cpp:s_glPixelStorei:536 GL error 0x500
2020-02-01 23:05:48.942 15840-15929/com.myapp.app W/appap: Accessing hidden method Ljava/nio/NIOAccess;->getBaseArray(Ljava/nio/Buffer;)Ljava/lang/Object; (greylist, JNI, allowed)
2020-02-01 23:05:48.942 15840-15929/com.myapp.app W/appap: Accessing hidden method Ljava/nio/NIOAccess;->getBaseArrayOffset(Ljava/nio/Buffer;)I (greylist, JNI, allowed)
2020-02-01 23:05:48.942 15840-15929/com.myapp.app E/emuglGLESv2_enc: device/generic/goldfish-opengl/system/GLESv2_enc/GL2Encoder.cpp:s_glReadPixels:4413 GL error 0x500
2020-02-01 23:05:48.976 15840-15929/com.myapp.app E/cv::error(): OpenCV(4.2.0) Error: Assertion failed (!_src.empty()) in cvtColor, file /build/master_pack-android/opencv/modules/imgproc/src/color.cpp, line 182
2020-02-01 23:05:48.977 15840-15929/com.myapp.app E/org.opencv.imgproc: imgproc::cvtColor_11() caught cv::Exception: OpenCV(4.2.0) /build/master_pack-android/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
2020-02-01 23:05:48.980 15840-15929/com.myapp.app E/AndroidRuntime: FATAL EXCEPTION: GLThread 721
Process: com.myapp.app, PID: 15840
CvException [org.opencv.core.CvException: cv::Exception: OpenCV(4.2.0) /build/master_pack-android/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
]
    at org.opencv.imgproc.Imgproc.cvtColor_1(Native Method)
    at org.opencv.imgproc.Imgproc.cvtColor(Imgproc.java:5051)
    at com.myapp.app.app.ui.appRenderer.onDrawFrame(AppRenderer.java:209)
    at android.opengl.GLSurfaceView$GLThread.guardedRun(GLSurfaceView.java:1573)
    at android.opengl.GLSurfaceView$GLThread.run(GLSurfaceView.java:1272)

Answer 1

最后通过修复缓冲区的分辨率(大小)和更改 GL_BGRA 来完成这项工作。至GL_RGBA在 GLES20.glReadPixels ，这是唯一允许的与 GL_UNSIGNED_BYTE 的组合根据文档。现在垫子已正确填充。