Android ImageReader YUV 420 888 重复数据

Question

我正在尝试将使用 Camera 2 API 从 ImageReader 接收到的 Image 转换为 OpenCV 矩阵，并使用 CameraBridgeViewBase 将其显示在屏幕上，更具体地说是函数 deliverAndDrawFrame。读者的 ImageFormat 是 YUV_420_888，据我所知，它有一个 Y 平面，每个像素都有灰度值，还有一个 U 平面，有 U/V每隔 4 个像素 1 个。但是，当我尝试显示此图像时，它看起来好像图像在重复并旋转了 90 度。下面的代码应该将 YUV 数据放入 OpenCV 矩阵(现在只是灰度，不是 rgba):

/**
 * Takes an {@link Image} in the {@link ImageFormat#YUV_420_888} and puts it into a provided {@link Mat} in rgba format.
 *
 * @param yuvImage {@link Image} in the {@link ImageFormat#YUV_420_888} format.
 */
public static void yuv420888imageToRgbaMat(final Image yuvImage, final Mat rgbaMat) {

    final Image.Plane
            Yp     = yuvImage.getPlanes()[0],
            UandVp = yuvImage.getPlanes()[1];

    final ByteBuffer
            Ybb     = Yp    .getBuffer(),
            UandVbb = UandVp.getBuffer();

    Ybb    .get(mYdata    , 0, 480*640        );
    UandVbb.get(mUandVData, 0, 480*640 / 2 - 8);

    for (int i = 0; i < 640*480; i++) {
        for (int j = 0; j < 4; j++) {
            mRawRGBAFrameData[i + 640*480*j] = mYdata[i];
        }
        mRawRGBAFrameData[i*4  ] = mYdata[i];
        mRawRGBAFrameData[i*4+1] = mYdata[i];
        mRawRGBAFrameData[i*4+2] = mYdata[i];
        mRawRGBAFrameData[i*4+3] = -1;
    }
}

这是我的 OpenCV 框架代码:

private class CameraFrame implements CvCameraViewFrame {

    private Mat mRgba;

    @Override
    public Mat gray() {

        return null;
    }

    @Override
    public Mat rgba() {

        mRgbaMat.put(0, 0, mRawRGBAFrameData);

        return mRgba;
    }

    public CameraFrame(final Mat rgba) {

        super();

        mRgba = rgba;
    }
}

接收画框的代码:

private final ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {

    @Override
    public void onImageAvailable(ImageReader reader) {

        final Image yuvImage = reader.acquireLatestImage();

        yuv420888imageToRgbaMat(yuvImage, mRgbaMat);

        deliverAndDrawFrame(mFrame);

        yuvImage.close();
    }
};

而且，这是制作图像阅读器的代码:

mRgbaMat = new Mat(mFrameHeight, mFrameWidth, CvType.CV_8UC4);

mFrame = new CameraFrame(mRgbaMat);

mImageReader = ImageReader.newInstance(mFrameWidth, mFrameHeight, ImageFormat.YUV_420_888, 1);

mImageReader.setOnImageAvailableListener(mOnImageAvailableListener, mBackgroundHandler);

AllocateCache();

这是数组的初始化:

protected static byte[] mRawRGBAFrameData = new byte[640*480*4], mYdata = new byte[640*480], mUandVData = new byte[640*480 / 2];

注意:mFrameWidth 是 480，mFrameHeight 是 640。一件奇怪的事情是 ImageReader 和 的高度和宽度从中接收到的图像具有倒置的尺寸。

这是带有上面代码的图像:/image/lcdzf.png

这是 yuv420888imageToRgbaMat 中的图片 /image/T2MOI.png

for (int i = 0; i < 640*480; i++) {
    mRawRGBAFrameData[i] = mYdata[i];
}

我们可以看到数据在 Y 帧中重复，出于某种原因，这给出了一个实际好看的图像。

Answer 1

对于尝试将 OpenCV 与 Camera 2 API 结合使用时遇到同样问题的任何人，我提出了一个解决方案。我发现的第一件事是 ImageReader 提供的 ByteBuffer 中有填充，所以如果不考虑它可能会导致输出失真.我选择做的另一件事是创建我自己的 SurfaceView 并使用 Bitmap 而不是使用 CameraViewBase 绘制到它，到目前为止它效果很好。 OpenCV 有一个函数 Util.matToBitmap，它接受 BGR 矩阵并将其转换为 android Bitmap，因此它很有用。我通过将来自 ImageReader 提供的前两个 Image.Plane 的信息放入格式化为 YUV 420 的 OpenCV 单 channel 矩阵并使用 Imgproc.cvtColor 与 Imgproc.COLOR_YUV420p2BGR。重要的是要知道图像的 Y 平面有完整的像素，但第二个 UV 平面有交错的像素映射一到四个 Y 像素，所以 UV 平面的总长度是 Y 平面的一半。参见 here .不管怎样，这里有一些代码:

矩阵的初始化

m_BGRMat = new Mat(Constants.VISION_IMAGE_HEIGHT, Constants.VISION_IMAGE_WIDTH, CvType.CV_8UC3);
m_Yuv420FrameMat = new Mat(Constants.VISION_IMAGE_HEIGHT * 3 / 2, Constants.VISION_IMAGE_WIDTH, CvType.CV_8UC1);

每一帧:

// Convert image to YUV 420 matrix
ImageUtils.imageToMat(image, m_Yuv420FrameMat, m_RawFrameData, m_RawFrameRowData);
// Convert YUV matrix to BGR matrix
Imgproc.cvtColor(m_Yuv420FrameMat, m_BGRMat, Imgproc.COLOR_YUV420p2BGR);
// Flip width and height then mirror vertically
Core.transpose(m_BGRMat, m_BGRMat);
Core.flip(m_BGRMat, m_BGRMat, 0);
// Draw to Surface View
m_PreviewView.drawImageMat(m_BGRMat);

这里是到 YUV 420 矩阵的转换:

/**
 * Takes an Android {@link Image} in the {@link ImageFormat#YUV_420_888} format and returns an OpenCV {@link Mat}.
 *
 * @param image {@link Image} in the {@link ImageFormat#YUV_420_888} format
 */
public static void imageToMat(final Image image, final Mat mat, byte[] data, byte[] rowData) {
    ByteBuffer buffer;
    int rowStride, pixelStride, width = image.getWidth(), height = image.getHeight(), offset = 0;
    Image.Plane[] planes = image.getPlanes();
    if (data == null || data.length != width * height) data = new byte[width * height * ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8];
    if (rowData == null || rowData.length != planes[0].getRowStride()) rowData = new byte[planes[0].getRowStride()];
    for (int i = 0; i < planes.length; i++) {
        buffer = planes[i].getBuffer();
        rowStride = planes[i].getRowStride();
        pixelStride = planes[i].getPixelStride();
        int
            w = (i == 0) ? width : width / 2,
            h = (i == 0) ? height : height / 2;
        for (int row = 0; row < h; row++) {
            int bytesPerPixel = ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8;
            if (pixelStride == bytesPerPixel) {
                int length = w * bytesPerPixel;
                buffer.get(data, offset, length);
                // Advance buffer the remainder of the row stride, unless on the last row.
                // Otherwise, this will throw an IllegalArgumentException because the buffer
                // doesn't include the last padding.
                if (h - row != 1)
                    buffer.position(buffer.position() + rowStride - length);
                offset += length;
            } else {
                // On the last row only read the width of the image minus the pixel stride
                // plus one. Otherwise, this will throw a BufferUnderflowException because the
                // buffer doesn't include the last padding.
                if (h - row == 1)
                    buffer.get(rowData, 0, width - pixelStride + 1);
                else
                    buffer.get(rowData, 0, rowStride);
                for (int col = 0; col < w; col++)
                    data[offset++] = rowData[col * pixelStride];
            }
        }
    }
    mat.put(0, 0, data);
}

最后，画画

/**
 * Given an {@link Mat} that represents a BGR image, draw it on the surface canvas.
 * use the OpenCV helper function {@link Utils#matToBitmap(Mat, Bitmap)} to create a {@link Bitmap}.
 *
 * @param bgrMat BGR frame {@link Mat}
 */
public void drawImageMat(final Mat bgrMat) {
    if (m_HolderReady) {
        // Create bitmap from BGR matrix
        Utils.matToBitmap(bgrMat, m_Bitmap);
        // Obtain the canvas and draw the bitmap on top of it
        final SurfaceHolder holder = getHolder();
        final Canvas canvas = holder.lockCanvas();
        canvas.drawBitmap(m_Bitmap, null, new Rect(0, 0, m_HolderWidth, m_HolderHeight), null);
        holder.unlockCanvasAndPost(canvas);
    }
}

这种方法可行，但我认为最好的方法是设置 OpenGL 渲染上下文并编写某种简单的着色器来显示矩阵。