android - 自定义 byteArray 数据到 WebRTC videoTrack

Question

我需要使用 WebRTC让 android 将特定的裁剪(面部)视频发送到 videoChannel。我能够操纵 Camera1Session类的 WebRTC 来裁剪脸部。现在我将它设置为 ImageView。Camera1Session.java的listenForBytebufferFrames()

private void listenForBytebufferFrames() {
    this.camera.setPreviewCallbackWithBuffer(new PreviewCallback() {
        public void onPreviewFrame(byte[] data, Camera callbackCamera) {
            Camera1Session.this.checkIsOnCameraThread();
            if(callbackCamera != Camera1Session.this.camera) {
                Logging.e("Camera1Session", "Callback from a different camera. This should never happen.");
            } else if(Camera1Session.this.state != Camera1Session.SessionState.RUNNING) {
                Logging.d("Camera1Session", "Bytebuffer frame captured but camera is no longer running.");
            } else {
                mFrameProcessor.setNextFrame(data, callbackCamera);
                long captureTimeNs = TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime());
                if(!Camera1Session.this.firstFrameReported) {
                    int startTimeMs = (int)TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - Camera1Session.this.constructionTimeNs);
                    Camera1Session.camera1StartTimeMsHistogram.addSample(startTimeMs);
                    Camera1Session.this.firstFrameReported = true;
                }

                ByteBuffer byteBuffer1 = ByteBuffer.wrap(data);
                Frame outputFrame = new Frame.Builder()
                        .setImageData(byteBuffer1,
                                Camera1Session.this.captureFormat.width,
                                Camera1Session.this.captureFormat.height,
                                ImageFormat.NV21)
                        .setTimestampMillis(mFrameProcessor.mPendingTimeMillis)
                        .setId(mFrameProcessor.mPendingFrameId)
                        .setRotation(3)
                        .build();
                int w = outputFrame.getMetadata().getWidth();
                int h = outputFrame.getMetadata().getHeight();
                SparseArray<Face> detectedFaces = mDetector.detect(outputFrame);
                if (detectedFaces.size() > 0) {

                    Face face = detectedFaces.valueAt(0);
                    ByteBuffer byteBufferRaw = outputFrame.getGrayscaleImageData();
                    byte[] byteBuffer = byteBufferRaw.array();
                    YuvImage yuvimage  = new YuvImage(byteBuffer, ImageFormat.NV21, w, h, null);
                    ByteArrayOutputStream baos = new ByteArrayOutputStream();

                    //My crop logic to get face co-ordinates

                    yuvimage.compressToJpeg(new Rect(left, top, right, bottom), 80, baos);
                    final byte[] jpegArray = baos.toByteArray();
                    Bitmap bitmap = BitmapFactory.decodeByteArray(jpegArray, 0, jpegArray.length);

                    Activity currentActivity = getActivity();
                    if (currentActivity instanceof CallActivity) {
                        ((CallActivity) currentActivity).setBitmapToImageView(bitmap); //face on ImageView is set just fine
                    }
                    Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, data, Camera1Session.this.captureFormat.width, Camera1Session.this.captureFormat.height, Camera1Session.this.getFrameOrientation(), captureTimeNs);
                    Camera1Session.this.camera.addCallbackBuffer(data);
                } else {
                    Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, data, Camera1Session.this.captureFormat.width, Camera1Session.this.captureFormat.height, Camera1Session.this.getFrameOrientation(), captureTimeNs);
                    Camera1Session.this.camera.addCallbackBuffer(data);
                }

            }
        }
    });
}

jpegArray 是我需要通过 WebRTC 流式传输的最终 byteArray，我尝试过这样的操作:

Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, jpegArray, (int) face.getWidth(), (int) face.getHeight(), Camera1Session.this.getFrameOrientation(), captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(jpegArray);

像这样设置它们会给我以下错误:

../../webrtc/sdk/android/src/jni/androidvideotracksource.cc line 82
Check failed: length >= width * height + 2 * uv_width * ((height + 1) / 2) (2630 vs. 460800)

我认为这是因为 androidvideotracksource 没有得到它期望的相同长度的 byteArray，因为帧现在被裁剪了。有人可以指出我如何实现它的方向吗？这是处理数据并输入 videoTrack 的正确方法/位置吗？

编辑:bitmap of byteArray data 不给我在 ImageView 上的相机预览，不像 byteArray jpegArray。可能是因为它们的包装方式不同？

Answer 1

我们能否使用 WebRTC 的数据通道来交换自定义数据，即在您的情况下裁剪的面部“图像”，并使用任何第三方库(即 OpenGL 等)在接收端进行相应的计算？我建议的原因是从 channel 接收的 WebRTC 视频提要是实时流而不是字节数组。另一方面，WebRTC Video 的固有架构并不意味着裁剪视频。如果我们想要裁剪或增强视频，我们必须使用任何 ar 库来完成这项工作。

我们始终可以利用 WebRTC 的数据通道来交换自定义数据。不推荐使用视频 channel ，因为它是实时流而不是字节数组。如有任何疑问，请回复。