android - 我如何在 android 上录制、加密(在内存中)和多路复用音频和视频而不会使文件不同步？

Question

我们正在尝试将 Android 设备中的视频和音频保存到加密文件中。我们当前的实现通过 MediaEncoder 类通过管道传输麦克风和摄像头的输出。当数据从 MediaEncoder 输出时，我们正在加密并将字节缓冲区的内容写入磁盘。但是，当尝试使用 FFMPEG 将文件重新拼接在一起时，这种方法有效，我们注意到这两个流似乎在流中间某处不同步。这种方法似乎丢失了许多重要的元数据，特别是演示时间戳和帧速率数据，因为 ffmpeg 必须做一些猜测工作才能混合文件。

是否有技术可以在不使用 MediaMuxer 的情况下保持这些流同步？视频使用 H.264 编码，音频使用 AAC。

其他方法:我们尝试使用 MediaMuxer 将输出数据多路复用到一个文件中，但我们的用例要求我们在将数据字节保存到磁盘之前对其进行加密，这消除了使用默认构造函数的可能性。

此外，我们尝试使用新添加的 (API 26) 构造函数，该构造函数采用 FileDescriptor 代替，并指向包装加密文档 (https://android.googlesource.com/platform/development/+/master/samples/Vault/src/com/example/android/vault/EncryptedDocument.java) 的 ParcelFileDescriptor。然而，这种方法导致了 native 层的崩溃，我们认为这可能与源代码 (https://android.googlesource.com/platform/frameworks/base.git/+/master/media/java/android/media/MediaMuxer.java#353) 中关于 native 编写器试图内存映射输出文件的评论有关。

import android.graphics.YuvImage
import android.media.MediaCodec
import android.media.MediaCodecInfo
import android.media.MediaFormat
import android.media.MediaMuxer
import com.callyo.video_10_21.Utils.YuvImageUtils.convertNV21toYUV420Planar
import java.io.FileDescriptor
import java.util.*
import java.util.concurrent.atomic.AtomicReference
import kotlin.properties.Delegates

class VideoEncoderProcessor(
   private val fileDescriptor: FileDescriptor,
   private val width: Int,
   private val height: Int,
   private val frameRate: Int
): MediaCodec.Callback() {
   private lateinit var videoFormat: MediaFormat
   private var trackIndex by Delegates.notNull<Int>()
   private var mediaMuxer: MediaMuxer
   private val mediaCodec = createEncoder()
   private val pendingVideoEncoderInputBufferIndices = AtomicReference<LinkedList<Int>>(LinkedList())

   companion object {
       private const val VIDEO_FORMAT = "video/avc"
   }

  init {
       mediaMuxer = MediaMuxer(fileDescriptor, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4)
       mediaCodec.setCallback(this)
       mediaCodec.start()
   }

   private fun createEncoder(): MediaCodec {
       videoFormat = MediaFormat.createVideoFormat(VIDEO_FORMAT, width, height).apply {
           setInteger(MediaFormat.KEY_FRAME_RATE, frameRate)
           setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible)
           setInteger(MediaFormat.KEY_BIT_RATE, width * height * 5)
           setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1)
       }

       return MediaCodec.createEncoderByType(VIDEO_FORMAT).apply {
           configure(videoFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
       }
   }

   override fun onInputBufferAvailable(codec: MediaCodec, index: Int) {
       // logic for handling stream end omitted for clarity

       /* Video frames come in asynchronously from input buffer availability
        * so we need to keep track of available buffers in queue */
       pendingVideoEncoderInputBufferIndices.get().add(index)
   }

   override fun onError(codec: MediaCodec, e: MediaCodec.CodecException) {}

   override fun onOutputFormatChanged(codec: MediaCodec, format: MediaFormat) {
       trackIndex = mediaMuxer.addTrack(format)
       mediaMuxer.start()
   }

   override fun onOutputBufferAvailable(codec: MediaCodec, index: Int, bufferInfo: MediaCodec.BufferInfo) {
       val buffer = mediaCodec.getOutputBuffer(index)
       buffer?.apply {
           if (bufferInfo.size != 0) {
               limit(bufferInfo.offset + bufferInfo.size)
               rewind()
               mediaMuxer.writeSampleData(trackIndex, this, bufferInfo)
           }
       }

       mediaCodec.releaseOutputBuffer(index, false)

       if (bufferInfo.flags == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
           mediaCodec.stop()
           mediaCodec.release()
           mediaMuxer.stop()
           mediaMuxer.release()
       }
   }

   // Public method that receives raw unencoded video data
   fun encode(yuvImage: YuvImage) {
       // logic for handling stream end omitted for clarity

       pendingVideoEncoderInputBufferIndices.get().poll()?.let { index ->
           val buffer = mediaCodec.getInputBuffer(index)
           buffer?.clear()
           // converting frame to correct color format
           val input =
                   yuvImage.convertNV21toYUV420Planar(ByteArray(yuvImage.yuvData.size), yuvImage.width, yuvImage.height)
           buffer?.put(input)
           buffer?.let {
               mediaCodec.queueInputBuffer(index, 0, input.size, System.nanoTime() / 1000, 0)
           }
       }
   }
}

---

附加信息:我正在使用 MediaCodec.Callback() ( https://developer.android.com/reference/kotlin/android/media/MediaCodec.Callback?hl=en ) 异步处理编码。

Answer 1

介绍

我将引用以下问答:sync audio and video with mediacodec and mediamuxer

由于信息丢失:

in order to sync audio and video you have to "calculate the number of audio samples that should play for each frame of video"

作者继续并提供了一个例子，例如

It depends on the sample rate and the frame rate:

at 24fps and 48000Hz every frame is long (48000hz/24fps)= 2000 sample

at 25 fps and 48000Hz: (48000hz/25fps)= 1920 sample

例子

看看下面这个混合视频和音频文件的示例，他们在其中设置样本大小并组合视频和音频(来自:https://github.com/Docile-Alligator/Infinity-For-Reddit/blob/61c5682b06fb3739a9f980700e6602ae0f39d5a2/app/src/main/java/ml/docilealligator/infinityforreddit/services/DownloadRedditVideoService.java#L506)

private boolean muxVideoAndAudio(String videoFilePath, String audioFilePath, String outputFilePath) {
    try {
        File file = new File(outputFilePath);
        file.createNewFile();
        MediaExtractor videoExtractor = new MediaExtractor();
        videoExtractor.setDataSource(videoFilePath);
        MediaExtractor audioExtractor = new MediaExtractor();
        audioExtractor.setDataSource(audioFilePath);
        MediaMuxer muxer = new MediaMuxer(outputFilePath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);

        videoExtractor.selectTrack(0);
        MediaFormat videoFormat = videoExtractor.getTrackFormat(0);
        int videoTrack = muxer.addTrack(videoFormat);

        audioExtractor.selectTrack(0);
        MediaFormat audioFormat = audioExtractor.getTrackFormat(0);
        int audioTrack = muxer.addTrack(audioFormat);
        boolean sawEOS = false;
        int offset = 100;
        int sampleSize = 2048 * 1024;
        ByteBuffer videoBuf = ByteBuffer.allocate(sampleSize);
        ByteBuffer audioBuf = ByteBuffer.allocate(sampleSize);
        MediaCodec.BufferInfo videoBufferInfo = new MediaCodec.BufferInfo();
        MediaCodec.BufferInfo audioBufferInfo = new MediaCodec.BufferInfo();

        videoExtractor.seekTo(0, MediaExtractor.SEEK_TO_CLOSEST_SYNC);
        audioExtractor.seekTo(0, MediaExtractor.SEEK_TO_CLOSEST_SYNC);

        muxer.start();

        while (!sawEOS) {
            videoBufferInfo.offset = offset;
            videoBufferInfo.size = videoExtractor.readSampleData(videoBuf, offset);

            if (videoBufferInfo.size < 0 || audioBufferInfo.size < 0) {
                sawEOS = true;
                videoBufferInfo.size = 0;
            } else {
                videoBufferInfo.presentationTimeUs = videoExtractor.getSampleTime();
                videoBufferInfo.flags = videoExtractor.getSampleFlags();
                muxer.writeSampleData(videoTrack, videoBuf, videoBufferInfo);
                videoExtractor.advance();
            }
        }

        boolean sawEOS2 = false;
        while (!sawEOS2) {
            audioBufferInfo.offset = offset;
            audioBufferInfo.size = audioExtractor.readSampleData(audioBuf, offset);

            if (videoBufferInfo.size < 0 || audioBufferInfo.size < 0) {
                sawEOS2 = true;
                audioBufferInfo.size = 0;
            } else {
                audioBufferInfo.presentationTimeUs = audioExtractor.getSampleTime();
                audioBufferInfo.flags = audioExtractor.getSampleFlags();
                muxer.writeSampleData(audioTrack, audioBuf, audioBufferInfo);
                audioExtractor.advance();

            }
        }

        try {
            muxer.stop();
            muxer.release();
        } catch (IllegalStateException ignore) {}
    } catch (IOException e) {
        e.printStackTrace();
        return false;
    }

    return true;
}

看看@以下页面:https://sisik.eu/blog/android/media/mix-audio-into-video

从那里开始，他们在部分中有一个很好的示例:使用 MediaMuxer 将帧混合到 MP4，您可以使用它来将文件拼接在一起。

从那里:

In my case, I want to get input from MPEG-4 video and from AAC/M4A audio file, > and mux both inputs into one MPEG-4 output video file. To accomplish that, Icreated the following mux() method

fun mux(audioFile: String, videoFile: String, outFile: String) {

    // Init extractors which will get encoded frames
    val videoExtractor = MediaExtractor()
    videoExtractor.setDataSource(videoFile)
    videoExtractor.selectTrack(0) // Assuming only one track per file. Adjust code if this is not the case.
    val videoFormat = videoExtractor.getTrackFormat(0)

    val audioExtractor = MediaExtractor()
    audioExtractor.setDataSource(audioFile)
    audioExtractor.selectTrack(0) // Assuming only one track per file. Adjust code if this is not the case.
    val audioFormat = audioExtractor.getTrackFormat(0)

    // Init muxer
    val muxer = MediaMuxer(outFile, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4)
    val videoIndex = muxer.addTrack(videoFormat)
    val audioIndex = muxer.addTrack(audioFormat)
    muxer.start()

    // Prepare buffer for copying
    val maxChunkSize = 1024 * 1024
    val buffer = ByteBuffer.allocate(maxChunkSize)
    val bufferInfo = MediaCodec.BufferInfo()

    // Copy Video
    while (true) {
        val chunkSize = videoExtractor.readSampleData(buffer, 0)

        if (chunkSize > 0) {
            bufferInfo.presentationTimeUs = videoExtractor.sampleTime
            bufferInfo.flags = videoExtractor.sampleFlags
            bufferInfo.size = chunkSize

            muxer.writeSampleData(videoIndex, buffer, bufferInfo)

            videoExtractor.advance()

        } else {
            break
        }
    }

    // Copy audio
    while (true) {
        val chunkSize = audioExtractor.readSampleData(buffer, 0)

        if (chunkSize >= 0) {
            bufferInfo.presentationTimeUs = audioExtractor.sampleTime
            bufferInfo.flags = audioExtractor.sampleFlags
            bufferInfo.size = chunkSize

            muxer.writeSampleData(audioIndex, buffer, bufferInfo)
            audioExtractor.advance()
        } else {
            break
        }
    }

    // Cleanup
    muxer.stop()
    muxer.release()

    videoExtractor.release()
    audioExtractor.release()
}

更新

根据您的评论，我认为主要问题是 fileDescriptor。具体来说，它们仅将 RandomAccessFile 用于文件描述符，但 native 接口(interface)是进行读取的接口(interface)。

那么我有一个建议，也许您应该考虑使用 内存中 而不是基于文件的 FileDescriptor。

因此，读取加密文件并在内存中对其进行解密，然后将这些字节转换为新的内存中文件描述符。将该内存中的文件描述符提供给多路复用器，看看会发生什么。

关于这个问题有一个很好的答案，他们使用安全的私有(private)应用专用套接字来创建文件描述符，请参阅:Create an in-memory FileDescriptor

从以下位置开始具体检查该答案的第二部分:

A better, but more complicated solution is to create a socket in thefilesystem namespace.Reference: https://stackoverflow.com/a/62651005/1688441

所以更详细:

1. 读取加密文件并解密成字节保存在内存中
2. 在您应用的私有(private)数据区域和服务器中创建一个 localSocket。
3. 开始监听您的服务器并接受未加密的字节。
4. 创建一个 localSocket 客户端并将未加密的字节发送到服务器。
5. 还将客户端的 fileDescriptor 传递给 muxor。

如回答所述:

This does create a file on the filesystem, but none of the data that passes through the socket is ever written to disk, it is entirelyin-memory. The file is just a name that represents the socket, similarto the files in /dev that represent devices. Because the socket isaccessed through the filesystem, it is subject to the usual filesystempermissions, so it is easy to restrict access to the socket by placingthe socket in your app's private data area.

Since this technique creates a file on the filesystem, it would be agood idea to delete the file after you're done, and also perhaps tocheck for and clean up old sockets every so often, in case your appcrashes and leaves old files laying around.