c++ - 如何保存两台摄像机的数据而不影响它们的图像采集速度？

Question

我正在使用多光谱相机来收集数据。一个是近红外的，另一个是彩色的。不是两个摄像机，而是一个摄像机可以同时获取两种不同类型的图像。我可以使用一些API函数，例如J_Image_OpenStream。核心代码的两个部分如下所示。一个用于打开两个流(实际上它们在一个示例中，我必须使用它们，但是我对它们的含义不太清楚)并设置两个avi文件的保存路径并开始采集。

 // Open stream
 retval0 = J_Image_OpenStream(m_hCam[0], 0, reinterpret_cast<J_IMG_CALLBACK_OBJECT>(this), reinterpret_cast<J_IMG_CALLBACK_FUNCTION>(&COpenCVSample1Dlg::StreamCBFunc0), &m_hThread[0], (ViewSize0.cx*ViewSize0.cy*bpp0)/8);
if (retval0 != J_ST_SUCCESS) {
    AfxMessageBox(CString("Could not open stream0!"), MB_OK | MB_ICONEXCLAMATION);
    return;
}
TRACE("Opening stream0 succeeded\n");
retval1 = J_Image_OpenStream(m_hCam[1], 0, reinterpret_cast<J_IMG_CALLBACK_OBJECT>(this), reinterpret_cast<J_IMG_CALLBACK_FUNCTION>(&COpenCVSample1Dlg::StreamCBFunc1), &m_hThread[1], (ViewSize1.cx*ViewSize1.cy*bpp1)/8);
if (retval1 != J_ST_SUCCESS) {
    AfxMessageBox(CString("Could not open stream1!"), MB_OK | MB_ICONEXCLAMATION);
    return;
}
TRACE("Opening stream1 succeeded\n");

const char *filename0 = "C:\\Users\\shenyang\\Desktop\\test0.avi"; 
const char *filename1 = "C:\\Users\\shenyang\\Desktop\\test1.avi";
int fps = 10; //frame per second
int codec = -1;//choose the compression method

writer0 = cvCreateVideoWriter(filename0, codec, fps, CvSize(1296,966), 1);
writer1 = cvCreateVideoWriter(filename1, codec, fps, CvSize(1296,964), 1);

// Start Acquision
retval0 = J_Camera_ExecuteCommand(m_hCam[0], NODE_NAME_ACQSTART);
retval1 = J_Camera_ExecuteCommand(m_hCam[1], NODE_NAME_ACQSTART);


// Create two OpenCV named Windows used for displaying "BGR" and "INFRARED" images
cvNamedWindow("BGR");
cvNamedWindow("INFRARED");

另一个是两个流函数，它们看起来非常相似。

void COpenCVSample1Dlg::StreamCBFunc0(J_tIMAGE_INFO * pAqImageInfo)
{
if (m_pImg0 == NULL)
{
    // Create the Image:
    // We assume this is a 8-bit monochrome image in this sample
    m_pImg0 = cvCreateImage(cvSize(pAqImageInfo->iSizeX, pAqImageInfo->iSizeY), IPL_DEPTH_8U, 1);
}

// Copy the data from the Acquisition engine image buffer into the OpenCV Image obejct
memcpy(m_pImg0->imageData, pAqImageInfo->pImageBuffer, m_pImg0->imageSize);

// Display in the "BGR" window
cvShowImage("INFRARED", m_pImg0);

frame0 = m_pImg0;
cvWriteFrame(writer0, frame0);

}

void COpenCVSample1Dlg::StreamCBFunc1(J_tIMAGE_INFO * pAqImageInfo)
{
if (m_pImg1 == NULL)
{
    // Create the Image:
    // We assume this is a 8-bit monochrome image in this sample
    m_pImg1 = cvCreateImage(cvSize(pAqImageInfo->iSizeX, pAqImageInfo->iSizeY), IPL_DEPTH_8U, 1);
}

// Copy the data from the Acquisition engine image buffer into the OpenCV Image obejct
memcpy(m_pImg1->imageData, pAqImageInfo->pImageBuffer, m_pImg1->imageSize);

// Display in the "BGR" window
cvShowImage("BGR", m_pImg1);

frame1 = m_pImg1;
cvWriteFrame(writer1, frame1);
}

问题是我是否不保存avi文件，因为

/*writer0 = cvCreateVideoWriter(filename0, codec, fps, CvSize(1296,966), 1);
writer1 = cvCreateVideoWriter(filename1, codec, fps, CvSize(1296,964), 1);*/
//cvWriteFrame(writer0, frame0);
//cvWriteFrame(writer0, frame0);

在两个显示窗口中，类似地捕获图像，这意味着它们是同步的。但是，如果我必须将数据写入avi文件，由于两种图片的大小不同且尺寸太大，事实证明这会影响两台相机的获取速度，并且所拍摄的图片是不同步的。但是我无法创建如此大的缓冲区将全部数据存储在内存中，并且I/O设备的运行速度相当慢。我该怎么办？非常非常感谢你。

一些类变量是:

 public:
FACTORY_HANDLE  m_hFactory;             // Factory Handle
CAM_HANDLE      m_hCam[MAX_CAMERAS];    // Camera Handles
THRD_HANDLE     m_hThread[MAX_CAMERAS]; // Stream handles
char            m_sCameraId[MAX_CAMERAS][J_CAMERA_ID_SIZE]; // Camera IDs

IplImage        *m_pImg0 = NULL;        // OpenCV Images
IplImage        *m_pImg1 = NULL;        // OpenCV Images

CvVideoWriter* writer0;
IplImage *frame0;
CvVideoWriter* writer1;
IplImage *frame1;

BOOL OpenFactoryAndCamera();
void CloseFactoryAndCamera();
void StreamCBFunc0(J_tIMAGE_INFO * pAqImageInfo);
void StreamCBFunc1(J_tIMAGE_INFO * pAqImageInfo);
void InitializeControls();
void EnableControls(BOOL bIsCameraReady, BOOL bIsImageAcquiring);

Answer 1

录制没有帧丢失的视频的正确方法是隔离两个任务(帧获取和帧序列化)，以使它们不会相互影响(具体来说，这样序列化的波动不会占用捕获帧的时间) ，必须立即进行以防止丢帧)。

这可以通过将序列化(帧的编码并将其写入视频文件)委派给单独的线程，并使用某种同步队列将数据馈送到工作线程来实现。

以下是一个简单的示例，显示了如何完成此操作。由于我只有一台相机而不是您所拥有的相机，因此我将仅使用网络摄像头复制帧，但是一般原理也适用于您的情况。

样例代码

一开始我们有一些包括:

#include <opencv2/opencv.hpp>

#include <chrono>
#include <condition_variable>
#include <iostream>
#include <mutex>
#include <queue>
#include <thread>
// ============================================================================
using std::chrono::high_resolution_clock;
using std::chrono::duration_cast;
using std::chrono::microseconds;
// ============================================================================

同步队列

第一步是定义同步队列，我们​​将使用该队列与编写视频的辅助线程进行通信。

我们需要的主要功能是能够:

将新图像推送到队列中

从队列中弹出图像，等待队列为空。

完成后，可以取消所有待处理的流行音乐。

我们使用 std::queue 来保存 cv::Mat 实例，并使用 std::mutex 来提供同步。 std::condition_variable 用于在图像已插入队列(或设置了取消标志)时通知使用者，而简单的 bool 标志用于通知取消。

最后，我们使用空的 struct cancelled作为从 pop()引发的异常，因此我们可以通过取消队列来干净地终止worker。

// ============================================================================
class frame_queue
{
public:
    struct cancelled {};

public:
    frame_queue();

    void push(cv::Mat const& image);
    cv::Mat pop();

    void cancel();

private:
    std::queue<cv::Mat> queue_;
    std::mutex mutex_;
    std::condition_variable cond_;
    bool cancelled_;
};
// ----------------------------------------------------------------------------
frame_queue::frame_queue()
    : cancelled_(false)
{
}
// ----------------------------------------------------------------------------
void frame_queue::cancel()
{
    std::unique_lock<std::mutex> mlock(mutex_);
    cancelled_ = true;
    cond_.notify_all();
}
// ----------------------------------------------------------------------------
void frame_queue::push(cv::Mat const& image)
{
    std::unique_lock<std::mutex> mlock(mutex_);
    queue_.push(image);
    cond_.notify_one();
}
// ----------------------------------------------------------------------------
cv::Mat frame_queue::pop()
{
    std::unique_lock<std::mutex> mlock(mutex_);

    while (queue_.empty()) {
        if (cancelled_) {
            throw cancelled();
        }
        cond_.wait(mlock);
        if (cancelled_) {
            throw cancelled();
        }
    }

    cv::Mat image(queue_.front());
    queue_.pop();
    return image;
}
// ============================================================================

仓储 worker

下一步是定义一个简单的 storage_worker，它将负责从同步队列中取出帧，并将它们编码为视频文件，直到队列被取消为止。

我添加了简单的计时，因此我们对花费了多少时间编码帧以及对控制台进行简单的日志记录有所了解，因此我们对程序中发生的事情有所了解。

// ============================================================================
class storage_worker
{
public:
    storage_worker(frame_queue& queue
        , int32_t id
        , std::string const& file_name
        , int32_t fourcc
        , double fps
        , cv::Size frame_size
        , bool is_color = true);

    void run();

    double total_time_ms() const { return total_time_ / 1000.0; }

private:
    frame_queue& queue_;

    int32_t id_;

    std::string file_name_;
    int32_t fourcc_;
    double fps_;
    cv::Size frame_size_;
    bool is_color_;

    double total_time_;
};
// ----------------------------------------------------------------------------
storage_worker::storage_worker(frame_queue& queue
    , int32_t id
    , std::string const& file_name
    , int32_t fourcc
    , double fps
    , cv::Size frame_size
    , bool is_color)
    : queue_(queue)
    , id_(id)
    , file_name_(file_name)
    , fourcc_(fourcc)
    , fps_(fps)
    , frame_size_(frame_size)
    , is_color_(is_color)
    , total_time_(0.0)
{
}
// ----------------------------------------------------------------------------
void storage_worker::run()
{
    cv::VideoWriter writer(file_name_, fourcc_, fps_, frame_size_, is_color_);

    try {
        int32_t frame_count(0);
        for (;;) {
            cv::Mat image(queue_.pop());
            if (!image.empty()) {
                high_resolution_clock::time_point t1(high_resolution_clock::now());

                ++frame_count;
                writer.write(image);

                high_resolution_clock::time_point t2(high_resolution_clock::now());
                double dt_us(static_cast<double>(duration_cast<microseconds>(t2 - t1).count()));
                total_time_ += dt_us;

                std::cout << "Worker " << id_ << " stored image #" << frame_count
                    << " in " << (dt_us / 1000.0) << " ms" << std::endl;
            }
        }
    } catch (frame_queue::cancelled& /*e*/) {
        // Nothing more to process, we're done
        std::cout << "Queue " << id_ << " cancelled, worker finished." << std::endl;
    }
}
// ============================================================================

加工

最后，我们可以将所有这些放在一起。

我们首先初始化和配置视频源。然后，我们创建两个 frame_queue实例，每个图像流一个。接下来，我们创建两个 storage_worker实例，每个队列一个。为了使事情有趣，我为每个设置了不同的编解码器。

下一步是创建并启动工作线程，该工作线程将执行每个 run()的 storage_worker方法。准备好我们的消费者之后，我们就可以开始从相机捕获帧并将其提供给 frame_queue实例。如上所述，我只有一个来源，因此我将同一帧的拷贝插入两个队列。

注意:我需要使用 clone()的 cv::Mat方法进行深度复制，否则出于性能原因，我将插入对OpenCV VideoCapture使用的单个缓冲区的引用。那将意味着工作线程将获得对该单个镜像的引用，并且将不会有同步访问此共享镜像缓冲区。您还需要确保在您的方案中也不会发生这种情况。

一旦我们读取了适当数量的帧(您可以实现所需的任何其他类型的停止条件)，我们将取消工作队列，并等待工作线程完成。

最后，我们编写一些有关不同任务所需时间的统计信息。

// ============================================================================
int main()
{
    // The video source -- for me this is a webcam, you use your specific camera API instead
    // I only have one camera, so I will just duplicate the frames to simulate your scenario
    cv::VideoCapture capture(0);

    // Let's make it decent sized, since my camera defaults to 640x480
    capture.set(CV_CAP_PROP_FRAME_WIDTH, 1920);
    capture.set(CV_CAP_PROP_FRAME_HEIGHT, 1080);
    capture.set(CV_CAP_PROP_FPS, 20.0);

    // And fetch the actual values, so we can create our video correctly
    int32_t frame_width(static_cast<int32_t>(capture.get(CV_CAP_PROP_FRAME_WIDTH)));
    int32_t frame_height(static_cast<int32_t>(capture.get(CV_CAP_PROP_FRAME_HEIGHT)));
    double video_fps(std::max(10.0, capture.get(CV_CAP_PROP_FPS))); // Some default in case it's 0

    std::cout << "Capturing images (" << frame_width << "x" << frame_height
        << ") at " << video_fps << " FPS." << std::endl;

    // The synchronized queues, one per video source/storage worker pair
    std::vector<frame_queue> queue(2);

    // Let's create our storage workers -- let's have two, to simulate your scenario
    // and to keep it interesting, have each one write a different format
    std::vector <storage_worker> storage;
    storage.emplace_back(std::ref(queue[0]), 0
        , std::string("foo_0.avi")
        , CV_FOURCC('I', 'Y', 'U', 'V')
        , video_fps
        , cv::Size(frame_width, frame_height)
        , true);

    storage.emplace_back(std::ref(queue[1]), 1
        , std::string("foo_1.avi")
        , CV_FOURCC('D', 'I', 'V', 'X')
        , video_fps
        , cv::Size(frame_width, frame_height)
        , true);

    // And start the worker threads for each storage worker
    std::vector<std::thread> storage_thread;
    for (auto& s : storage) {
        storage_thread.emplace_back(&storage_worker::run, &s);
    }

    // Now the main capture loop
    int32_t const MAX_FRAME_COUNT(10);
    double total_read_time(0.0);
    int32_t frame_count(0);
    for (; frame_count < MAX_FRAME_COUNT; ++frame_count) {
        high_resolution_clock::time_point t1(high_resolution_clock::now());

        // Try to read a frame
        cv::Mat image;
        if (!capture.read(image)) {
            std::cerr << "Failed to capture image.\n";
            break;
        }

        // Insert a copy into all queues
        for (auto& q : queue) {
            q.push(image.clone());
        }        

        high_resolution_clock::time_point t2(high_resolution_clock::now());
        double dt_us(static_cast<double>(duration_cast<microseconds>(t2 - t1).count()));
        total_read_time += dt_us;

        std::cout << "Captured image #" << frame_count << " in "
            << (dt_us / 1000.0) << " ms" << std::endl;
    }

    // We're done reading, cancel all the queues
    for (auto& q : queue) {
        q.cancel();
    }

    // And join all the worker threads, waiting for them to finish
    for (auto& st : storage_thread) {
        st.join();
    }

    if (frame_count == 0) {
        std::cerr << "No frames captured.\n";
        return -1;
    }

    // Report the timings
    total_read_time /= 1000.0;
    double total_write_time_a(storage[0].total_time_ms());
    double total_write_time_b(storage[1].total_time_ms());

    std::cout << "Completed processing " << frame_count << " images:\n"
        << "  average capture time = " << (total_read_time / frame_count) << " ms\n"
        << "  average write time A = " << (total_write_time_a / frame_count) << " ms\n"
        << "  average write time B = " << (total_write_time_b / frame_count) << " ms\n";

    return 0;
}
// ============================================================================

控制台输出

运行此小示例，我们将在控制台中获得以下日志输出，以及磁盘上的两个视频文件。

注意:既然这实际上比捕获要快很多，所以我在storage_worker中添加了一些等待时间以更好地显示分离。

Capturing images (1920x1080) at 20 FPS.
Captured image #0 in 111.009 ms
Captured image #1 in 67.066 ms
Worker 0 stored image #1 in 94.087 ms
Captured image #2 in 62.059 ms
Worker 1 stored image #1 in 193.186 ms
Captured image #3 in 60.059 ms
Worker 0 stored image #2 in 100.097 ms
Captured image #4 in 78.075 ms
Worker 0 stored image #3 in 87.085 ms
Captured image #5 in 62.061 ms
Worker 0 stored image #4 in 95.092 ms
Worker 1 stored image #2 in 193.187 ms
Captured image #6 in 75.074 ms
Worker 0 stored image #5 in 95.093 ms
Captured image #7 in 63.061 ms
Captured image #8 in 64.061 ms
Worker 0 stored image #6 in 102.098 ms
Worker 1 stored image #3 in 201.195 ms
Captured image #9 in 76.074 ms
Worker 0 stored image #7 in 90.089 ms
Worker 0 stored image #8 in 91.087 ms
Worker 1 stored image #4 in 185.18 ms
Worker 0 stored image #9 in 82.08 ms
Worker 0 stored image #10 in 94.092 ms
Queue 0 cancelled, worker finished.
Worker 1 stored image #5 in 179.174 ms
Worker 1 stored image #6 in 106.102 ms
Worker 1 stored image #7 in 105.104 ms
Worker 1 stored image #8 in 103.101 ms
Worker 1 stored image #9 in 104.102 ms
Worker 1 stored image #10 in 104.1 ms
Queue 1 cancelled, worker finished.
Completed processing 10 images:
  average capture time = 71.8599 ms
  average write time A = 93.09 ms
  average write time B = 147.443 ms
  average write time B = 176.673 ms

可能的改进

当前，当序列化根本无法跟上相机生成新图像的速度时，就无法防止队列太满。为队列大小设置一些上限，并在推送框架之前检入生产者。您将需要确定要如何精确地处理这种情况。