c++ - 读取 .oni 文件会导致乱序帧

Question

我的问题与 this one 重复.那里没有解决办法。为了找到解决方案并详细说明我的具体设置，下面显示了用于从 .oni 文件读取帧的函数。如果使用 Type == 2 运行此函数(即运行 # of RGBD 图像，其中 Criteria 是 #)，在 for 循环中运行此函数应该允许用户访问每个图像。但是，彩色图像和深度图像的索引不匹配并且乱序。这一直持续到 waitForAnyStream 对以下所有对 IMG_pipeline::listen(...) 的调用超时为止。

void IMG_pipeline::listen(int Type, int Criteria){
    int exitNumber;
    clock_t start = clock();
    double elapsedtime;
    openni::VideoFrameRef frame;
    int CurrentIMGCount=0;

    switch (Type){
    case 0:
    {
        exitNumber = -1;
        break;
    }
    case 1:
    {
        exitNumber = Criteria;
        break;
    }
    case 2:
    {
        exitNumber = -1;
        break;
    }
    }


    for (int i = 0;i!=exitNumber;i++){
        readyStream = -1;
        rc = openni::OpenNI::waitForAnyStream(streams, 2, &readyStream, SAMPLE_READ_WAIT_TIMEOUT);
        if (rc != openni::STATUS_OK)
        {
            printf("Wait failed! (timeout is %d ms)\n%s\n", SAMPLE_READ_WAIT_TIMEOUT, openni::OpenNI::getExtendedError());
            //break;
        }

        switch (readyStream)
        {
        case 0:
        {
            // Depth
            depth.readFrame(&frame);
            break;
        }
        case 1:
        {
            // Color
            color.readFrame(&frame);
            break;
        }
        default:
        {
            printf("Unexpected stream: %i\n", readyStream);
            continue;
        }
        }


        int Height = frame.getHeight();
        int Width = frame.getWidth();

        cvColor.release();
        cvX.release();
        cvY.release();
        cvZ.release();

        cvColor = cv::Mat(Height, Width, CV_8UC3);
        cvX = cv::Mat(Height, Width, CV_32F);
        cvY = cv::Mat(Height, Width, CV_32F);
        cvZ = cv::Mat(Height, Width, CV_32F);

        switch (frame.getVideoMode().getPixelFormat())
        {
        case openni::PIXEL_FORMAT_DEPTH_1_MM:
        case openni::PIXEL_FORMAT_DEPTH_100_UM:
        {
            openni::DepthPixel* pDepth = (openni::DepthPixel*)frame.getData();
            int k =0;
            for (int ri = 0; ri<Height; ri++)
            {
                for (int ci = 0; ci<Width; ci++)
                {
                    float pdepth_val = pDepth[k];
                    openni::CoordinateConverter::convertDepthToWorld(depth, (float)ri, (float)ci, pdepth_val, &cvX.at<float>(ri,ci), &cvY.at<float>(ri,ci), &cvZ.at<float>(ri,ci));
                    k++;
                }
            }
            TotalFrames[0]++;
            XYZCaptured = true;
            printf("Frame Index: %i \n", frame.getFrameIndex());
            printf("Depth Captured. \n");
            break;
        }
        case openni::PIXEL_FORMAT_RGB888:
        {
            cvColor.data = (uchar*)frame.getData();
            TotalFrames[1]++;
            ColorCaptured = true;
            printf("Frame Index: %i \n", frame.getFrameIndex());
            printf("Color Captured. \n");
            break;
        }
        default:
            printf("Unknown format \n");
        }

        printf("Frame extracted. \n");
        if (ColorCaptured && XYZCaptured){
            if (NewButNotRead == true){
                IMGsMissed++;


            }
            else
                NewButNotRead = true;

            ColorCaptured = false;
            XYZCaptured = false;
            RGBD_out.clear();
            RGBD_out.push_back(cvX);
            RGBD_out.push_back(cvY);
            RGBD_out.push_back(cvZ);
            RGBD_out.push_back(cvColor);
            CurrentIMGCount++;
            printf("Image overwritten. \n");

        }
        elapsedtime=(clock()-start)/((double)CLOCKS_PER_SEC);

        printf("Time since listen initiation: %f \n \n", elapsedtime);
        if (CurrentIMGCount ==Criteria && Type == 2)
            return;
        else if (elapsedtime>(double)Criteria && Type==0)
            return;

    }
    frame.release();

}

这是一个控制台输出示例:

Frame Index: 1 
Depth Captured. 
Frame extracted. 
Time since listen initiation: 0.004846 

Frame Index: 2 
Depth Captured. 
Frame extracted. 
Time since listen initiation: 0.011601 

Frame Index: 1 
Color Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.012640 

Depth frame count: 3 
Color frame count: 2 
Frame Index: 54 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000067 

Frame Index: 57 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.005878 

Depth frame count: 4 
Color frame count: 3 
Frame Index: 96 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000079 

Frame Index: 99 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.003628 

Depth frame count: 5 
Color frame count: 4 
Frame Index: 126 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000048 

Frame Index: 130 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.004782 

Depth frame count: 6 
Color frame count: 5 
Frame Index: 152 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000065 

Frame Index: 156 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.008294 

Depth frame count: 7 
Color frame count: 6 
Frame Index: 181 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000045 

Frame Index: 185 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.004095 

Depth frame count: 8 
Color frame count: 7 
Frame Index: 208 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000054 

Frame Index: 212 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.004242 

Depth frame count: 9 
Color frame count: 8 
Frame Index: 236 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000092 

Frame Index: 240 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.005918 

Depth frame count: 10 
Color frame count: 9 
Frame Index: 261 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000731 

Frame Index: 262 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000877 

Frame Index: 266 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.009347 

Depth frame count: 11 
Color frame count: 11 
Frame Index: 286 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000047 

Frame Index: 290 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.006080 

Depth frame count: 12 
Color frame count: 12 
Frame Index: 311 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000072 

Frame Index: 315 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.006453 

Depth frame count: 13 
Color frame count: 13 
Frame Index: 337 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000062 

Frame Index: 341 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007485 

Depth frame count: 14 
Color frame count: 14 
Frame Index: 367 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000042 

Frame Index: 371 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.003758 

Depth frame count: 15 
Color frame count: 15 
Frame Index: 390 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000073 

Frame Index: 395 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007917 

Depth frame count: 16 
Color frame count: 16 
Frame Index: 416 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000105 

Frame Index: 421 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007554 

Depth frame count: 17 
Color frame count: 17 
Frame Index: 453 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000060 

Frame Index: 458 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.006150 

Depth frame count: 18 
Color frame count: 18 
Frame Index: 481 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000074 

Frame Index: 486 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007169 

Depth frame count: 19 
Color frame count: 19 
Frame Index: 517 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000045 

Frame Index: 522 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.004196 

Depth frame count: 20 
Color frame count: 20 
Frame Index: 547 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000071 

Frame Index: 552 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007375 

Depth frame count: 21 
Color frame count: 21 
Frame Index: 625 
Color Captured. 
Frame extracted. 
Time since listen initiation: 0.000179 

Frame Index: 631 
Depth Captured. 
Frame extracted. 
Image overwritten. 
Time since listen initiation: 0.007922 

Depth frame count: 22 
Color frame count: 22 
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
    waitForStreams: timeout reached

Unexpected stream: -1

这是对 IMG_pipeline::listen(...) 的调用:

IMG_pipeline pip_inst;
std::string FileName = "/home/derek/Test Data/RGBD/RGBD_S2_R1";
int Type = 2;
int Criteria = 1;
std::vector<cv::Mat> OUT;
int NumMissedIMGs;

int Start;
int Stop;

pip_inst.connect(FileName);

while (true)
{
  pip_inst.listen(Type, Criteria);
  if (pip_inst.IsNewIMG()){
    OUT = pip_inst.GetImage();
    cv::imshow("Current Frame", OUT.at(3));
    char c = cv::waitKey(0);
    if (c == 'f')
    {
      printf("Depth frame count: %i \n", pip_inst.GetDepthFrameCount());
      printf("Color frame count: %i \n", pip_inst.GetColorFrameCount());

    }
    else
    {
      Start = pip_inst.GetColorFrameCount();
      break;
    }
    cv::destroyWindow("Current Frame");

  }


}

彩色图像也是交替的 R、G、B 色调。但是，我确信这与数据在 cv::Mat 中的顺序有关。

更有趣的是，通过许多帧调用 IMG_pipeline::listen(...) 与多次运行 IMG_pipeline::listen(...) 具有不同的索引结果，通过 .oni 文件递增。

Answer 1

因此，由于编译器之间的 uchar 差异，RGB 信息没有被正确复制。所以RGB信息的case语句改成这样:

case openni::PIXEL_FORMAT_RGB888:
        {
            openni::RGB888Pixel* imgbuffer = (openni::RGB888Pixel*)frame.getData();
            //cvColor.data = (uchar*)imgbuffer;
            memcpy( cvColor.data, imgbuffer, 3*frame.getHeight()*frame.getWidth()*sizeof(uint8_t));
            cv::cvtColor(cvColor,cvColor,cv::COLOR_BGR2RGB);
            TotalFrames[1]++;
            ColorCaptured = true;
            printf("Frame Index: %i \n", frame.getFrameIndex());
            printf("Color Captured. \n");
            break;
        }

此外，为了确保我能够捕捉到每一帧，我放慢了文件播放速度。有趣的是，打开 .oni 文件的播放速度比抓取帧索引快(我只能假设这是捕获它的速度)。不管怎样，这是用

Source.getPlaybackControl()->setSpeed(Ratio);

其中 Source 是我的设备，Ratio 是用户指定的 float 。希望这对将来的人有所帮助。