{"id":404,"date":"2016-04-27T21:27:16","date_gmt":"2016-04-27T11:27:16","guid":{"rendered":"http:\/\/www.samontab.com\/web\/?p=404"},"modified":"2021-03-03T09:32:28","modified_gmt":"2021-03-02T22:32:28","slug":"interfacing-intel-realsense-f200-with-opencv","status":"publish","type":"post","link":"https:\/\/www.samontab.com\/web\/2016\/04\/interfacing-intel-realsense-f200-with-opencv\/","title":{"rendered":"Interfacing Intel RealSense F200 with OpenCV"},"content":{"rendered":"

RealSense from Intel is an interesting technology that brings 3D cameras into the mainstream.
\nAlthough the RealSense SDK provides a good starting point for many applications, some users would prefer to have a bit more control over the images. In this post, I’ll describe how to access the raw streams from an Intel RealSense F200 camera, and how to convert those images into OpenCV cv::Mat objects.<\/p>\n

Here is the video with all the explanations<\/a><\/p>\n

And here are the files used in the video:<\/p>\n

01-alignedSingleThreaded.cpp:<\/strong><\/p>\n

\n#include &lt;pxcsensemanager.h&gt;\n#include &lt;opencv2\/opencv.hpp&gt;\n\nPXCSenseManager *pxcSenseManager;\n\nPXCImage * CVMat2PXCImage(cv::Mat cvImage)\n{\n    PXCImage::ImageInfo iinfo;\n    memset(&amp;iinfo,0,sizeof(iinfo));\n    iinfo.width=cvImage.cols;\n    iinfo.height=cvImage.rows;\n\n    PXCImage::PixelFormat format;\n    int type = cvImage.type();\n    if(type == CV_8UC1)\n        format = PXCImage::PIXEL_FORMAT_Y8;\n    else if(type == CV_8UC3)\n        format = PXCImage::PIXEL_FORMAT_RGB24;\n    else if(type == CV_32FC1)\n        format = PXCImage::PIXEL_FORMAT_DEPTH_F32;\n\n    iinfo.format = format;\n\n    PXCImage *pxcImage = pxcSenseManager-&gt;QuerySession()-&gt;CreateImage(&amp;iinfo);\n\n    PXCImage::ImageData data;\n    pxcImage-&gt;AcquireAccess(PXCImage::ACCESS_WRITE, format, &amp;data);\n\n    data.planes[0] = cvImage.data;\n\n    pxcImage-&gt;ReleaseAccess(&amp;data);\n    return pxcImage;\n}\n\ncv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format)\n{\n    PXCImage::ImageData data;\n    pxcImage-&gt;AcquireAccess(PXCImage::ACCESS_READ, format, &amp;data);\n\n    int width = pxcImage-&gt;QueryInfo().width;\n    int height = pxcImage-&gt;QueryInfo().height;\n    if(!format)\n        format = pxcImage-&gt;QueryInfo().format;\n\n    int type;\n    if(format == PXCImage::PIXEL_FORMAT_Y8)\n        type = CV_8UC1;\n    else if(format == PXCImage::PIXEL_FORMAT_RGB24)\n        type = CV_8UC3;\n    else if(format == PXCImage::PIXEL_FORMAT_DEPTH_F32)\n        type = CV_32FC1;\n    else if(format == PXCImage::PIXEL_FORMAT_DEPTH)\n        type = CV_16UC1;\n\n    cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0]);\n\n    pxcImage-&gt;ReleaseAccess(&amp;data);\n    return ocvImage;\n}\n\nint main(int argc, char* argv[])\n{\n    \/\/Define some parameters for the camera\n    cv::Size frameSize = cv::Size(640, 480);\n    float frameRate = 60;\n\n    \/\/Create the OpenCV windows and images\n    cv::namedWindow(\"IR\", cv::WINDOW_NORMAL);\n    cv::namedWindow(\"Color\", cv::WINDOW_NORMAL);\n    cv::namedWindow(\"Depth\", cv::WINDOW_NORMAL);\n    cv::Mat frameIR = cv::Mat::zeros(frameSize, CV_8UC1);\n    cv::Mat frameColor = cv::Mat::zeros(frameSize, CV_8UC3);\n    cv::Mat frameDepth = cv::Mat::zeros(frameSize, CV_8UC1);\n\n    \/\/Initialize the RealSense Manager\n    pxcSenseManager = PXCSenseManager::CreateInstance();\n\n    \/\/Enable the streams to be used\n    pxcSenseManager-&gt;EnableStream(PXCCapture::STREAM_TYPE_IR, frameSize.width, frameSize.height, frameRate);\n    pxcSenseManager-&gt;EnableStream(PXCCapture::STREAM_TYPE_COLOR, frameSize.width, frameSize.height, frameRate);\n    pxcSenseManager-&gt;EnableStream(PXCCapture::STREAM_TYPE_DEPTH, frameSize.width, frameSize.height, frameRate);\n\n    \/\/Initialize the pipeline\n    pxcSenseManager-&gt;Init();\n\n    bool keepRunning = true;\n    while(keepRunning)\n    {\n        \/\/Acquire all the frames from the camera\n        pxcSenseManager-&gt;AcquireFrame();\n        PXCCapture::Sample *sample = pxcSenseManager-&gt;QuerySample();\n\n        \/\/Convert each frame into an OpenCV image\n        frameIR = PXCImage2CVMat(sample-&gt;ir, PXCImage::PIXEL_FORMAT_Y8);\n        frameColor = PXCImage2CVMat(sample-&gt;color, PXCImage::PIXEL_FORMAT_RGB24);\n        cv::Mat frameDepth_u16 = PXCImage2CVMat(sample-&gt;depth, PXCImage::PIXEL_FORMAT_DEPTH);\n        frameDepth_u16.convertTo(frameDepth, CV_8UC1);\n\n        cv::Mat frameDisplay;\n        cv::equalizeHist(frameDepth, frameDisplay);\n\n        \/\/Display the images\n        cv::imshow(\"IR\", frameIR);\n        cv::imshow(\"Color\", frameColor);\n        cv::imshow(\"Depth\", frameDisplay);\n\n        \/\/Check for user input\n        int key = cv::waitKey(1);\n        if(key == 27)\n            keepRunning = false;\n\n        \/\/Release the memory from the frames\n        pxcSenseManager-&gt;ReleaseFrame();\n    }\n\n    \/\/Release the memory from the RealSense manager\n    pxcSenseManager-&gt;Release();\n\n    return 0;\n}\n<\/pre>\n
02-unalignedSingleThreaded.cpp:<\/strong><\/p>\n
\n#include &lt;pxcsensemanager.h&gt;\n#include &lt;opencv2\/opencv.hpp&gt;\n\ncv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format)\n{\n    PXCImage::ImageData data;\n    pxcImage-&gt;AcquireAccess(PXCImage::ACCESS_READ, format, &amp;data);\n\n    int width = pxcImage-&gt;QueryInfo().width;\n    int height = pxcImage-&gt;QueryInfo().height;\n\n    if(!format)\n        format = pxcImage-&gt;QueryInfo().format;\n\n    int type;\n    if(format == PXCImage::PIXEL_FORMAT_Y8)\n        type = CV_8UC1;\n    else if(format == PXCImage::PIXEL_FORMAT_RGB24)\n        type = CV_8UC3;\n    else if(format == PXCImage::PIXEL_FORMAT_DEPTH_F32)\n        type = CV_32FC1;\n\n    cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0]);\n\n    pxcImage-&gt;ReleaseAccess(&amp;data);\n    return ocvImage;\n}\n\nint main(int argc, char* argv[])\n{\n    \/\/Define some parameters for the camera\n    cv::Size frameSize = cv::Size(640, 480);\n    float frameRate = 60;\n\n    \/\/Create the OpenCV windows and images\n    cv::namedWindow(\"IR\", cv::WINDOW_NORMAL);\n    cv::namedWindow(\"Color\", cv::WINDOW_NORMAL);\n    cv::namedWindow(\"Depth\", cv::WINDOW_NORMAL);\n    cv::Mat frameIR = cv::Mat::zeros(frameSize, CV_8UC1);\n    cv::Mat frameColor = cv::Mat::zeros(frameSize, CV_8UC3);\n    cv::Mat frameDepth = cv::Mat::zeros(frameSize, CV_8UC1);\n\n    \/\/Initialize the RealSense Manager\n    PXCSenseManager *pxcSenseManager = PXCSenseManager::CreateInstance();\n\n    \/\/Enable the streams to be used\n    pxcSenseManager-&gt;EnableStream(PXCCapture::STREAM_TYPE_IR, frameSize.width, frameSize.height, frameRate);\n    pxcSenseManager-&gt;EnableStream(PXCCapture::STREAM_TYPE_COLOR, frameSize.width, frameSize.height, frameRate);\n    pxcSenseManager-&gt;EnableStream(PXCCapture::STREAM_TYPE_DEPTH, frameSize.width, frameSize.height, frameRate);\n\n    \/\/Initialize the pipeline\n    pxcSenseManager-&gt;Init();\n\n    bool keepRunning = true;\n    while(keepRunning)\n    {\n        \/\/Acquire any frame from the camera\n        pxcSenseManager-&gt;AcquireFrame(false);\n        PXCCapture::Sample *sample = pxcSenseManager-&gt;QuerySample();\n\n        \/\/Convert each frame into an OpenCV image\n        \/\/You need to make sure that the image is there first\n        if(sample-&gt;ir)\n            frameIR = PXCImage2CVMat(sample-&gt;ir, PXCImage::PIXEL_FORMAT_Y8);\n        if(sample-&gt;color)\n            frameColor = PXCImage2CVMat(sample-&gt;color, PXCImage::PIXEL_FORMAT_RGB24);\n        if(sample-&gt;depth)\n            PXCImage2CVMat(sample-&gt;depth, PXCImage::PIXEL_FORMAT_DEPTH_F32).convertTo(frameDepth, CV_8UC1);\n\n        \/\/Display the images\n        cv::imshow(\"IR\", frameIR);\n        cv::imshow(\"Color\", frameColor);\n        cv::imshow(\"Depth\", frameDepth);\n\n        \/\/Check for user input\n        int key = cv::waitKey(1);\n        if(key == 27)\n            keepRunning = false;\n\n        \/\/Release the memory from the frames\n        pxcSenseManager-&gt;ReleaseFrame();\n    }\n\n    \/\/Release the memory from the RealSense manager\n    pxcSenseManager-&gt;Release();\n\n    return 0;\n}\n<\/pre>\n
03-unalignedMultiThreaded.cpp:<\/strong><\/p>\n
\n#include &lt;pxcsensemanager.h&gt;\n#include &lt;iostream&gt;\n#include &lt;opencv2\/opencv.hpp&gt;\n\ncv::Mat frameIR;\ncv::Mat frameColor;\ncv::Mat frameDepth;\ncv::Mutex framesMutex;\n\ncv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format)\n{\n    PXCImage::ImageData data;\n    pxcImage-&gt;AcquireAccess(PXCImage::ACCESS_READ, format, &amp;data);\n\n    int width = pxcImage-&gt;QueryInfo().width;\n    int height = pxcImage-&gt;QueryInfo().height;\n\n    if(!format)\n        format = pxcImage-&gt;QueryInfo().format;\n\n    int type;\n    if(format == PXCImage::PIXEL_FORMAT_Y8)\n        type = CV_8UC1;\n    else if(format == PXCImage::PIXEL_FORMAT_RGB24)\n        type = CV_8UC3;\n    else if(format == PXCImage::PIXEL_FORMAT_DEPTH_F32)\n        type = CV_32FC1;\n\n    cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0]);\n\n    pxcImage-&gt;ReleaseAccess(&amp;data);\n    return ocvImage;\n}\n\nclass FramesHandler:public PXCSenseManager::Handler\n{\npublic:\n    virtual pxcStatus PXCAPI OnNewSample(pxcUID, PXCCapture::Sample *sample)\n    {\n            framesMutex.lock();\n                if(sample-&gt;ir)\n                    frameIR = PXCImage2CVMat(sample-&gt;ir, PXCImage::PIXEL_FORMAT_Y8);\n                if(sample-&gt;color)\n                    frameColor = PXCImage2CVMat(sample-&gt;color, PXCImage::PIXEL_FORMAT_RGB24);\n                if(sample-&gt;depth)\n                    PXCImage2CVMat(sample-&gt;depth, PXCImage::PIXEL_FORMAT_DEPTH_F32).convertTo(frameDepth, CV_8UC1);\n            framesMutex.unlock();\n    return PXC_STATUS_NO_ERROR;\n    }\n};\n\nint main(int argc, char* argv[])\n{\n    cv::Size frameSize = cv::Size(640, 480);\n    float frameRate = 60;\n\n    cv::namedWindow(\"IR\", cv::WINDOW_NORMAL);\n    cv::namedWindow(\"Color\", cv::WINDOW_NORMAL);\n    cv::namedWindow(\"Depth\", cv::WINDOW_NORMAL);\n    frameIR = cv::Mat::zeros(frameSize, CV_8UC1);\n    frameColor = cv::Mat::zeros(frameSize, CV_8UC3);\n    frameDepth = cv::Mat::zeros(frameSize, CV_8UC1);\n\n    PXCSenseManager *pxcSenseManager = PXCSenseManager::CreateInstance();\n\n    \/\/Enable the streams to be used\n    pxcSenseManager-&gt;EnableStream(PXCCapture::STREAM_TYPE_IR, frameSize.width, frameSize.height, frameRate);\n    pxcSenseManager-&gt;EnableStream(PXCCapture::STREAM_TYPE_COLOR, frameSize.width, frameSize.height, frameRate);\n    pxcSenseManager-&gt;EnableStream(PXCCapture::STREAM_TYPE_DEPTH, frameSize.width, frameSize.height, frameRate);\n\n    FramesHandler handler;\n    pxcSenseManager-&gt;Init(&amp;handler);\n    pxcSenseManager-&gt;StreamFrames(false);\n\n    \/\/Local images for display\n    cv::Mat displayIR = frameIR.clone();\n    cv::Mat displayColor = frameColor.clone();\n    cv::Mat displayDepth = frameDepth.clone();\n\n    bool keepRunning = true;\n    while(keepRunning)\n    {\n        framesMutex.lock();\n            displayIR = frameIR.clone();\n            displayColor = frameColor.clone();\n            displayDepth = frameDepth.clone();\n        framesMutex.unlock();\n\n        cv::imshow(\"IR\", displayIR);\n        cv::imshow(\"Color\", displayColor);\n        cv::imshow(\"Depth\", displayDepth);\n\n        int key = cv::waitKey(1);\n        if(key == 27)\n            keepRunning = false;\n    }\n    \/\/Stop the frame acqusition thread\n    pxcSenseManager-&gt;Close();\n\n    pxcSenseManager-&gt;Release();\n\n    return 0;\n}\n<\/pre>\n
04-alignedMultiThreaded.cpp:<\/strong><\/p>\n
\n#include &lt;pxcsensemanager.h&gt;\n#include &lt;iostream&gt;\n#include &lt;opencv2\/opencv.hpp&gt;\n\ncv::Mat frameIR;\ncv::Mat frameColor;\ncv::Mat frameDepth;\ncv::Mutex framesMutex;\n\ncv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format)\n{\n    PXCImage::ImageData data;\n    pxcImage-&gt;AcquireAccess(PXCImage::ACCESS_READ, format, &amp;data);\n\n    int width = pxcImage-&gt;QueryInfo().width;\n    int height = pxcImage-&gt;QueryInfo().height;\n\n    if(!format)\n        format = pxcImage-&gt;QueryInfo().format;\n\n    int type;\n    if(format == PXCImage::PIXEL_FORMAT_Y8)\n        type = CV_8UC1;\n    else if(format == PXCImage::PIXEL_FORMAT_RGB24)\n        type = CV_8UC3;\n    else if(format == PXCImage::PIXEL_FORMAT_DEPTH_F32)\n        type = CV_32FC1;\n\n    cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0]);\n\n    pxcImage-&gt;ReleaseAccess(&amp;data);\n    return ocvImage;\n}\n\nclass FramesHandler:public PXCSenseManager::Handler\n{\npublic:\n    virtual pxcStatus PXCAPI OnNewSample(pxcUID, PXCCapture::Sample *sample)\n    {\n            framesMutex.lock();\n                frameIR = PXCImage2CVMat(sample-&gt;ir, PXCImage::PIXEL_FORMAT_Y8);\n                frameColor = PXCImage2CVMat(sample-&gt;color, PXCImage::PIXEL_FORMAT_RGB24);\n                PXCImage2CVMat(sample-&gt;depth, PXCImage::PIXEL_FORMAT_DEPTH_F32).convertTo(frameDepth, CV_8UC1);\n            framesMutex.unlock();\n    return PXC_STATUS_NO_ERROR;\n    }\n};\n\nint main(int argc, char* argv[])\n{\n    cv::Size frameSize = cv::Size(640, 480);\n    float frameRate = 60;\n\n    cv::namedWindow(\"IR\", cv::WINDOW_NORMAL);\n    cv::namedWindow(\"Color\", cv::WINDOW_NORMAL);\n    cv::namedWindow(\"Depth\", cv::WINDOW_NORMAL);\n    frameIR = cv::Mat::zeros(frameSize, CV_8UC1);\n    frameColor = cv::Mat::zeros(frameSize, CV_8UC3);\n    frameDepth = cv::Mat::zeros(frameSize, CV_8UC1);\n\n    PXCSenseManager *pxcSenseManager = PXCSenseManager::CreateInstance();\n\n    \/\/Enable the streams to be used\n    PXCVideoModule::DataDesc ddesc={};\n    ddesc.deviceInfo.streams = PXCCapture::STREAM_TYPE_IR | PXCCapture::STREAM_TYPE_COLOR | PXCCapture::STREAM_TYPE_DEPTH;\n\n    pxcSenseManager-&gt;EnableStreams(&amp;ddesc);\n\n    FramesHandler handler;\n    pxcSenseManager-&gt;Init(&amp;handler);\n    pxcSenseManager-&gt;StreamFrames(false);\n\n    \/\/Local images for display\n    cv::Mat displayIR = frameIR.clone();\n    cv::Mat displayColor = frameColor.clone();\n    cv::Mat displayDepth = frameDepth.clone();\n\n    bool keepRunning = true;\n    while(keepRunning)\n    {\n        framesMutex.lock();\n            displayIR = frameIR.clone();\n            displayColor = frameColor.clone();\n            displayDepth = frameDepth.clone();\n        framesMutex.unlock();\n\n        cv::imshow(\"IR\", displayIR);\n        cv::imshow(\"Color\", displayColor);\n        cv::imshow(\"Depth\", displayDepth);\n\n        int key = cv::waitKey(1);\n        if(key == 27)\n            keepRunning = false;\n    }\n    \/\/Stop the frame acqusition thread\n    pxcSenseManager-&gt;Close();\n\n    pxcSenseManager-&gt;Release();\n\n    return 0;\n}\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
RealSense from Intel is an interesting technology that brings 3D cameras into the mainstream. Although the RealSense SDK provides a good starting point for many applications, some users would prefer to have a bit more control over the images. In this post, I’ll describe how to access the raw streams from an Intel RealSense F200 […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29,22,30,6,4],"tags":[],"class_list":["post-404","post","type-post","status-publish","format-standard","hentry","category-computer-vision","category-iot","category-opencv","category-photography","category-programming"],"_links":{"self":[{"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/posts\/404","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/comments?post=404"}],"version-history":[{"count":0,"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/posts\/404\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/media?parent=404"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/categories?post=404"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.samontab.com\/web\/wp-json\/wp\/v2\/tags?post=404"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}