On Thursday 10th of December 2020 I presented an online webinar about OpenVINO. Here are some slides from the presentation.
OpenCV Video Course
Learn What OpenVINO Is and How to Use It
Posted in Computer Vision, IoT, openvino.
– December 10, 2020
Align text images with OpenCV using Python
Most optical character recognition(OCR) software first aligns the image properly before detecting the text in it. Here I’ll show you how to do that with OpenCV:
First we need to import opencv:
import cv2
Let’s read an image in. You probably will have a color image, so first we need to convert it to gray scale (one channel only).
image = cv2.imread("text_rotated.png")
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
We’re interested in the text itself and nothing else, so we’re going to create an image that represents this. Non-zero pixels represent what we want, and zero pixels represent background. To do that, we will threshold the grayscale image, so that only the text has non-zero values. The explanation of the inputs is simple: gray is the grayscale image used as input, the next argument is the threshold value, set to 0, although it is ignored since it will be calculated by the Otsu algorithm(it’s 170 in this case) because we’re using THRESH_OTSU flag. The next argument is 255 which is the value to set the pixels that pass the threshold, and finally the other flag indicates that we want to use a binary output (all or nothing), and that we want the output reversed (since the text is black in the original). The returned values are the actual value of the threshold to be used as calculated by the Otsu algorithm(otsu_thresh), and the thresholded image itself (thresh).
otsu_thresh, thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)
First we need to remove those extra white pixels outside of the area of the text. We can do that with the morphological operator Open, which basically erodes the image (makes the white areas smaller), and then dilates it back (make the white areas larger again). By doing that any small dots, or noise in the image, will be removed. The larger the kernel size, the more noise will be removed. You can do this either by hand, or with an iterative method that checks, for example, the ratio of non-zero to zero pixels for each value, and select the one that maximizes that metric.
kernel_size = 4 ksize=(kernel_size, kernel_size) kernel = cv2.getStructuringElement(cv2.MORPH_RECT, ksize) thresh_filtered = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, kernel)
Now that we have an image that represents the text, we’re interested in knowing the angle in which it is rotated. There are a few different ways of doing this. Since OpenCV has a convenient function that gives you the minimum rectangle that contains all non-zero values, we could use that. Let’s see how it works:
nonZeroCoordinates = cv2.findNonZero(thresh_filtered)
imageCopy = image.copy()
for pt in nonZeroCoordinates:
imageCopy = cv2.circle(imageCopy, (pt[0][0], pt[0][1]), 1, (255, 0, 0))
We can now use those coordinates and ask OpenCV to give us the minimum rectangle that contains them:
box = cv2.minAreaRect(nonZeroCoordinates)
boxPts = cv2.boxPoints(box)
for i in range(4):
pt1 = (boxPts[i][0], boxPts[i][1])
pt2 = (boxPts[(i+1)%4][0], boxPts[(i+1)%4][1])
cv2.line(imageCopy, pt1, pt2, (0,255,0), 2, cv2.LINE_AA);
The estimated angle can then be simply retrieved from the returned rectangle. Note: Remember to double check the returned angle, as it might be different to what you’re expecting. The function minAreaRect always returns angles between 0 and -90 degrees.
angle = box[2]
if(angle < -45):
angle = 90 + angle
Once you have the estimated angle, it’s time to rotate the image back. First we calculate a rotation matrix based on the angle, and then we apply it. The rotation angle is expressed in degrees, and positive values mean a counter-clockwise rotation.
h, w, c = image.shape scale = 1. center = (w/2., h/2.) M = cv2.getRotationMatrix2D(center, angle, scale) rotated = image.copy() cv2.warpAffine(image, M, (w, h), rotated, cv2.INTER_CUBIC, cv2.BORDER_REPLICATE )
And that’s it. You now have an aligned image, ready to be parsed for OCR, or any other application.
Posted in Image Processing, Open Source, OpenCV, Photography, Programming.
– November 13, 2020
Align text images with OpenCV
Most optical character recognition(OCR) software first aligns the image properly before detecting the text in it. Here I’ll show you how to do that with OpenCV:
First we need to include the required header files:
#include <opencv2/core.hpp> #include <opencv2/imgcodecs.hpp> #include <opencv2/highgui.hpp> #include <opencv2/imgproc.hpp> #include <iostream> #include <iomanip> #include <string>
Let’s read an image in. You probably will have a color image, so first we need to convert it to gray scale (one channel only).
cv::Mat image = cv::imread( "text_rotated.png", cv::IMREAD_COLOR); cv::Mat gray; cv::cvtColor(image, gray, cv::COLOR_BGR2GRAY);
We’re interested in the text itself and nothing else, so we’re going to create an image that represents this. Non-zero pixels represent what we want, and zero pixels represent background. To do that, we will threshold the grayscale image, so that only the text has non-zero values. The explanation of the inputs is simple: gray is the grayscale image used as input, thresh is where the thresholded image will be stored. The next argument is the threshold value, set to 0, although it is ignored since it will be calculated by the Otsu algorithm because we’re using THRESH_OTSU flag. The next argument is 255 which is the value to set the pixels that pass the threshold, and finally the other flag indicates that we want to use a binary output (all or nothing), and that we want the output reversed (since the text is black in the original).
cv::Mat thresh; cv::threshold(gray, thresh, 0, 255, cv::THRESH_BINARY_INV | cv::THRESH_OTSU);
First we need to remove those extra white pixels outside of the area of the text. We can do that with the morphological operator Open, which basically erodes the image (makes the white areas smaller), and then dilates it back (make the white areas larger again). By doing that any small dots, or noise in the image, will be removed. The larger the kernel size, the more noise will be removed. You can do this either by hand, or with an iterative method that checks, for example, the ratio of non-zero to zero pixels for each value, and select the one that maximizes that metric.
double kernel_size = 4; cv::Mat thresh_filtered; cv::Mat kernel = cv::getStructuringElement( cv::MORPH_RECT, cv::Size(kernel_size, kernel_size)); cv::morphologyEx(thresh, thresh_filtered, cv::MORPH_OPEN, kernel);
Now that we have an image that represents the text, we’re interested in knowing the angle in which it is rotated. There are a few different ways of doing this. Since OpenCV has a convenient function that gives you the minimum rectangle that contains all non-zero values, we could use that. Let’s see how it works:
cv::Mat nonZeroCoordinates;
cv::findNonZero(thresh, nonZeroCoordinates);
cv::Mat imageCopy = image.clone();
for (int i = 0; i < nonZeroCoordinates.total(); i++ )
{
cv::Point pt = nonZeroCoordinates.at<cv::Point>(i);
cv::circle(imageCopy, pt, 1, cv::Scalar(255, 0, 0), 1);
}
We can now use those coordinates and ask OpenCV to give us the minimum rectangle that contains them:
cv::RotatedRect box = cv::minAreaRect(nonZeroCoordinates);
cv::Point2f vertices[4];
box.points(vertices);
for (int i = 0; i < 4; i++)
{
cv::line(imageCopy, vertices[i], vertices[(i+1)%4], cv::Scalar(0,255,0), 2);
}
The estimated angle can then be simply retrieved from the returned rectangle. Note: Remember to double check the returned angle, as it might be different to what you’re expecting. The function minAreaRect always returns angles between 0 and -90 degrees.
float angle = box.angle;
if (angle < -45.0f)
{
angle = (90.0f + angle);
}
Once you have the estimated angle, it’s time to rotate the image back. First we calculate a rotation matrix based on the angle, and then we apply it. The rotation angle is expressed in degrees, and positive values mean a counter-clockwise rotation.
cv::Point2f center((image.cols) / 2.0f, (image.rows) / 2.0f); double scale = 1.; cv::Mat M = cv::getRotationMatrix2D(center, angle, scale); cv::Mat rotated; cv::warpAffine(image, rotated, M, image.size(), cv::INTER_CUBIC, cv::BORDER_REPLICATE);
And that’s it. You now have an aligned image, ready to be parsed for OCR, or any other application.
Posted in Image Processing, Open Source, OpenCV, Photography, Programming.
– November 12, 2020
Installing OpenCV 4.5.0 in Ubuntu 20.04 LTS
OpenCV was initially released about 20 years ago. It’s one of the most well established computer vision libraries in the world, with thousands of algorithm implementations ready to be used in commercial and research applications.
In this guide I’ll show you how to install OpenCV 4.5.0 in your Ubuntu 20.04 LTS and how to create computer vision applications with C++ and Python.
Note: I have noticed some copies of my posts elsewhere, so make sure that you are reading this from the original source, at samontab dot com, accessible from here so that you don’t miss the comments.
First, make sure you have the latest software installed:
sudo apt-get update sudo apt-get upgrade
Now, you need to install some dependencies, such as support for reading and writing video files, drawing on the screen, some needed tools, etc… This step is very easy, you only need to write the following command in the Terminal:
sudo apt-get install build-essential cmake python3-numpy python3-dev python3-tk libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libavresample-dev libdc1394-dev libeigen3-dev libgtk-3-dev libvtk7-qt-dev
Time to download and compile OpenCV 4.5.0:
wget https://github.com/opencv/opencv/archive/4.5.0.tar.gz tar -xvzf 4.5.0.tar.gz rm 4.5.0.tar.gz cd opencv-4.5.0 mkdir build;cd build cmake -DBUILD_EXAMPLES=ON ..
Check that the above command produces no error and that in particular it reports FFMPEG as YES. If this is not the case you will not be able to read or write videos. Double check that every feature you want present in OpenCV is reported correctly. If you’re missing something, make sure to install the missing package and run the cmake line again. In some cases you might need to delete the whole build directory first to get a correct detection. Once you’re happy with what’s shown then we’ll build it. After running the following lines you can go grab a coffee as it will take a long time.
make -j4 sudo make install echo '/usr/local/lib' | sudo tee --append /etc/ld.so.conf.d/opencv.conf sudo ldconfig echo 'PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig' | sudo tee --append ~/.bashrc echo 'export PKG_CONFIG_PATH' | sudo tee --append ~/.bashrc source ~/.bashrc
After that’s finished you should be able to use OpenCV with C++ and Python. Let’s do a quick check:
python3 import cv2 cv2.__version__
You should see no errors and 4.5.0 as the version.
Now let’s run some Python demos. You can start all of them from a central place:
cd ~/opencv-4.5.0/samples/python python3 demo.py
You can also execute any of the already compiled C++ demos. For example let’s run the intelligent scissors demo:
cd ~/opencv-4.5.0/build/bin ./example_cpp_intelligent_scissors
If you click on the image and move your mouse you’ll see that the selection moves to match the object edges. Once you are ready selecting the contour of the object press right click and the object will be selected.
Now you’ll learn how to compile an application that uses OpenCV with C++ using cmake. Let’s copy another example source code and prepare it for compilation.
cd ~ mkdir sample cd sample cp ~/opencv-4.5.0/samples/cpp/text_skewness_correction.cpp . cp ~/opencv-4.5.0/samples/data/imageTextR.png .
Now we’re going to create a cmake file that will allow us to compile the sample. Create a text file called CMakeLists.txt and put this content in it:
cmake_minimum_required (VERSION 3.0.2)
project (text_skewness_correction)
find_package(OpenCV REQUIRED)
include_directories( ${OpenCV_INCLUDE_DIRS} )
add_executable (${PROJECT_NAME} text_skewness_correction.cpp)
target_link_libraries (${PROJECT_NAME} ${OpenCV_LIBS})
OK, now we’re ready to build it and run it:
mkdir build cd build cmake .. make ./text_skewness_correction ../imageTextR.png
Now you have OpenCV 4.5.0 correctly installed in your Ubuntu 20.04 LTS system, and you know how to create applications using C++ and Python.
Posted in Computer Vision, Image Processing, Open Source, OpenCV.
– November 6, 2020
Minimal install of OpenVINO 2020.4 in Kubuntu 20.04 LTS for C++ inference on CPU
The official installer of OpenVINO 2020.4 requires Ubuntu 18.04 LTS, but since Kubuntu 20.04 LTS is already out I wanted to use that instead. Here’s what you need to do to get a minimal install for C++ inference on the CPU.
First make sure you have everything up to date:
sudo apt-get update sudo apt-get upgrade
You’ll need git installed to get the source code. Also, python3 comes already pre-installed with 20.04 but not python, so we’re going to assign any python calls to python3.
sudo apt-get install git python-is-python3
Now get the OpenVINO source code into your home directory (or wherever you prefer). Don’t download the zip file directly from github as there are some 3rd party dependencies that need to get downloaded as well. That’s the –recursive part of the following command. The specific –branch we’re using is the same as the official installer uses, releases/2020/4.
cd ~ git clone --recursive --branch releases/2020/4 https://github.com/openvinotoolkit/openvino.git
Install the dependencies by running the included script:
~/openvino/install_dependencies.sh
Before continuing you need to change some of the files you just downloaded. I’m using xdg-open which will invoke your preferred application but you can use whatever you prefer instead like vi, nano, etc.
First open up the gna_helper.cpp file.
xdg-open ~/openvino/inference-engine/src/gna_plugin/gna_helper.cpp
Update the code for these functions as follows:
void profilerRtcStart(intel_gna_profiler_rtc *p) {
if (nullptr == p) return;
clearTimeB(p->passed);
clearTimeB(p->stop);
//ftime(&p->start);
timespec start;
clock_gettime(CLOCK_REALTIME, &start);
p->start.time = start.tv_sec;
p->start.millitm = start.tv_nsec/1000000;
}
void profilerRtcStop(intel_gna_profiler_rtc *p) {
if (nullptr == p) return;
//ftime(&p->stop);
timespec stop;
clock_gettime(CLOCK_REALTIME, &stop);
p->stop.time = stop.tv_sec;
p->stop.millitm = stop.tv_nsec/1000000;
}
Now open the execution_engine.cpp file:
xdg-open ~/openvino/inference-engine/thirdparty/ade/sources/ade/source/execution_engine.cpp
In line 141, simply update this:
//return std::move(ret); return ret;
Now you’re ready to build it. And this will take a long time.
- ENABLE_MKL_DNN=ON means we want the CPU plugin
- ENABLE_CLDNN=OFF means we don’t want the GPU plugin
cd ~/openvino mkdir build cd build cmake -DCMAKE_BUILD_TYPE=Release -DENABLE_MKL_DNN=ON -DENABLE_CLDNN=OFF -DENABLE_OPENCV=OFF -DENABLE_PYTHON=OFF .. make -j4
You should now be able to run the compiled applications, for example to get information about your device you can run the following:
~/openvino/bin/intel64/Release/hello_query_device
It should output detailed information about your device. Check out the Intel Model Zoo for models and samples that you can use.
Posted in Image Processing, Open Source, openvino, Programming.
– July 25, 2020
Installing OpenCV 3.2.0 with contrib modules in Ubuntu 16.04 LTS
UPDATE: You can also install OpenCV 4.5.0 in Ubuntu 20.04LTS.
OpenCV 3.2.0 has been out for a while and contains many improvements and exciting new features, so it’s time to update this guide using the latest Ubuntu 16.04LTS.
A big change in OpenCV 3.2.0 is that now many of the newest algorithms now reside separately in the contrib repository.
Some of these modules include Face Recognition, RGB-Depth processing, Image Registration, Saliency, Structure From Motion, Tracking, and much more.
So let’s install OpenCV 3.2.0 with the contrib modules and other good stuff by executing the following code on the command line:
sudo apt-get update sudo apt-get upgrade sudo apt-get install build-essential libgtk2.0-dev libjpeg-dev libtiff5-dev libjasper-dev libopenexr-dev cmake python-dev python-numpy python-tk libtbb-dev libeigen3-dev yasm libfaac-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev libx264-dev libqt4-dev libqt4-opengl-dev sphinx-common texlive-latex-extra libv4l-dev libdc1394-22-dev libavcodec-dev libavformat-dev libswscale-dev default-jdk ant libvtk5-qt4-dev cd ~ mkdir opencv cd opencv wget https://github.com/opencv/opencv/archive/3.2.0.tar.gz tar -xvzf 3.2.0.tar.gz wget https://github.com/opencv/opencv_contrib/archive/3.2.0.zip unzip 3.2.0.zip cd opencv-3.2.0
But before we build it, we need to fix one problem currently present in the contrib modules, specifically in the freetype module, which allows you to draw UTF-8 strings. If you are getting an error similar to ImportError: /usr/local/lib/libopencv_freetype.so.3.2: undefined symbol: hb_shape, this will fix it:
sed -i 's/${freetype2_LIBRARIES} ${harfbuzz_LIBRARIES}/${FREETYPE_LIBRARIES} ${HARFBUZZ_LIBRARIES}/g' ../opencv_contrib-3.2.0/modules/freetype/CMakeLists.txt
Since now that problem is solved, now we are ready to build OpenCV.
mkdir build cd build cmake -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=ON -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D WITH_VTK=ON .. -DCMAKE_BUILD_TYPE=RELEASE -DOPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-3.2.0/modules .. make sudo make install echo '/usr/local/lib' | sudo tee --append /etc/ld.so.conf.d/opencv.conf sudo ldconfig echo 'PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig' | sudo tee --append ~/.bashrc echo 'export PKG_CONFIG_PATH' | sudo tee --append ~/.bashrc source ~/.bashrc
Now you should be able to compile with the OpenCV libraries, including the contrib repositories.
For example, let’s compute some Fine-Grained Saliency, which is available in the saliency module of the contrib repository:
cd ~ mkdir saliency cd saliency cp ../opencv/opencv_contrib-3.2.0/modules/saliency/samples/computeSaliency.cpp . cp ../opencv/opencv-3.2.0/samples/data/Megamind.avi . g++ -o computeSaliency `pkg-config opencv --cflags` computeSaliency.cpp `pkg-config opencv --libs` ./computeSaliency FINE_GRAINED Megamind.avi 23
Original Image:
Computed Saliency:
Posted in Image Processing, Open Source, Programming.
– June 3, 2017
Interfacing Intel RealSense F200 with OpenCV
RealSense from Intel is an interesting technology that brings 3D cameras into the mainstream.
Although the RealSense SDK provides a good starting point for many applications, some users would prefer to have a bit more control over the images. In this post, I’ll describe how to access the raw streams from an Intel RealSense F200 camera, and how to convert those images into OpenCV cv::Mat objects.
Here is the video with all the explanations
And here are the files used in the video:
01-alignedSingleThreaded.cpp:
#include <pxcsensemanager.h> #include <opencv2/opencv.hpp> PXCSenseManager *pxcSenseManager; PXCImage * CVMat2PXCImage(cv::Mat cvImage) { PXCImage::ImageInfo iinfo; memset(&iinfo,0,sizeof(iinfo)); iinfo.width=cvImage.cols; iinfo.height=cvImage.rows; PXCImage::PixelFormat format; int type = cvImage.type(); if(type == CV_8UC1) format = PXCImage::PIXEL_FORMAT_Y8; else if(type == CV_8UC3) format = PXCImage::PIXEL_FORMAT_RGB24; else if(type == CV_32FC1) format = PXCImage::PIXEL_FORMAT_DEPTH_F32; iinfo.format = format; PXCImage *pxcImage = pxcSenseManager->QuerySession()->CreateImage(&iinfo); PXCImage::ImageData data; pxcImage->AcquireAccess(PXCImage::ACCESS_WRITE, format, &data); data.planes[0] = cvImage.data; pxcImage->ReleaseAccess(&data); return pxcImage; } cv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format) { PXCImage::ImageData data; pxcImage->AcquireAccess(PXCImage::ACCESS_READ, format, &data); int width = pxcImage->QueryInfo().width; int height = pxcImage->QueryInfo().height; if(!format) format = pxcImage->QueryInfo().format; int type; if(format == PXCImage::PIXEL_FORMAT_Y8) type = CV_8UC1; else if(format == PXCImage::PIXEL_FORMAT_RGB24) type = CV_8UC3; else if(format == PXCImage::PIXEL_FORMAT_DEPTH_F32) type = CV_32FC1; else if(format == PXCImage::PIXEL_FORMAT_DEPTH) type = CV_16UC1; cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0]); pxcImage->ReleaseAccess(&data); return ocvImage; } int main(int argc, char* argv[]) { //Define some parameters for the camera cv::Size frameSize = cv::Size(640, 480); float frameRate = 60; //Create the OpenCV windows and images cv::namedWindow("IR", cv::WINDOW_NORMAL); cv::namedWindow("Color", cv::WINDOW_NORMAL); cv::namedWindow("Depth", cv::WINDOW_NORMAL); cv::Mat frameIR = cv::Mat::zeros(frameSize, CV_8UC1); cv::Mat frameColor = cv::Mat::zeros(frameSize, CV_8UC3); cv::Mat frameDepth = cv::Mat::zeros(frameSize, CV_8UC1); //Initialize the RealSense Manager pxcSenseManager = PXCSenseManager::CreateInstance(); //Enable the streams to be used pxcSenseManager->EnableStream(PXCCapture::STREAM_TYPE_IR, frameSize.width, frameSize.height, frameRate); pxcSenseManager->EnableStream(PXCCapture::STREAM_TYPE_COLOR, frameSize.width, frameSize.height, frameRate); pxcSenseManager->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, frameSize.width, frameSize.height, frameRate); //Initialize the pipeline pxcSenseManager->Init(); bool keepRunning = true; while(keepRunning) { //Acquire all the frames from the camera pxcSenseManager->AcquireFrame(); PXCCapture::Sample *sample = pxcSenseManager->QuerySample(); //Convert each frame into an OpenCV image frameIR = PXCImage2CVMat(sample->ir, PXCImage::PIXEL_FORMAT_Y8); frameColor = PXCImage2CVMat(sample->color, PXCImage::PIXEL_FORMAT_RGB24); cv::Mat frameDepth_u16 = PXCImage2CVMat(sample->depth, PXCImage::PIXEL_FORMAT_DEPTH); frameDepth_u16.convertTo(frameDepth, CV_8UC1); cv::Mat frameDisplay; cv::equalizeHist(frameDepth, frameDisplay); //Display the images cv::imshow("IR", frameIR); cv::imshow("Color", frameColor); cv::imshow("Depth", frameDisplay); //Check for user input int key = cv::waitKey(1); if(key == 27) keepRunning = false; //Release the memory from the frames pxcSenseManager->ReleaseFrame(); } //Release the memory from the RealSense manager pxcSenseManager->Release(); return 0; }02-unalignedSingleThreaded.cpp:
#include <pxcsensemanager.h> #include <opencv2/opencv.hpp> cv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format) { PXCImage::ImageData data; pxcImage->AcquireAccess(PXCImage::ACCESS_READ, format, &data); int width = pxcImage->QueryInfo().width; int height = pxcImage->QueryInfo().height; if(!format) format = pxcImage->QueryInfo().format; int type; if(format == PXCImage::PIXEL_FORMAT_Y8) type = CV_8UC1; else if(format == PXCImage::PIXEL_FORMAT_RGB24) type = CV_8UC3; else if(format == PXCImage::PIXEL_FORMAT_DEPTH_F32) type = CV_32FC1; cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0]); pxcImage->ReleaseAccess(&data); return ocvImage; } int main(int argc, char* argv[]) { //Define some parameters for the camera cv::Size frameSize = cv::Size(640, 480); float frameRate = 60; //Create the OpenCV windows and images cv::namedWindow("IR", cv::WINDOW_NORMAL); cv::namedWindow("Color", cv::WINDOW_NORMAL); cv::namedWindow("Depth", cv::WINDOW_NORMAL); cv::Mat frameIR = cv::Mat::zeros(frameSize, CV_8UC1); cv::Mat frameColor = cv::Mat::zeros(frameSize, CV_8UC3); cv::Mat frameDepth = cv::Mat::zeros(frameSize, CV_8UC1); //Initialize the RealSense Manager PXCSenseManager *pxcSenseManager = PXCSenseManager::CreateInstance(); //Enable the streams to be used pxcSenseManager->EnableStream(PXCCapture::STREAM_TYPE_IR, frameSize.width, frameSize.height, frameRate); pxcSenseManager->EnableStream(PXCCapture::STREAM_TYPE_COLOR, frameSize.width, frameSize.height, frameRate); pxcSenseManager->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, frameSize.width, frameSize.height, frameRate); //Initialize the pipeline pxcSenseManager->Init(); bool keepRunning = true; while(keepRunning) { //Acquire any frame from the camera pxcSenseManager->AcquireFrame(false); PXCCapture::Sample *sample = pxcSenseManager->QuerySample(); //Convert each frame into an OpenCV image //You need to make sure that the image is there first if(sample->ir) frameIR = PXCImage2CVMat(sample->ir, PXCImage::PIXEL_FORMAT_Y8); if(sample->color) frameColor = PXCImage2CVMat(sample->color, PXCImage::PIXEL_FORMAT_RGB24); if(sample->depth) PXCImage2CVMat(sample->depth, PXCImage::PIXEL_FORMAT_DEPTH_F32).convertTo(frameDepth, CV_8UC1); //Display the images cv::imshow("IR", frameIR); cv::imshow("Color", frameColor); cv::imshow("Depth", frameDepth); //Check for user input int key = cv::waitKey(1); if(key == 27) keepRunning = false; //Release the memory from the frames pxcSenseManager->ReleaseFrame(); } //Release the memory from the RealSense manager pxcSenseManager->Release(); return 0; }03-unalignedMultiThreaded.cpp:
#include <pxcsensemanager.h> #include <iostream> #include <opencv2/opencv.hpp> cv::Mat frameIR; cv::Mat frameColor; cv::Mat frameDepth; cv::Mutex framesMutex; cv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format) { PXCImage::ImageData data; pxcImage->AcquireAccess(PXCImage::ACCESS_READ, format, &data); int width = pxcImage->QueryInfo().width; int height = pxcImage->QueryInfo().height; if(!format) format = pxcImage->QueryInfo().format; int type; if(format == PXCImage::PIXEL_FORMAT_Y8) type = CV_8UC1; else if(format == PXCImage::PIXEL_FORMAT_RGB24) type = CV_8UC3; else if(format == PXCImage::PIXEL_FORMAT_DEPTH_F32) type = CV_32FC1; cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0]); pxcImage->ReleaseAccess(&data); return ocvImage; } class FramesHandler:public PXCSenseManager::Handler { public: virtual pxcStatus PXCAPI OnNewSample(pxcUID, PXCCapture::Sample *sample) { framesMutex.lock(); if(sample->ir) frameIR = PXCImage2CVMat(sample->ir, PXCImage::PIXEL_FORMAT_Y8); if(sample->color) frameColor = PXCImage2CVMat(sample->color, PXCImage::PIXEL_FORMAT_RGB24); if(sample->depth) PXCImage2CVMat(sample->depth, PXCImage::PIXEL_FORMAT_DEPTH_F32).convertTo(frameDepth, CV_8UC1); framesMutex.unlock(); return PXC_STATUS_NO_ERROR; } }; int main(int argc, char* argv[]) { cv::Size frameSize = cv::Size(640, 480); float frameRate = 60; cv::namedWindow("IR", cv::WINDOW_NORMAL); cv::namedWindow("Color", cv::WINDOW_NORMAL); cv::namedWindow("Depth", cv::WINDOW_NORMAL); frameIR = cv::Mat::zeros(frameSize, CV_8UC1); frameColor = cv::Mat::zeros(frameSize, CV_8UC3); frameDepth = cv::Mat::zeros(frameSize, CV_8UC1); PXCSenseManager *pxcSenseManager = PXCSenseManager::CreateInstance(); //Enable the streams to be used pxcSenseManager->EnableStream(PXCCapture::STREAM_TYPE_IR, frameSize.width, frameSize.height, frameRate); pxcSenseManager->EnableStream(PXCCapture::STREAM_TYPE_COLOR, frameSize.width, frameSize.height, frameRate); pxcSenseManager->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, frameSize.width, frameSize.height, frameRate); FramesHandler handler; pxcSenseManager->Init(&handler); pxcSenseManager->StreamFrames(false); //Local images for display cv::Mat displayIR = frameIR.clone(); cv::Mat displayColor = frameColor.clone(); cv::Mat displayDepth = frameDepth.clone(); bool keepRunning = true; while(keepRunning) { framesMutex.lock(); displayIR = frameIR.clone(); displayColor = frameColor.clone(); displayDepth = frameDepth.clone(); framesMutex.unlock(); cv::imshow("IR", displayIR); cv::imshow("Color", displayColor); cv::imshow("Depth", displayDepth); int key = cv::waitKey(1); if(key == 27) keepRunning = false; } //Stop the frame acqusition thread pxcSenseManager->Close(); pxcSenseManager->Release(); return 0; }04-alignedMultiThreaded.cpp:
#include <pxcsensemanager.h> #include <iostream> #include <opencv2/opencv.hpp> cv::Mat frameIR; cv::Mat frameColor; cv::Mat frameDepth; cv::Mutex framesMutex; cv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format) { PXCImage::ImageData data; pxcImage->AcquireAccess(PXCImage::ACCESS_READ, format, &data); int width = pxcImage->QueryInfo().width; int height = pxcImage->QueryInfo().height; if(!format) format = pxcImage->QueryInfo().format; int type; if(format == PXCImage::PIXEL_FORMAT_Y8) type = CV_8UC1; else if(format == PXCImage::PIXEL_FORMAT_RGB24) type = CV_8UC3; else if(format == PXCImage::PIXEL_FORMAT_DEPTH_F32) type = CV_32FC1; cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0]); pxcImage->ReleaseAccess(&data); return ocvImage; } class FramesHandler:public PXCSenseManager::Handler { public: virtual pxcStatus PXCAPI OnNewSample(pxcUID, PXCCapture::Sample *sample) { framesMutex.lock(); frameIR = PXCImage2CVMat(sample->ir, PXCImage::PIXEL_FORMAT_Y8); frameColor = PXCImage2CVMat(sample->color, PXCImage::PIXEL_FORMAT_RGB24); PXCImage2CVMat(sample->depth, PXCImage::PIXEL_FORMAT_DEPTH_F32).convertTo(frameDepth, CV_8UC1); framesMutex.unlock(); return PXC_STATUS_NO_ERROR; } }; int main(int argc, char* argv[]) { cv::Size frameSize = cv::Size(640, 480); float frameRate = 60; cv::namedWindow("IR", cv::WINDOW_NORMAL); cv::namedWindow("Color", cv::WINDOW_NORMAL); cv::namedWindow("Depth", cv::WINDOW_NORMAL); frameIR = cv::Mat::zeros(frameSize, CV_8UC1); frameColor = cv::Mat::zeros(frameSize, CV_8UC3); frameDepth = cv::Mat::zeros(frameSize, CV_8UC1); PXCSenseManager *pxcSenseManager = PXCSenseManager::CreateInstance(); //Enable the streams to be used PXCVideoModule::DataDesc ddesc={}; ddesc.deviceInfo.streams = PXCCapture::STREAM_TYPE_IR | PXCCapture::STREAM_TYPE_COLOR | PXCCapture::STREAM_TYPE_DEPTH; pxcSenseManager->EnableStreams(&ddesc); FramesHandler handler; pxcSenseManager->Init(&handler); pxcSenseManager->StreamFrames(false); //Local images for display cv::Mat displayIR = frameIR.clone(); cv::Mat displayColor = frameColor.clone(); cv::Mat displayDepth = frameDepth.clone(); bool keepRunning = true; while(keepRunning) { framesMutex.lock(); displayIR = frameIR.clone(); displayColor = frameColor.clone(); displayDepth = frameDepth.clone(); framesMutex.unlock(); cv::imshow("IR", displayIR); cv::imshow("Color", displayColor); cv::imshow("Depth", displayDepth); int key = cv::waitKey(1); if(key == 27) keepRunning = false; } //Stop the frame acqusition thread pxcSenseManager->Close(); pxcSenseManager->Release(); return 0; }
Posted in 3D, Image Processing, Photography, Programming.
– April 27, 2016
Understanding how virtual make-up apps work
Virtual make-up is an interesting technology that can be used to aid in the decision of buying cosmetics, enhancing portraits, or just for fun.
Since the final color of an applied cosmetic depends both, on the color of the cosmetic, and on skin color, most of the time people have to go to the stores and try the products on themselves to see how they would look like. With virtual make-up technology, this can be conveniently simulated on a computer, or a mobile phone. The only thing needed is a photograph of their face looking towards the camera, and the software is able to simulate how a particular make-up would look on that particular skin color. This can also be applied to a live camera feed for a more realistic application with real time rendering.
To create an application like this, the first step is to estimate where the faces are in the photograph. This can be solved with computer vision. In the general case, this problem is called object detection. You need to define how your object looks like, and then train an algorithm with many images of the object. Most computer vision algorithms that perform this task assume that the face appears on a roughly frontal view, with little or no objects covering it. This is because most faces have a similar structure when viewed from the front, whereas profile or back views of the head change a lot from person to person because of hair styles, among other things.
In order to capture the facial structure, these algorithms are usually trained with features such as Haar-like, lbp, or HoG. Once trained, the algorithm is able to detect faces in images.
So, for example, let’s say that you start with an image like this:
After detecting the face, you will end up with an region of interest on the image. Something like this:
Now, inside that region of interest, we need to detect specific face landmarks. These landmarks represent the position of different parts of the face, such as eyes, mouth, and eyebrows. This is again, an object detection problem. You need to train an algorithm with many annotated images and a specific number of facial landmarks. Then, you can use this trained algorithm to detect those facial features in a new image. Just like this for example:
Once you have the position of those landmarks, you need to design your own make-up, and align it to those features. Once you have that, you can then blend together the original image with your designed make-up. Here are some basic examples with a few different colors:
The position of the detected landmarks and the design of the make-up are crucial to make it appear realistic. On top of that, there are many different computer vision techniques that can be applied in order to blend the make-up into the face in a more realistic manner.
Posted in General, Image Processing, Photography, Programming.
– March 11, 2016
Interfacing Intel RealSense 3D camera with Google Camera’s Lens Blur
There is an interesting mode in the Google Camera app, called Lens Blur, that allows you to refocus a picture after it was taken, basically simulating what a light-field camera is able to do but using just a regular smartphone.
To do this, the app uses an array of computer vision techniques such as Structure-from-Motion(SfM), and Multi-View-Stereo(MVS) to create a depth map of the scene. Having this entire pipeline running on a smartphone is remarkable to say the least. You can read more about how this is done here.
Once the depth map and the photo are acquired, the user can select the focus location, and the desired depth of field. A thin lens model is then used to simulate a real lens matching the desired focus location and depth of field, generating a new image.
In theory, any photograph with its corresponding depth map could be used with this technique. To test this, I decided to use an Intel RealSense F200 camera, and it worked. Here are the results:
Focused on the front:
Focused on the back:
Those two images were created on the smartphone using the Lens Blur feature of the Google Camera app. The input image was created by me externally, but the app was happy to process it since I used the same encoding that the app uses.
To do that, I first took a color and depth image from the Realsense camera. Then, projected the depth image into the color camera:
Color photograph from RealSense F200:
Projected depth image from RealSense F200:
The next step is to encode the depth image into a format that Google Camera understands, so I followed the encoding instructions from the documentation. The RangeLinear encoding of the previous depth map looks something like this:
The next step is just to embed the encoded image into the metadata of the original color image, and copy the final photo into the smartphone gallery. After that, you can just open the app, select the image, and refocus it!.
Posted in 3D, Image Processing, Photography, Programming.
– November 5, 2015
Cross Platform Development for Intel Edison using CMake and Qt Creator(32 and 64 bits)
The Intel Edison is an amazing little device, but the current SDK only works out of the box with Eclipse, and creating new projects is not very easy.
In this post, I will show you how to use CMake and Qt Creator to develop for the Intel Edison remotely. This allows you to use the power of CMake to create new projects, and the convenience of Qt Creator, a great cross platform IDE for C/C++.
This guide should work for all major platforms (Windows, Mac, Linux, 32 and 64 bits) with little or no changes at all.
Here you can see a Windows machine running a debugging session with the Edison using Qt Creator:
Required software
- First of all, make sure that you have everything installed for your system to communicate with the Edison. The guys at SparkFun already covered this here.
- Also, make sure that you have Qt Creator and CMake installed as well. Note that we are only using the Qt Creator IDE, so you don’t need to install the entire Qt SDK if you are not going to use it in your application.
Cross compiling
We are going to cross compile for the Edison. This means that we are going to build an executable (or library) for a target OS that is different from the host OS. In this case, the target OS is the Edison’s Yocto Linux and the host OS is the OS that you are running (Windows, Linux, Mac, 32 or 64 bits). Cross compiling means that the host OS compiles something for the target OS. Easy. In order to do this, you will need to have a set of applications for your host OS that allow you to compile for the target OS, as well as the libraries and include files of the target system. These things as a whole are called a toolchain.
The toolchain
Intel provides this toolchain for all the major OS on the main downloads site. Download the one for your OS under the section SDK Cross compile tools.
For Linux, just install it under the default directory:
/opt/poky-edison/1.6.1
For Windows, decompress the files (twice) and save them under C:/Edison.
You will end up with this directory structure for the 64 bits SDK:
C:/Edison/poky-edison-eglibc-x86_64-edison-image-core2-32-toolchain-1.6.1
And this directory structure for the 32 bits SDK:
C:/Edison/poky-edison-eglibc-i686-edison-image-core2-32-toolchain-1.6.1
Note that these locations are important because they will be referred to in the Qt Creator configuration and in the CMake script.
Make under Windows
There is a utility that was not included in the toolchain, a Make generator. The Eclipse version of the SDK uses an Eclipse internal tool to do the Make step. Since we are not using Eclipse, we will provide our own make generator tool. MinGW64 will help us here. It is a “Minimalist GNU for Windows”. Just download and install it in the default directory. Take a note of the installation directory because it is referred to in the CMake script, and it will probably change in future releases of MinGW64.
Qt Creator configuration
We are going to define a new device for development on Qt.
Open Qt Creator. Go to Tools->Options. On the left side, select Devices.
Click on Add, and select Generic Linux Device. Click on Start Wizard.
Set the name as Edison, or any name that you want here, the IP address of the Edison (something that may look like 192.168.1.42 for example), root as the username, and your current password. Click next when done.
The device will then be tested for connectivity, and everything should be fine. If there is any problem, edit the settings and try again.
Now we are going to create a Qt kit to use with this device. Go to Build & Run and select the Compilers tab.
Once you are there, and select Add->GCC. Under Name, you can change it to anything you want. Under compiler path, you have to select the C++ compiler for your host OS. These are specified in the CMake script, but I will put them here as well for reference, but always make sure to double check your actual path:
Windows hosts(64 bits):
C:/Edison/poky-edison-eglibc-x86_64-edison-image-core2-32-toolchain-1.6.1/sysroots/x86_64-pokysdk-mingw32/usr/bin/i586-poky-linux/i586-poky-linux-g++.exe
Windows hosts(32 bits):
C:/Edison/poky-edison-eglibc-i686-edison-image-core2-32-toolchain-1.6.1/sysroots/i686-pokysdk-mingw32/usr/bin/i586-poky-linux/i586-poky-linux-g++.exe
Linux hosts (64 bits):
/opt/poky-edison/1.6.1/sysroots/x86_64-pokysdk-linux/usr/bin/i586-poky-linux/i586-poky-linux-g++
Linux hosts (32 bits):
/opt/poky-edison/1.6.1/sysroots/i686-pokysdk-linux/usr/bin/i586-poky-linux/i586-poky-linux-g++
Now go to the cmake tab and make sure that you have the cmake executable listed there.
Let’s add debugging capabilities. Go to the Debuggers tab and click on Add. Under Path, select the path to your gdb executable (This is part of the previously installed Mingw64 if you are on Windows, or just gdb if you are on Linux). You can change the name if you want.
Now we are ready to define a Qt Kit. Click on the Kits tab and click on Add.
Here you will set all the values that you created previously. Under Name you can put Edison or anything you want. Leave File system name blank. Under Device type select Generic Linux Device. This should automatically select Edison as your Device. Set your Sysroot to ({PATH_TO_SDK}/sysroots/core2-32-poky-linux), where PATH_TO_SDK depends on your OS as you saw earlier. Select GCC as your compiler (the one you previously defined). Under debugger set it to New Debugger, and finally under Qt version set it to None as well since we are not using the Qt SDK.
Click OK and now Qt Creator is configured to build Edison projects.
The CMake script
This is what defines your project or application. This is going to be the script that describes what we want to do. In particular we will tell CMake that we want to do a cross compile for the Edison, and specify which libraries we want to use, as well as where our source files are, and what libraries we are going to use. There are two files, one that deals with all the details about cross compiling (Edison.cmake), and the other file that deals with your actual project (CMakeLists.txt).
You can download the project files here. I added a simple mraa example as the main.cpp that reads ADC A0 analog input, provided by Brendan Le Foll. Note that some boards like the mini breakout do not have this ability built in so they will return an error. That’s expected. The key here is to make the program compile and run on the Edison, not what the program actually does.
Edison.cmake
This file defines everything you need to deploy your application into the Edison. You will only need to setup this once, and you will be able to use this file in other projects. I will go through each section of the file explaining what it does.
The following code basically allows you to manually select which version of the SDK you are planning to use, 32 or 64 bits. Comment (add a # symbol) to the line you don’t want, and uncomment the line you want. Easy. By default it is set to use the 64 bits SDK.
#Are you using the 32 or the 64 bits version of the SDK?. set(SDK_32bits FALSE) #Use this line for the 64 bits SDK #set(SDK_32bits TRUE) #Use this line for the 32 bits SDK
This is straight forward. Just define where you want your application to be copied to into the Edison (currently it will be copied to /home/root).
#You can change this to any absolute folder in your Edison #This is the folder where the executable will be copied to. set(deploymentFolderAbsolutPath /home/root)
The following code defines the properties of the target OS. No need to change anything here.
#Set the target parameters set(CMAKE_SYSTEM_NAME Linux) set(CMAKE_SYSTEM_VERSION 3.10.17-poky-edison+) set(CMAKE_SYSTEM_PROCESSOR i686)
This is where all the paths to the SDK are set. Your OS is identified by cmake, and the version of the SDK (32 or 64 bits) was manually defined earlier. You may only need to change the paths to your specific installations but it may just work as it is.
#Set the host parameters
if(WIN32)
#Windows host
if(${SDK_32bits})
#32 bit SDK
set(edison_sdk_root D:/Edison/poky-edison-eglibc-i686-edison-image-core2-32-toolchain-1.6.1)
set(cross_compiler_sysroot ${edison_sdk_root}/sysroots/i686-pokysdk-mingw32)
#MinGW make
set(CMAKE_MAKE_PROGRAM "C:/Program Files/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/mingw32-make.exe")
else()
#64 bit SDK
set(edison_sdk_root C:/Edison/poky-edison-eglibc-x86_64-edison-image-core2-32-toolchain-1.6.1)
set(cross_compiler_sysroot ${edison_sdk_root}/sysroots/x86_64-pokysdk-mingw32)
#MinGW make
set(CMAKE_MAKE_PROGRAM "C:/Program Files (x86)/mingw-w64/i686-4.9.2-posix-dwarf-rt_v3-rev1/mingw32/bin/mingw32-make.exe")
endif()
set(CMAKE_C_COMPILER ${cross_compiler_sysroot}/usr/bin/i586-poky-linux/i586-poky-linux-gcc.exe)
set(CMAKE_CXX_COMPILER ${cross_compiler_sysroot}/usr/bin/i586-poky-linux/i586-poky-linux-g++.exe)
else()
#Linux host
if(${SDK_32bits})
#32 bit SDK
set(edison_sdk_root /opt/poky-edison/1.6.1)
set(cross_compiler_sysroot ${edison_sdk_root}/sysroots/i686-pokysdk-linux)
else()
#64 bit SDK
set(edison_sdk_root /opt/poky-edison/1.6.1)
set(cross_compiler_sysroot ${edison_sdk_root}/sysroots/x86_64-pokysdk-linux)
endif()
set(CMAKE_C_COMPILER ${cross_compiler_sysroot}/usr/bin/i586-poky-linux/i586-poky-linux-gcc)
set(CMAKE_CXX_COMPILER ${cross_compiler_sysroot}/usr/bin/i586-poky-linux/i586-poky-linux-g++)
ENDIF(WIN32)
The following defines the rest of the cross platform configuration for the Edison, include folders, compiler flags, etc. No need to make changes here unless you know what you are doing.
set(CMAKE_SYSROOT ${edison_sdk_root}/sysroots/core2-32-poky-linux)
set(CMAKE_FIND_ROOT_PATH ${CMAKE_SYSROOT})
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
SET(CMAKE_C_FLAGS "-Os -g3 -Wall --sysroot=${CMAKE_SYSROOT} -m32 -march=i586 -ffunction-sections -fdata-sections" CACHE STRING "" FORCE)
SET(CMAKE_CXX_FLAGS "-Os -g3 -Wall --sysroot=${CMAKE_SYSROOT} -m32 -march=i586 -ffunction-sections -fdata-sections" CACHE STRING "" FORCE)
INCLUDE_DIRECTORIES(${CMAKE_SYSROOT}/usr/include)
INCLUDE_DIRECTORIES(${CMAKE_SYSROOT}/usr/include/c++)
INCLUDE_DIRECTORIES(${CMAKE_SYSROOT}/usr/include/c++/i586-poky-linux)
And finally this section defines a way to tell Qt to deploy a file
#This is for telling Qt to deploy files
file(WRITE "${CMAKE_SOURCE_DIR}/QtCreatorDeployment.txt" "/\n")
macro(add_deployment_file SRC DEST)
file(RELATIVE_PATH path ${CMAKE_SOURCE_DIR} ${CMAKE_CURRENT_SOURCE_DIR})
file(APPEND "${CMAKE_SOURCE_DIR}/QtCreatorDeployment.txt" "${path}/${SRC}:${DEST}\n")
endmacro()
macro(add_deployment_directory SRC DEST)
file(GLOB_RECURSE files RELATIVE "${CMAKE_CURRENT_SOURCE_DIR}" "${SRC}/*")
foreach(filename ${files})
get_filename_component(path ${filename} PATH)
add_deployment_file("${filename}" "${DEST}/${path}")
endforeach(filename)
endmacro()
CMakeLists.txt
This is the main file for your project. Every cmake project has at least one of these files.
First, I start defining the minimum version of cmake required for this project, and include the Edison.cmake file that I just described.
cmake_minimum_required(VERSION 2.6)
#This is where all the settings to compile for the Edison are stored.
INCLUDE(${CMAKE_SOURCE_DIR}/Edison.cmake)
This is the actual definition of the project. The name of the project (not the executable) is set as EdisonCrossCompile, and the project is set to compile either C or C++ sources. You can set exe_name to define the name for your application (currently it is set as executableName). I also included a script to update the SDK to the latest libraries versions. It is disabled by default as it overwrites the files. If you want to add this feature (at your own risk), uncomment the line. If you do that, every time CMake is called, you will update your SDK. This is usually at the beginning of the project, and when you click on Build->Run CMake, not when you build your project.
Also, any file that is put in this root directory with extension .cpp or .h will be included in the project. This is a simple way to start a new project. More complex projects can, of course, use the full cmake syntax to define their source structure. Also, I add libmraa as an example of how to use external libraries. The rest is to build and link the executable.
PROJECT(EdisonCrossCompile C CXX)
#This is for updating the local SDK to the latest version of mraa and upm
#Uncomment this line to update your SDK every time you run CMake (not every time you build your project)
#INCLUDE(${CMAKE_SOURCE_DIR}/updateSDK.cmake)
#This is the name of the compiled program. Change executableName to anything you want.
set(exe_name executableName)
#Here I am defining the source and header files of the project. You can just put .cpp and .h files into this folder and they will get compiled.
file(GLOB cpp_files *.cpp)
file(GLOB h_files *.h)
#Here you can include extra libraries for your application, like libmraa (http://iotdk.intel.com/docs/master/mraa/) for example
set(extra_libraries -lmraa)
#The actual compilation and linkage
add_executable(${exe_name} ${cpp_files} ${h_files})
target_link_libraries(${exe_name} ${extra_libraries})
Finally, this is used to copy the executable over to the Edison and execute it there.
#This tells Qt which file to copy over to the Edison.
file(RELATIVE_PATH relativeDir ${PROJECT_SOURCE_DIR} ${PROJECT_BINARY_DIR}/${exe_name})
add_deployment_file(${relativeDir} ${deploymentFolderAbsolutPath})
Opening your project in Qt Creator
Once you have your project configured, you will be able to open it under Qt Creator. Click on File->Open File or Project and select your CMakeLists.txt file.
You will see a windows asking you for the build location. This is where the build files are going to be created, and where by default your executable will be created. Usually this is a folder called build under the root of your project. Click Next.
Now under Generator select Unix Generator(Edison). If you don’t see this option, go back to the section called Qt Creator configuration in this guide and do it properly.
Click on Run CMake and wait for a couple of seconds. Everything should work fine. Click on Finish when it is done.
You should now see all your project files listed in the Qt Creator IDE.
Click on the build icon (the brown hammer icon in the lower left section), and your project should build fine. You can actually see the compiled executable file for the Edison ready to be used in the build folder that you selected before.
You can now manually copy this file over to the Edison and it will run fine there. But we will add an extra setting that will allow us to execute it directly from the Qt Creator IDE.
Click on Projects and click on Run (there is a small Build | Run selection right below the name of the project, on the upper side of the window). Under the Run section, under Run configuration, click on Add and select executableName(on Remote Device). Now you should be able to click on the build icon(Hammer symbol), or the build and run icon (green play symbol) and the application will get transferred to the Edison, executed remotely, and you will see the console output on the Qt Creator IDE.
Troubleshooting
Here I will list the most common problems and their solutions.
CMake issues
If anything goes wrong, check that all the paths are correctly set in the cmake script. This is probably where most of the issues will be. You can use message(varName values is: ${varName}) to output the values of cmake variables for debugging.
Also, if you are stuck with an error while changing parameters it is a good idea to delete the entire build directory and start again as sometimes cached variables may still persist with the old values.
Qt Creator issues
If you see this error when building on Qt Creator on Windows: Could not start process “make”, it means that the mingw64 executable was not correctly recognized. The solution here is to go to Projects->Build, and under Build Steps click on Details. Under override command (Available in newer Qt Creator versions) click on Browse and select your mingw32-make.exe file. If you don’t have an override command available, just remove the make step and add a new custom process step and select your mingw32-make.exe file as the command. Repeat the same for the make clean step.
Posted in IoT, Programming, Qt.
– February 26, 2015
