Qt and Intel realsense Getting started

Question

I didn't know where to ask as a beginner. So If you can help...

I want to create a project using:

• Qt 5.15.0
• Visual studio 2019
• C++
• openCV 4.2.0 (if is necessary)
• Intel realsense camera

Actually to start, I need a simpler project like those from the: Examples for Intel RealSense SDK 2.0,

1. the "im-show"
2. or/and the "hello-realsense"

A code example will be the best or even better a tutorial.
Till now I didn't found something on google.
In case the above is difficult an advice on how to get start will be fine.

For example, I found: "Qt and openCV on Windows with MSVC"
Is it a good place for me to start, do I need the openCV to show/display the depth image like the project "im-show" does?

Thank you in advance.

Answer 1

This is a very simple example, which uses only Qt and Intel Realsense SDK.

We start by writing a class that handles our camera:

#ifndef CAMERA_H
#define CAMERA_H

// Import QT libs, one for threads and one for images
#include <QThread> 
#include <QImage>

// Import librealsense header
#include <librealsense2/rs.hpp>

// Let's define our camera as a thread, it will be constantly running and sending frames to
// our main window
class Camera : public QThread 
{
    Q_OBJECT
public:
    // We need to instantiate a camera with both depth and rgb resolution (as well as fps)
    Camera(int rgb_width, int rgb_height, int depth_width, int depth_height, int fps);
    ~Camera() {}

    // Member function that handles thread iteration
    void run();
    
    // If called it will stop the thread
    void stop() { camera_running = false; }

private:
    // Realsense configuration structure, it will define streams that need to be opened
    rs2::config cfg;

    // Our pipeline, main object used by realsense to handle streams
    rs2::pipeline pipe;

    // Frames returned by our pipeline, they will be packed in this structure
    rs2::frameset frames;

    // A bool that defines if our thread is running
    bool camera_running = true;

signals:
    // A signal sent by our class to notify that there are frames that need to be processed
    void framesReady(QImage frameRGB, QImage frameDepth);
};
// A function that will convert realsense frames to QImage
QImage realsenseFrameToQImage(const rs2::frame& f);

#endif // CAMERA_H

To fully understand what this class do I redirect you to these two pages: Signals & Slots and QThread. This class is a QThread, that means that it can run parallelly with our main window. When a couple of frames is ready, the signal framesReady will be emitted and the window will show the images.

Let's start by saying how to open the camera streams with librealsense:

Camera::Camera(int rgb_width, int rgb_height, int depth_width, int depth_height, int fps)
{
    // Enable depth stream with given resolution. Pixel will have a bit depth of 16 bit
    cfg.enable_stream(RS2_STREAM_DEPTH, depth_width, depth_height, RS2_FORMAT_Z16, fps);
    
    // Enable RGB stream as frames with 3 channel of 8 bit
    cfg.enable_stream(RS2_STREAM_COLOR, rgb_width, rgb_height, RS2_FORMAT_RGB8, fps);

    // Start our pipeline
    pipe.start(cfg);
}

As you can see our constructor is very simple and it will only open the pipeline with the given stream.

Now that the pipeline is started we only need to get the corresponding frames. We'll do it in our 'run' method, the method that will be launched when the QThread starts:

void Camera::run()
{
    while(camera_running)
    {
        // Wait for frames and get them as soon as they are ready
        frames = pipe.wait_for_frames();

        // Let's get our depth frame
        rs2::depth_frame depth = frames.get_depth_frame();
        // And our rgb frame
        rs2::frame rgb = frames.get_color_frame();

        // Let's convert them to QImage
        auto q_rgb = realsenseFrameToQImage(rgb);
        auto q_depth = realsenseFrameToQImage(depth);

        // And finally we'll emit our signal
        emit framesReady(q_rgb, q_depth);
    }
}

The function that does the conversion is the following:

QImage realsenseFrameToQImage(const rs2::frame &f)
{
    using namespace rs2;

    auto vf = f.as<video_frame>();
    const int w = vf.get_width();
    const int h = vf.get_height();

    if (f.get_profile().format() == RS2_FORMAT_RGB8)
    {
        auto r = QImage((uchar*) f.get_data(), w, h, w*3, QImage::Format_RGB888);
        return r;
    }
    else if (f.get_profile().format() == RS2_FORMAT_Z16)
    {
        // only if you have Qt > 5.13
        auto r = QImage((uchar*) f.get_data(), w, h, w*2, QImage::Format_Grayscale16);
        return r;
    }

    throw std::runtime_error("Frame format is not supported yet!");
}

Our Camera is done.

Now we'll define our main window. We'll need a slot that receives our frames and two labels where we will put our images:

#ifndef MAINWINDOW_H
#define MAINWINDOW_H

#include <QMainWindow>
#include <QLabel>

class MainWindow : public QMainWindow
{
    Q_OBJECT

public:
    explicit MainWindow(QWidget *parent = 0);

public slots:
    // Slot that will receive frames from the camera
    void receiveFrame(QImage rgb, QImage depth);

private:
    QLabel *rgb_label;
    QLabel *depth_label;
};

#endif // MAINWINDOW_H

We create a simple view for the window, with the images that will be shown vertically.

#include "mainwindow.h"

MainWindow::MainWindow(QWidget *parent) :
    QMainWindow(parent)
{
    // Creates our central widget that will contain the labels
    QWidget *widget = new QWidget();
 
    // Create our labels with an empty string
    rgb_label = new QLabel("");
    depth_label = new QLabel("");

    // Define a vertical layout
    QVBoxLayout *widgetLayout = new QVBoxLayout;

    // Add the labels to the layout
    widgetLayout->addWidget(rgb_label);
    widgetLayout->addWidget(depth_label);

    // And then assign the layout to the central widget
    widget->setLayout(widgetLayout);

    // Lastly assign our central widget to our window
    setCentralWidget(widget);
}

And now we need to define the slot function. The only job assigned to that function is to change the images related to the labels:

void MainWindow::receiveFrame(QImage rgb, QImage depth)
{
    rgb_label->setPixmap(QPixmap::fromImage(rgb));
    depth_label->setPixmap(QPixmap::fromImage(depth));
}

And we're done!

Lastly, we write our main, which will starts our thread and show our window.

#include <QApplication>
#include "mainwindow.h"
#include "camera.h"

int main(int argc, char *argv[])
{
    QApplication a(argc, argv);

    MainWindow window;
    Camera camera(640, 480, 320, 240, 30);

    // Connect the signal from the camera to the slot of the window
    QApplication::connect(&camera, &Camera::framesReady, &window, &MainWindow::receiveFrame);

    window.show();

    camera.start();

    return a.exec();
}

Answer 2

From your question, I understand you are trying to preview the camera data in Qt application.

1. writing imshow() inside Qt application is not possible.
2. You have to convert the OpenCV Mat to QImage like below and display it in your viewer(widget/QML)

Mat img;
QImage img1 = QImage((uchar *) img.data,img.cols,img.rows,img.step,QImage::Format_Indexed8);