Skip to main content
Version: 3.17.4

Processing Block API

The Processing Block API is an alternative, scalable interface that replaces existing APIs for easier integration of SDK capabilities into your application.

Key features of Processing Block API:

  • Combining many components into a single integration
  • Simplicity and ease of learning
  • Rapid implementation
  • Long term support and updates

The Processing Block API is a part of the upcoming 3DiVi solutions. For more details, contact your 3DiVi Sales representative.

Set of Processing Blocks​

note

You can find the examples of using Processing Block API below:

Face Detector​

Face Detector is a component used to detect human faces on an image. The detection result is a bounding rectangle (a frame) around the detected face.

Timing characteristics of Face Detector:

  • CPU (Intel Core i5-9400 @ 4.10GHz Turbo, 1 core) ~115ms
  • GPU (GeForce GTX 1080 Ti) ~16ms

You can find the information about using Face Detector on the page Face Detection.

Human Body Detector​

Human Body Detector is a component used to detect human bodies on an image. The detection result is a bounding rectangle (a frame) around the detected body.

Timing characteristics of Human Body Detector:

  • CPU (Intel Core i5-9400 @ 4.10GHz Turbo, 1 core) ~170ms
  • GPU (GeForce GTX 1080 Ti) ~16ms

You can find the information about using Human Body Detector on the page Body Detection.

Object Detector​

Object Detector is a component used to detect multiple objects on an image. The detection result is a bounding rectangle (a frame) around the detected object with classification name. Object Detector is used to detect the following classes of objects on an image: "body", "bicycle", "car", "motorcycle", "bus", "train", "truck", "traffic_light", "fire_hydrant", "stop_sign", "bird", "cat", "dog", "horse", "sheep", "cow", "bear", "backpack", "umbrella", "handbag", "suitcase", "sports_ball", "baseball_bat", "skateboard", "tennis_racket", "bottle", "wine_glass", "cup", "fork", "knife", "laptop", "phone", "book", "scissors".

Timing characteristics of Object Detector:

  • CPU (Intel Core i5-11400H @ 2.70GHz, 1 core) ~1440ms
  • GPU (GeForce GTX 1080 Ti) ~35,6ms

You can find the information about using Object Detector, and a list of objects for detection on the page Object Detection.

Human Pose Estimator​

Human Pose Estimator is a component used to estimate human body skeleton keypoints on an image. The detection result is a list of keypoints with their coordinates and confidence score of the detected human body. Human Pose Estimator is used to detect the following keypoints on an image: "nose", "left_eye", "right_eye", "left_ear", "right_ear", "left_shoulder", "right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist", "left_hip", "right_hip", "left_knee", "right_knee", "left_ankle", "right_ankle"

Timing characteristics of Human Pose Estimator:

  • CPU (Intel Core i5-11400H @ 2.70GHz, 1 core) ~55ms
  • GPU (GeForce GTX 1080 Ti) ~15ms

You can find the information about using Human Pose Estimator on the page Human Pose Estimator.

Emotion Estimator​

Emotion Estimator is a component used to estimate human emotions from the cropped face image. The estimation result is a confidence for every emotion estimated.

Timing characteristics of Emotion Estimator:

  • CPU (Intel Core i5-9400 @ 4.10GHz Turbo, 1 core) ~18ms
  • GPU (GeForce GTX 1080 Ti) ~8ms

You can find the information about using Emotion Estimator on the page Emotion Estimation.

Age Estimator​

Age Estimator is a component used to estimate a human age from the cropped image with a face. The estimation result is a human age.

Timing characteristics of Age Estimator:

  • CPU (Intel Core i5-9400 @ 4.10GHz Turbo, 1 core) ~5ms
  • GPU (GeForce GTX 1080 Ti) ~1ms

You can find the information about using Age Estimator on the page Age Estimation.

Gender Estimator​

Gender Estimator is a component used to estimate a human gender from the cropped image with a face. The estimation result is a verdict about gender identity.

Timing characteristics of Gender Estimator:

  • CPU (Intel Core i5-9400 @ 4.10GHz Turbo, 1 core) ~5ms
  • GPU (GeForce GTX 1080 Ti) ~1ms

You can find the information about using Gender Estimator on the page Gender Estimation.

Mask Estimator​

Mask Estimator is a component used to estimate the presence of a medical mask on the cropped face image. The estimation result is a verdict about the presence of a mask.

You can find the information about using Mask Estimator on the page Mask Estimation.

2D RGB Liveness Estimator​

2D RGB Liveness Estimator is a component used to estimate human liveness on a single colored image. The detection result is a bounding rectangle (a frame) around the detected human face with liveness verdict and score.

You can find the information about using 2D RGB Liveness Estimator on the page Liveness Estimation.

Quality Assessment Estimator​

Quality Assessment Estimator is a component used to assess the quality of a face in an image for identification tasks on a single colored image. The detection result is a list of the detected human faces with a verbose quality score.

You can find the information about using Quality Assessment Estimator on the page Quality Assessment Estimation.

Context​

Processing Block API is based on the use of Context.

Context is a heterogeneous container that consists of a set of hierarchically organized data presented in the form of key–value pairs. The closest analogue of Context is a JSON object. Each Context object can contain a scalar object (integer, real, boolean, string), a memory area or pointer, a sequential array of Context objects, or an associative container of string-Context pairs, with unlimited nesting.

How to create and use a Context object​

  1. Create a FacerecService.

  2. Then create a Context container:

auto array_elem0 = service->createContext();
  1. Common set of operations with a Context container:
  • creating an associative container by calling ["key"] on empty Context:
    array_elem0["name"] = "Julius Zeleny";      // put string
    array_elem0["phone"] = 11111111111l;        // put integer (long)
    array_elem0["social_score"] = 0.999;        // put double
    array_elem0["verified"] = true;             // put bool
  • getters:
    ASSERT_EQ( array_elem0["name"].getString(), "Julius Zeleny" );
    ASSERT_EQ( array_elem0["phone"].getLong(), 11111111111l );
    ASSERT_EQ( array_elem0["social_score"].getDouble(),  0.999 );
    ASSERT_EQ( array_elem0["verified"].getBool(), true );
  • creating a sequence array by calling push_back on empty Context:
    auto service->createContext();
    array.push_back(array_elem0);
  • iterating over array:

    // get by index
    ASSERT_EQ( array[0]["phone"].getLong(), 11111111111l );

    // iterate with index
    size_t array_sz = array.size();
    for(size_t i = 0; i < array_sz; ++i)
        array[i]["phone"];

    // or with iterators
    for(auto iter = array.begin(); iter != array.end(); ++iter)
        (*iter)["phone"]; // deference returns nested Context

    // with foreach
    for(auto val : array)
        val["phone"];
  • operations with a nested associative container:
    auto full = service->createContext();
    full["friends"] = std::move(array); // move assignment without copying

    // access nested object
    ASSERT_EQ( full["friends"][0]["social_score"].getDouble(), 0.999 );

    // iterate over associative containers values
    for(auto iter = full.begin(); iter != full.end(); ++iter) {
        iter.key(); // get key value from iterator
        (*iter)[0]["social_score"].getDouble(); // get value
    }

    // with foreach
    for(auto val : full)
        val[0]["social_score"].getDouble();
  • other Context's convenient methods:
    void clear()
    bool contains(const std::string& key)       // for an assosiative container
    Context operator[](size_t index)            // for a sequence array,  access specified element with bounds checking
    Context operator[](const std::string& key)  // for an assosiative container, access or insert
    Context at(const std::string& key)          // for an assosiative container, with bounds checking
    size_t size()                               // return elements count for a container
    bool isNone()                               // is empty
    bool isArray()                              // is a sequence array
    bool isObject()                             // is an assosiative container
    bool isLong(), isDouble(), isString(), isBool() // check if contains a certain scalar type

Binary Image Format​

Most of the processing blocks operates on Context with an image in binary format:

/*
{
 "image" : { "format": "NDARRAY",
             "blob": <data pointer>,
             "dtype": "uint8_t",
             "shape": [height, width, channels] }
}
*/

The "blob" key contains a smart pointer to data. The pointer is set by the function void Context::setDataPtr(void* ptr, int copy_sz), where copy_sz is the size of memory in Bytes, that will be copied, and then automatically released when Context object's lifetime ends. Copying will not perform if 0 is passed as argument copy_sz. In this case the Context object does not control the lifetime of the object it points to.

You can also allocate a raw memory, f.e. to copy data later, passing nullptr and size as arguments of setDataPtr.

The "dtype" can contain one of these values: "uint8_t", "int8_t", "uint16_t", "int16_t", "int32_t", "float", "double". This is according to OpenCV types: CV_8U, CV_8S, CV_16U, CV_16S, CV_32S, CV_32F, CV_64F.

Creating a Context container with RGB-image​

  1. Create a FacerecService.

  2. Create a Context container for image using the createContext() method:

auto imgCtx = service->createContext();
  1. Read an RGB-image from file:
// read the image from file
std::string input_image_path = "<path_to_image>";
cv::Mat image = cv::imread(input_image_path, cv::IMREAD_COLOR);
cv::Mat input_image;
cv::cvtColor(image, input_image, cv::COLOR_BGR2RGB);
  1. a. Put an image into container OR
// using pbio::context_utils::putImage(Context& ctx, unsigned char* data, size_t height, size_t width, pbio::IRawImage::Format format, bool copy)
pbio::context_utils::putImage(imgCtx, input_image.data, input_image.rows, input_image.cols, pbio::IRawImage::FORMAT_RGB, true);

  1. b. OR copy an image from pbio::RawImage, pbio::CVRawImage, pbio::InternalImageBuffer to binary format and put it to Context container:
// constructing pbio::RawImage
pbio::RawImage input_rawimg(input_image.cols, input_image.rows, pbio::RawImage::Format::FORMAT_RGB, input_image.data);

// using void putImage(Context& ctx, const RawImage& raw_image)
pbio::context_utils::putImage(imgCtx, input_rawimg);

Creating a Processing Block​

This template can be used to create any processing block. Keys unit_type and model_path must be specified according to block you want to use (see description of specific processing block).

  1. Create a FacerecService.

  2. Create a Context container:

auto configCtx = service->createContext();
  1. Define fields in the created context container for creating a Processing Block:
// mandatory, specify the name of processing block
configCtx["unit_type"] = "<name_of_processing_block>";

// mandatory, specify the path to model file of processing block
configCtx["model_path"] = "<path_to_model_file>";

// default location of the onnxruntime library in <FaceSDKShortProductName /> distributive: the "lib" folder for Linux platfrom or the "bin" folder for Windows platfrom
// you can specify your own path to onnxruntime library
// if value is not specified, the os-specific default search order will be used
configCtx["ONNXRuntime"]["library_path"] = "../lib"; // for Linux
configCtx["ONNXRuntime"]["library_path"] = "../bin"; // for Windows

// optional, "true" if you want to use GPU acceleration (CUDA) for processing block that support it
configCtx["use_cuda"] = false;
  1. Create a Processing Block:
pbio::ProcessingBlock processing_block = service->createProcessingBlock(configCtx);

GPU Acceleration​

Processing Blocks can be used with GPU acceleration (CUDA). To activate acceleration you need to define the ["use_cuda"] key with the true value for Processing Block configuration container (see Creating a Processing Block).

Requirements​