Skip to main content
Version: 3.25.1 (latest)

Facial recognition

Overview

Face SDK allows you to perform the following operations of comparing biometric face templates:

  • Verification (1:1) — comparison of two face templates for belonging to the same person (comparison of two faces).

  • Identification (1:N) — comparison of one biometric face template with other face templates (search for a face in a face database).

The result of recognition is an assessment of the similarity between the compared templates.

Processing blocks

Classes for working with face recognition modules

  • ContextTemplate stores a biometric face template in binary format. Used for comparison and search in recognition modules.
  • DynamicTemplateIndex stores a collection of biometric templates for quick search in MATCHER_MODULE. Supports adding and deleting templates without re-creating the index.

Modifications and versions of face recognition processing blocks

The FACE_TEMPLATE_EXTRACTOR processing block modification determines the template generation speed and recognition accuracy. The higher the speed, the lower the accuracy.

Modification series:

  • 30-1000 - standard modifications, suitable for PCs.
  • 30m-1000m - optimized for mobile processors, work faster, but with less accuracy compared to the standard series.
ModificationVersionTemplate size (bytes)Template creation (ms)
PC*
Template creation (ms)
Mobile**
301 280 19 50
501 288 31 92
1001 280 68 212
10001 296 611 2200

* – Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz.
** – MediaTek Helio G95.

note

The default modification is "1000".

For VERIFICATION_MODULE and MATCHER_MODULE processing blocks, the modification and version determine the type of template that these modules will work with.

Description of ContextTemplate and DynamicTemplateIndex

ContextTemplate

Creating a ContextTemplate

An instance of ContextTemplate can be created in several ways:

  1. Obtained after running the FACE_TEMPLATE_EXTRACTOR module.
  2. Call the service.loadContextTemplate method. This method allows you to load a ContextTemplate object from a binary data stream.

ContextTemplate methods

// Getting the name of a template modification
std::string getMethodName() const;

// Template size
size_t size() const;

// Saving to a binary data stream
void save(std::ostream &binary_stream) const;

DynamicTemplateIndex

Creating a DynamicTemplateIndex

To create a DynamicTemplateIndex instance, use the service.createDynamicTemplateIndex method. The method accepts a Context container with the following parameters:

  • max_license_count (type: long) - The maximum number of templates that the index can contain. Cannot exceed the limit specified in the license. The default value is 100.
  • capacity (type: long) - Reserves space for templates in the index. The default value is max_license_count.
  • async (type: bool) - If true is specified, enables asynchronous mode for adding and removing templates. This allows these operations to be performed in parallel. The mode is in beta.
  • model_version (type: string) - The modification and version of the model, for example, "1000_1". This parameter determines which modification of the FACE_TEMPLATE_EXTRACTOR module the index will work with.

DynamicTemplateIndex methods

// Getting the name of a template method in a database
std::string getMethodName() const;

// Number of templates in the database
size_t size() const;

// Current capacity of the database
size_t capacity() const;

// Getting UUID of a template by index in the database
std::string getUUID(size_t index) const;

// Getting a template by UUID
pbio::ContextTemplate::Ptr at(const std::string& uuid) const;

// Getting a template by index in the database
pbio::ContextTemplate::Ptr at(int64_t index) const;

// Adding a template to the database
void add(pbio::Template::Ptr templ, const std::string& uuid);
void add(const std::vector<pbio::Template::Ptr>& templates, const std::vector<std::string>& uuids);

void add(pbio::ContextTemplate::Ptr templ, const std::string& uuid);
void add(const std::vector<pbio::ContextTemplate::Ptr>& templates, const std::vector<std::string>& uuids);

// Removing a template from the database by UUID
void remove(const std::string& uuid);
void remove(const std::vector<std::string>& uuids);

// Merging template databases
// otherIndex becomes invalid
void concat(DynamicTemplateIndex::Ptr otherIndex);

Specification of facial recognition processing blocks

List of configurable parameters of processing blocks

Face template extractor

  1. The input container Context must contain a binary image and an array of objects obtained after the operation of the face detection and fitter processing blocks:
Click to expand the Context input container specification
{
"image" : {
"format": "NDARRAY",
"blob": "data pointer",
"dtype": "uint8_t",
"shape": [height, width, channels]
},
"objects": [{
"id": {"type": "long", "minimum": 0},
"class": "face",
"confidence": {"double", "minimum": 0, "maximum": 1},
"bbox": [x1, y2, x2, y2]
"keypoints": {
"left_eye_brow_left": {"proj" : [x, y]},
"left_eye_brow_up": {"proj" : [x, y]},
"left_eye_brow_right": {"proj" : [x, y]},
"right_eye_brow_left": {"proj" : [x, y]},
"right_eye_brow_up": {"proj" : [x, y]},
"right_eye_brow_right": {"proj" : [x, y]},
"left_eye_left": {"proj" : [x, y]},
"left_eye": {"proj" : [x, y]},
"left_eye_right": {"proj" : [x, y]},
"right_eye_left": {"proj" : [x, y]},
"right_eye": {"proj" : [x, y]},
"right_eye_right": {"proj" : [x, y]},
"left_ear_bottom": {"proj" : [x, y]},
"nose_left": {"proj" : [x, y]},
"nose": {"proj" : [x, y]},
"nose_right": {"proj" : [x, y]},
"right_ear_bottom": {"proj" : [x, y]},
"mouth_left": {"proj" : [x, y]},
"mouth": {"proj" : [x, y]},
"mouth_right": {"proj" : [x, y]},
"chin": {"proj" : [x, y]},
"points": ["proj": [x, y]]
}
}]
}
  1. After executing the evaluation processing block, the "face_template" field in each object of the "objects" array will contain the biometric template in the "template" field and a unique identifier in the "uuid" field. The template is stored in the ContextTemplate class object. The issued uuid can be used to work with DynamicTemplateIndex or replaced with your own identifier.
Click to expand the Context output container specification
[{
"keypoints": {},
"face_template": {
"template": {"ContextTemplate"},
"uuid": {"string"}
}
}]

Verification module

  1. The Context input container must contain two biometric templates written in the "template1" and "template2" fields. The template type must correspond to the modification of the processing block.
Click to expand the Context input container specification
{
"template1" : {
"template": {"ContextTemplate"}
},
"template2" : {
"template": {"ContextTemplate"}
}
}
  1. After calling the verification processing block, the result will be placed in "result".
Click to expand the Context output container specification
{
"template2": {},
"result": {
"distance": {"double", "minimum": 0},
"score": {"double", "minimum": 0, "maximum": 1},
"far": {"double", "minimum": 0, "maximum": 1},
"frr": {"double", "minimum": 0, "maximum": 1},
}
}

Matcher module

  1. The input container Context must contain an object of the DynamicTemplateIndex class in the "template_index" field and a set of the required biometric templates placed in another object of the DynamicTemplateIndex class in the "queries" field, or one template in the form of an object of the ContextTemplate class.
Click to expand the Context input container specification
{
"queries": {"DynamicTemplateIndex"},
"template_index": {"DynamicTemplateIndex"}
}

OR

{
"queries": {"ContextTemplate"},
"template_index": {"DynamicTemplateIndex"}
}
  1. After calling the verification processing block, the result will be placed in the "results" array.
Click to expand the Context output container specification
{
"template_index": {"DynamicTemplateIndex"}
"result": [{
"index": {"long", "minimum": 0}
"uuid": {"string"}
"distance": {"double", "minimum": 0},
"score": {"double", "minimum": 0, "maximum": 1},
"far": {"double", "minimum": 0, "maximum": 1},
"frr": {"double", "minimum": 0, "maximum": 1},
}]
}

Recognition results

  • distance – distance between the compared template vectors. The lower the value, the higher the confidence in correct recognition.
  • far – probability of false positives, when the system takes images of different people for images of the same person.
  • frr – probability of false negatives, when the system takes two images of the same person for images of different people.
  • score – degree of similarity of faces from 0 (0%) to 1 (100%). A high degree of similarity means that two biometric templates belong to the same person.
  • index - position of the found template within the DynamicTemplateIndex object. (only for MATCHER_MODULE)
  • uuid - unique template identifier passed when adding to DynamicTemplateIndex. (only for MATCHER_MODULE)

Examples of working with face recognition processing blocks

Extracting face templates

To get a face template from an image, do the following:

  1. Create a configuration Context container and specify the "unit_type" values ​​for "FACE_TEMPLATE_EXTRACTOR" and the "modification" and "version" of the modification you are interested in.

An example of creating a processing block can be found here

List of configurable parameters of processing blocks

  1. Pass the Context container obtained after running the face detection and fitter processing blocks.

  2. Call the face template extraction processing block.

auto configCtx = service->createContext();
configCtx["unit_type"] = "FACE_TEMPLATE_EXTRACTOR";
configCtx["modification"] = "{modification}";
pbio::ProcessingBlock blockFaceExtractor = service->createProcessingBlock(configCtx);

//------------------
// creating a processing block and a Context container with a binary image
//------------------

faceDetector(ioData)
faceFitter(ioData)
blockFaceExtractor(ioData);

Saving a face template

To save a face template to a file, follow these steps:

  1. Get the ContextTemplate object from the data.

  2. Open the binary file for writing.

  3. Save the template to a file using the save() method.

pbio::ContextTemplate::Ptr face_template = ioData["objects"][0]["face_template"]["template"].getContextTemplate();

std::ofstream out("template.bin", std::ios::binary);

face_template->save(out);

Loading a face template

To load a face template from a file, follow these steps:

  1. Open the file to read the binary data.

  2. Load the template using loadContextTemplate().

std::ifstream input("template.bin", std::ios::binary);

pbio::ContextTemplate::Ptr face_template = service->loadContextTemplate(input);

Face verification

  1. Create a configuration Context container and specify the "unit_type" values ​​for "VERIFICATION_MODULE" and "modification" and "version" of the modification you are interested in.

  2. Generate a Context container and add biometric templates to the "template1" and "template2" fields:.

  • You can add the contents of the "face_template" field obtained when calling the "FACE_TEMPLATE_EXTRACTOR" module to the "template1" and "template2" fields
  • Or create a new Context container and add an object of the ContextTemplate class to it.
  1. Call the verification processing block.
pbio::Context verificationConfig = service->createContext();
verificationConfig["unit_type"] = "VERIFICATION_MODULE";
verificationConfig["modification"] = "{modification}";

pbio::ProcessingBlock verificationModule = service->createProcessingBlock(verificationConfig);

pbio::Context verificationData = service->createContext();
verificationData["template1"] = ioData["objects"][0]["face_template"];
verificationData["template2"]["template"] = face_template;

verificationModule(verificationData);

Generating DynamicTemplateIndex

To generate DynamicTemplateIndex, follow these steps:

  1. Generate a container-Context and specify the fields "model_version" and "max_license_count" and pass this container to the createDynamicTemplateIndex method

  2. Add templates to the index using the add() method, passing ContextTemplate and uuid

pbio::Context indexConfig = service->createContext();

indexConfig["model_version"] = "{modification}_1";
indexConfig["max_license_count"] = 1000;

pbio::DynamicTemplateIndex::Ptr template_index = service->createDynamicTemplateIndex(indexConfig);

for (const pbio::Context& object : ioData["objects"])
{
template_index->add(object["face_template"]["template"].getContextTemplate(), "<uuid>");
}

note

The uuid passed must be unique

Search for persons in the database

  1. Create a configuration Context container and specify the "unit_type" values ​​for "MATCHER_MODULE" and "modification" and "version" of the modification you are interested in.

  2. Form an input Context container according to the specification for "MATCHER_MODULE".

  3. Call the processing block to search the database.

pbio::Context matcherConfig = service->createContext();
matcherConfig["unit_type"] = "MATCHER_MODULE";
matcherConfig["modification"] = "{modification}";

pbio::ProcessingBlock matcherModule = service->createProcessingBlock(matcherConfig);

// Forming the input Context container for matcherModule
pbio::Context matcherData = service->createContext();
matcherData["template_index"] = template_index;
matcherData["queries"] = ioData["objects"][0]["face_template"]["template"];

matcherModule(matcherData);