Components
Face SDK consists of components that perform the main functional tasks: detection, estimation of faces and human pose, facial recognition and processing of video streams. Face SDK components are implemented as processing blocks of Processing Block API and/or Legacy API objects.
Detection of faces, bodies and objects
Face Detector
Face Detector is a component for detecting faces in images. The result of the component's work is a list of detected faces with the following attributes:
- Bounding Box (bbox) - coordinates of a rectangle that represents face bounds in the image.
- Face Landmarks - 2D/3D coordinates of the anthropometric points of the face.
- Iris Landmarks - coordinates of 40 points of the eyes (pupils and eyelids).
- Pitch, Yaw, Roll - head rotation angles.
Component implementation:
- Processing Block API: Face Detector and Face Fitter processing blocks
- Legacy API: Capturer object
For face detection on videos or time-ordered image sequences, we recommend that you use Video Engine.
Body Detector
Body Detector is used to detect human bodies within images, bboosting the potential to detect people within the frame even when their faces are not discernible. The detection result is a bbox around the detected body.
Component implementation:
- Processing Block API: Body Detector processing block
Object Detector
Object Detector is used to detect multiple various objects in images.
The detection result is a bbox around the detected object with object class: "body"
, "bicycle"
, "car"
, "motorcycle"
, "bus"
, "train"
, "truck"
, "traffic_light"
, "fire_hydrant"
, "stop_sign"
, "bird"
, "cat"
, "dog"
, "horse"
, "sheep"
, "cow"
, "bear"
, "backpack"
, "umbrella"
, "handbag"
, "suitcase"
, "sports_ball"
, "baseball_bat"
, "skateboard"
, "tennis_racket"
, "bottle"
, "wine_glass"
, "cup"
, "fork"
, "knife"
, "laptop"
, "phone"
, "book"
, "scissors"
.
Component implementation:
- Processing Block API: Object Detector processing block
Estimation of faces and human pose
Face SDK provides a set of tools for estimating images received from Face Detector component, offering insightful demographic analysis.Gender-Age Estimator
Gender-Age Estimator is used for estimating the gender and age of people based on their face images.
Component implementation:
- Processing Block API: Gender Estimator and Age Estimator processing blocks
- Legacy API: AgeGenderEstimator object
Emotions Estimator
Emotions Estimator is used for estimating the prevailing emotional state of a person:
- Happy
- Surprised
- Neutral
- Angry
- Disgusted
- Sad
- Scared
Component implementation:
- Processing Block API: Emotion Estimator processing block
- Legacy API: EmotionsEstimator object
Quality Estimator
Quality Estimator is used to estimate face image quality. The result is a list of the human face images with a detailed quality score.
Component implementation:
- Processing Block API: Quality Assessment Estimator processing block
- Legacy API: QualityEstimator и FaceQualityEstimator object
Mask Estimator
Mask Estimator determines the presence/absence of a mask in the face image.
Component implementation:
- Processing Block API: Mask Estimator processing block
- Legacy API: FaceAttributesEstimator object
Eyes Openness Estimator
Eyes Openness Estimator is used for estimating the eyes’ state on the face image. This component provides the verdict “open” or “closed” for the right and left eye.
Component implementation:
- Processing Block API: as a part of Quality Assessment Estimator processing block
- Legacy API: FaceAttributesEstimator object
Liveness Estimators
Liveness components are used to assess whether the face detected in the image or in video is real or fake. These components protect against malicious actions (spoofing attacks) using printed face image, a photo or video of a face from the screens of mobile devices and monitors, as well as various kinds of masks (paper, silicone, etc.).
Active Liveness Estimator analyzes certain human actions according to the check script, for example: “blink”, “smile”, “turn your head”.
Component implementation:
- Face SDK API: VideoWorker object
2D / RGB Liveness Estimator assesses face liveness in an RGB image. To perform the check, the appearance of the face in the field of view of the camera is sufficient.
Component implementation:
- Processing Block API: Liveness Estimator processing block
- Legacy API: Liveness2DEstimator object
3D / Depth Liveness Estimator protects against attempts to use an image instead of a real face by analyzing the face surface using a depth map obtained from a 3D (RGBD) sensor.
Component implementation:
- Legacy API: DepthLivenessEstimator object
IR Liveness Estimator determines the reality of a human face based on an image taken from an infrared camera, combined with a color image.
Component implementation:
- Legacy API: IRLivenessEstimator object
Human Pose Estimator
Human Pose Estimator is a component used to estimate human body skeleton keypoints in the image.
Component implementation:
- Processing Block API: Human Pose Estimator processing block
Facial recognition
Face SDK provides components and algorithms to recognize faces. This functionality is based on the operations with a biometric face template.Encoder
Encoder extracts biometric face template from a face image received from Face Detector.
A biometric face template is a unique set of biometric features extracted from a face image. Templates are used to compare two face images and to determine a degree of their similarity.
A biometric face template has the following key characteristics:
- It does not contain personal data
- It cannot be used to restore a face image
- It can be serialized and saved to file, database, or sent over a network
- It can be indexed. That helps accelerate face template matching by using a special index for face template batch.
Extracting a biometric template is one of the most computation-heavy operations, so Face SDK provides the ability to use a GPU accelerator to increase performance.
Component implementation:
- Processing Block API: Face Template Extractor
- Legacy API: Recognizer object
Matcher
Matcher allows performing the following comparison operations with templates created by Encoder:
- Verification 1:1 - comparing of two biometric templates (faces) between each other, estimating of coincidence.
- Identification 1:N - comparing of one biometric template (face) with other templates (faces), searching and estimating of coincidence.
When comparing face templates, Matcher calculates the difference between biometric features of faces. The calculation result is a measure of coincidence between face images and the probability of belonging to one person.
Templates, extracted using different algorithms, have different properties and cannot be compared with each other.
Component implementation:
- Processing Block API: Verification module and Matcher module
- Legacy API: Recognizer object
Video stream processing
Video Engine
Video Engine is used for real-time processing of video streams. This component solves the following tasks:
- Face detection and tracking
- Facial recognition (optional)
- Liveness checking (optional)
- Estimation of age, gender, and emotions (optional)
Video Engine works in a multi-stream mode. Each stream is a sequence of images (frames) obtained from one source (for example, a camera or video).
All streams are processed by Video Engine at the same time. Streams, frames, and detected faces in the frame are assigned their own identifiers. During face tracking, on the sequence of stream images a face track is formed, which is also indicated by its own ID (track_id).
The set of identifiers allows you to accurately track the generated events for each stream. To handle the events, Video Engine implements a callback interface that provides event data.
Component implementation:
- Legacy API: VideoWorker object