Skip to main content
Version: 3.22.0 (latest)

Face detection

Overview​

The face detection process in Face SDK consists of face detection and determination of face landmarks.

Face detection​

Face detection in Face SDK is performed using a set of detectors. Detector is an algorithm of the libfacerec library that uses neural networks to detect faces in images. The result of the detector is the coordinates of a bounding rectangle (bbox) around the detected face.

Detectors​

The following modifications are currently available:

Modification Version Default parameters Detection time (ms)
640x4801280x7201920x1080
uld1precision_level=1, confidence_threshold=0.7, coarse_confidence_threshold=0.37 8 9
precision_level=2, confidence_threshold=0.7, coarse_confidence_threshold=0.337 40 43
precision_level=3, confidence_threshold=0.7, coarse_confidence_threshold=0.3193 192 203
ssyv1confidence_threshold=0.5, iou_threshold=0.5153 153 157
247 49 54
396 99 104
41355 1419 1420
blf_front1confidence_threshold=0.67, iou_threshold=0.54 6 11
blf_back112 14 20
note

"ssyv" is modification by default.

Detector parameters​

  • confidence_threshold — detection confidence threshold.
  • iou_threshold — parameter. The metric determines whether two bboxes refer to the same face. For example, with a threshold of 0.5, two bboxes with an IOU greater than 0.5 are considered to belong to the same face.
  • coarse_confidence_threshold(uld only) — coarse detection confidence threshold. During the detection the detector creates a set of bboxes, for each of them the confidence value is specified (number from 0 to 1, shows the degree of confidence that there is a face in the bbox). The bboxes with confidence_thresholds are fed to the nms algorithm, which determines the intersections (matches) between the bboxes. The coarse_confidence_threshold parameter allows to cut off bboxes with low confidence, which reduces the number of calculations performed by the nms-algorithm.
  • precision_level(uld only) — defines the level of precision. The value is from the range [1, 3], the higher the value the higher the accuracy and the lower the speed. The default value is 1.

Examples of detectors operation in different conditions are presented below. You can customize detection thresholds (confidence_threshold) and other parameters when creating a processing block.

Click here to expand the table
BLF (confidence_threshold=0.6) ULD (precision_level=3, confidence_threshold=0.7)
Click here to expand the table
ULD (confidence_threshold=0.4) ULD (confidence_threshold=0.7)

Face detector specification​

  • The input Context must contain an image in binary format
{
    "image" : {
        "format": "NDARRAY",
        "blob": "data pointer",
        "dtype": "uint8_t",
        "shape": [height, width, channels]
    }
}

Face landmarks​

There are three sets of face landmarks: fda, mesh, tddfa.

  • The tddfa set contains 68 facial points.
  • The mesh set contains 470 3D facial points. We recommend to use it to get a 3D face mask.

The following modifications are currently available:

Modification Version Detection time (ms)
tddfa_faster13
tddfa17
mesh17
note

The default modification is tddfa_faster.

Fitter specification​

  • The input Context must contain an image in binary format and objects array from Face Detector.
{
    "image" : {
    "format": "NDARRAY",
        "blob": "data pointer",
        "dtype": "uint8_t",
        "shape": [height, width, channels]
    },
    "objects": [{
        "id": {"type": "long", "minimum": 0},
        "class": "face",
        "confidence": {"double",  "minimum": 0,  "maximum": 1},
        "bbox": [x1, y2, x2, y2]
    }]
}

Example of face detection and estimation of face landmarks​

Create Processing Blocks​

Create a detector and fitter processing block object using the FacerecService.createProcessingBlock method, passing a Context container with set parameters as an argument.

auto detectorConfigCtx = service->createContext();
detectorConfigCtx["unit_type"] = "FACE_DETECTOR";
detectorConfigCtx["modification"] = "ssyv";

auto fitterConfigCtx = service->createContext();
fitterConfigCtx["unit_type"] = "FACE_FITTER";

pbio::ProcessingBlock faceDetector = service->createProcessingBlock(detectorConfigCtx);
pbio::ProcessingBlock faceFitter = service->createProcessingBlock(fitterConfigCtx);

Processing Block configurable parameters

Run face detection​

Pass the Context with a binary image into the Detector Processing Block:

ioData["image"] = imgCtx;
faceDetector(ioData)

The result of face detection is stored by the passed Context container according to the specification of the Processing Block.

Run fitting of face landmarks​

Pass the Context container received after the Face Detector:

faceFitter(ioData)

The resulting Context can be passed to methods for estimating the age, gender, quality, and Liveness (Face Estimation) and to Recognizer.processing to create a template (See Face Recognition).