Skip to main content
Version: 3.24.1 (latest)

Face detection

Overview​

The face detection process in Face SDK consists of face detection and determination of face landmarks.

Face detection​

Face detection in Face SDK is performed using a set of detectors. Detector is an algorithm of the libfacerec library that uses neural networks to detect faces in images. The result of the detector is the coordinates of a bounding rectangle (bbox) around the detected face.

Detectors​

The following modifications are currently available:

Modification Version Face SDK version Default parameters Detection time CPU (ms)*
640x4801280x7201920x1080
uld13.19precision_level=1, confidence_threshold=0.7, coarse_confidence_threshold=0.37 7 8
precision_level=2, confidence_threshold=0.7, coarse_confidence_threshold=0.337 38 40
precision_level=3, confidence_threshold=0.7, coarse_confidence_threshold=0.3194 187 197
ssyv13.19confidence_threshold=0.5, iou_threshold=0.5151 150 152
246 46 47
396 94 96
43.201517 1506 1502
ssyv_light13.2411 11 12
blf_front13.19confidence_threshold=0.67, iou_threshold=0.53 5 9
blf_back111 13 18
* - CPU Intel Xeon E5-2683 v4 (single-core)
note

"ssyv" is modification by default.

Detector parameters​

  • confidence_threshold — detection confidence threshold.
  • iou_threshold — parameter. The metric determines whether two bboxes refer to the same face. For example, with a threshold of 0.5, two bboxes with an IOU greater than 0.5 are considered to belong to the same face.
  • coarse_confidence_threshold(uld only) — coarse detection confidence threshold. During the detection the detector creates a set of bboxes, for each of them the confidence value is specified (number from 0 to 1, shows the degree of confidence that there is a face in the bbox). The bboxes with confidence_thresholds are fed to the nms algorithm, which determines the intersections (matches) between the bboxes. The coarse_confidence_threshold parameter allows to cut off bboxes with low confidence, which reduces the number of calculations performed by the nms-algorithm.
  • precision_level(uld only) — defines the level of precision. The value is from the range [1, 3], the higher the value the higher the accuracy and the lower the speed. The default value is 1.

Examples of detectors operation in different conditions are presented below. You can customize detection thresholds (confidence_threshold) and other parameters when creating a processing block.

Click here to expand the table
BLF (confidence_threshold=0.6) ULD (precision_level=3, confidence_threshold=0.7)
Click here to expand the table
ULD (confidence_threshold=0.4) ULD (confidence_threshold=0.7)

Face detector specification​

  • The input Context must contain an image in binary format
{
    "image" : {
        "format": "NDARRAY",
        "blob": "data pointer",
        "dtype": "uint8_t",
        "shape": [height, width, channels]
    }
}

Face landmarks​

There are three sets of face landmarks: fda, mesh, tddfa.

  • The fda set contains 21 facial points.
  • The tddfa set contains 68 facial points.
  • The mesh set contains 470 3D facial points. We recommend to use it to get a 3D face mask.

The following modifications are currently available:

Modification Version Face SDK version Detection time CPU (ms)* Detection time GPU (ms)**
fda13.2333
tddfa_faster13.1921
tddfa13.1962
mesh13.1963
* - CPU Intel Xeon E5-2683 v4 (single-core)
** - GPU (NVIDIA GTX 10xx series)
note

The default modification is fda.

Fitter specification​

  • The input Context must contain an image in binary format and objects array from Face Detector.
{
    "image" : {
    "format": "NDARRAY",
        "blob": "data pointer",
        "dtype": "uint8_t",
        "shape": [height, width, channels]
    },
    "objects": [{
        "id": {"type": "long", "minimum": 0},
        "class": "face",
        "confidence": {"double",  "minimum": 0,  "maximum": 1},
        "bbox": [x1, y2, x2, y2]
    }]
}

Example of face detection and estimation of face landmarks​

Create Processing Blocks​

Create a detector and fitter processing block object using the FacerecService.createProcessingBlock method, passing a Context container with set parameters as an argument.

auto detectorConfigCtx = service->createContext();
detectorConfigCtx["unit_type"] = "FACE_DETECTOR";
detectorConfigCtx["modification"] = "ssyv";

auto fitterConfigCtx = service->createContext();
fitterConfigCtx["unit_type"] = "FACE_FITTER";

pbio::ProcessingBlock faceDetector = service->createProcessingBlock(detectorConfigCtx);
pbio::ProcessingBlock faceFitter = service->createProcessingBlock(fitterConfigCtx);

Processing Block configurable parameters

Run face detection​

Pass the Context with a binary image into the Detector Processing Block:

ioData["image"] = imgCtx;
faceDetector(ioData);

The result of face detection is stored by the passed Context container according to the specification of the Processing Block.

Run fitting of face landmarks​

Pass the Context container received after the Face Detector:

faceFitter(ioData)

The resulting Context can be passed to methods for estimating the age, gender, quality, and Liveness (Face Estimation) and to Recognizer.processing to create a template (See Face Recognition).