Video Stream Processing
The VideoWorker
interface object is used to:
- track faces on video streams
- create templates
- match templates with the database
- estimate age, gender, and emotions
- estimate liveness
- match the faces detected in a specified period with each other
The VideoWorker
object is responsible for thread control and synchronization routine. You only need to provide decoded video frames and register a few callback functions.
See an example of using VideoWorker
in video_recognition_demo.
You can learn how to detect and track faces in a video stream in our tutorial Face Detection and Tracking in a Video Stream.
You can learn how to recognize faces in a video stream in our tutorial Face Recognition in a Video Stream.
Track Faces
Note: Learn how to detect masked faces in our tutorial.
VideoWorker
can be created with FacerecService.createVideoWorker
.
Examples
- C++
- C#
- Java
- Python
pbio::FacerecService::Config video_worker_config("video_worker_lbf.xml");
video_worker_config.overrideParameter("search_k", 3);
pbio::VideoWorker::Ptr video_worker = service->createVideoWorker(
pbio::VideoWorker::Params()
.video_worker_config(video_worker_config)
.recognizer_ini_file(recognizer_config)
.streams_count(streams_count)
.processing_threads_count(processing_threads_count)
.matching_threads_count(matching_threads_count)
.age_gender_estimation_threads_count(age_gender_estimation_threads_count)
.emotions_estimation_threads_count(emotions_estimation_threads_count)
.short_time_identification_enabled(enable_sti)
.short_time_identification_distance_threshold(sti_recognition_threshold)
.short_time_identification_outdate_time_seconds(sti_outdate_time)
);
FacerecService.Config video_worker_config = new FacerecService.Config("video_worker_lbf.xml");
video_worker_config.overrideParameter("search_k", 3);
VideoWorker video_worker = service.createVideoWorker(
new VideoWorker.Params()
.video_worker_config(video_worker_config)
.recognizer_ini_file(recognizer_config)
.streams_count(streams_count)
.processing_threads_count(processing_threads_count)
.matching_threads_count(matching_threads_count)
.age_gender_estimation_threads_count(age_gender_estimation_threads_count)
.emotions_estimation_threads_count(emotions_estimation_threads_count)
.short_time_identification_enabled(enable_sti)
.short_time_identification_distance_threshold(sti_recognition_threshold)
.short_time_identification_outdate_time_seconds(sti_outdate_time)
);
FacerecService.Config video_worker_config = service.new Config("video_worker_lbf.xml");
video_worker_config.overrideParameter("search_k", 3);
VideoWorker video_worker = service.createVideoWorker(
new VideoWorker.Params()
.video_worker_config(video_worker_config)
.recognizer_ini_file(recognizer_config)
.streams_count(streams_count)
.processing_threads_count(processing_threads_count)
.matching_threads_count(matching_threads_count)
.age_gender_estimation_threads_count(age_gender_estimation_threads_count)
.emotions_estimation_threads_count(emotions_estimation_threads_count)
.short_time_identification_enabled(enable_sti)
.short_time_identification_distance_threshold(sti_recognition_threshold)
.short_time_identification_outdate_time_seconds(sti_outdate_time)
);
video_worker_config = Config("video_worker_lbf.xml")
video_worker_config.override_parameter("search_k", 3)
video_worker_params = video_worker.Params()
video_worker_params.video_worker_config = video_worker_config
video_worker_params.recognizer_ini_file = recognizer_config
video_worker_params.streams_count = streams_count
video_worker_params.processing_threads_count = processing_threads_count
video_worker_params.matching_threads_count = matching_threads_count
video_worker_params.age_gender_estimation_threads_count = age_gender_estimation_threads_count
video_worker_params.emotions_estimation_threads_count = emotions_estimation_threads_count
video_worker = service.create_video_worker(video_worker_params)
Where:
video_worker_config
– Path to the configuration file forVideoWorker
orFacerecService.Config
object.video_worker_params
– Parameters of theVideoWorker
constructor.recognizer_config
– The configuration file for the recognizer used (see Face Identification).streams_count
– The number of video streams; a tracking stream is created for each stream.processing_threads_count
– The number of threads for template creation. These threads are common to all video streams and distribute resources evenly across all video streams regardless of their workload (except for the video streams without faces in the frame).matching_threads_count
– The number of threads for comparison of templates created from video streams with the database. Like processing threads, these threads distribute the workload evenly across all video streams.age_gender_estimation_threads_count
– The number of threads for age and gender estimation. Like processing threads, these threads distribute the workload evenly across all video streams.emotions_estimation_threads_count
– The number of threads for emotions estimation. Like processing threads, these threads distribute the workload evenly across all video streams.enable_sti
– The flag enabling short time identification.sti_recognition_threshold
– The recognition distance threshold for short time identification.sti_outdate_time
– Time period in seconds for short time identification.
Currently, there are three configuration files with the tracking method from common_video_capturer.xml
:
video_worker.xml
with the esr points set.video_worker_lbf.xml
with the singlelbf points set.video_worker_fda.xml
with the fda points set.
and three configuration files with the tracking method from fda_tracker_capturer.xml
:
video_worker_fdatracker.xml
with the fda points set.video_worker_fdatracker_fake_detector.xml
with the fda points set.video_worker_fdatracker_blf_fda.xml
with the fda points set.
(see Anthropometric Points, Capturer Class Reference).
If VideoWorker
is used only for face tracking, it must be created with matching_thread=0 and processing_thread=0
using the standard Face Detector license. To create Face Detector for one stream, specify the streams_count=1
parameter.
To provide video frames, call VideoWorker.addVideoFrame
. This method is thread-safe, so you can provide frames from different streams created for each video stream, without additional synchronization. The method returns an integer frame id that will be used to identify this frame in the callback.
You have to use two callbacks for face tracking:
VideoWorker.TrackingCallbackU
provides the tracking results. This callback is called every time the frame is processed by the tracking conveyor. Tracking callback will be called withframe_id
equal toX
not earlier thanVideoWorker.addVideoFrame
returns the value ofX + N - 1
, whereN
is the value returned byVideoWorker.getTrackingConveyorSize
. Tracking callbacks with the samestream_id
are called in ascendingframe_id
order. Therefore, if a callback withstream_id = 2
andframe_id = 102
was received immediately after a callback withstream_id = 2
andframe_id = 100
, then the frame withframe_id = 101
was skipped for the video stream 2. Most of the samples are created from theframe_id
frame, but some samples can be obtained from previous frames. To determine which frame the sample actually belongs to, use theRawSample.getFrameID
method. To subscribe to this callback, use theVideoWorker.addTrackingCallbackU
method. To unsubscribe from this method, use theVideoWorker.removeTrackingCallback
method by submitting thecallback_id
you received from theVideoWorker.addTrackingCallbackU
method.VideoWorker.TrackingLostCallbackU
returns the best sample and face template when tracking is lost (for example, when a person leaves the frame). The best sample can be empty if theweak_tracks_in_tracking_callback
configuration parameter is enabled. It is guaranteed that this is the last callback for the pair<stream_id, track_id>
(track_id
is equal tosample.getID()
for a sample given in anyVideoWorker
callback). That is, after this callback noTracking
,MatchFound
orTrackingLost
callback for thisstream_id
can contain a sample with the sametrack_id
identifier. It is also guaranteed that for each pair<stream_id, track_id>
, mentioned in theTracking
callback, there is exactly oneTrackingLost
callback, except for the tracks removed duringVideoWorker.resetStream
– theTrackingLost
callback will not be called for these tracks. To release the memory allocated for these tracks, use thereturn
value ofVideoWorker.resetStream
. To subscribe to this callback, use theVideoWorker.addTrackingLostCallbackU
method. To unsubscribe from this callback, use theVideoWorker.removeTrackingLostCallback
method by providing thecallback_id
that you received from theVideoWorker.addTrackingLostCallbackU
method.
Note: Exceptions thrown in the callbacks will be catched and rethrown in the VideoWorker.checkExceptions
member function. Therefore, call the VideoWorker.checkExceptions
method from time to time to check for errors.
WARNING: Do not call the methods that change the state of VideoWorker
inside the callbacks in order to avoid a deadlock. That is, only the VideoWorker.getMethodName
and VideoWorker.getStreamsCount
member functions are safe for calling in callbacks.
Create Templates
If besides detection, the creation of templates is required, VideoWorker
must be created with matching_thread=0
and processing_thread>0
, using the Video Engine Standard license. To create Video Engine Standard for one stream, specify the parameters streams_count=1
, processing_threads_count=1
, matching_threads_count=0
.
You can enable / disable the creation of templates for a specific video stream using the VideoWorker.disableProcessingOnStream
and VideoWorker.enableProcessingOnStream
member functions. At start, template creation is enabled for all video streams.
VideoWorker.TemplateCreatedCallbackU
provides template generation results. This callback is called, whenever a template is created within the VideoWorker
. It is guaranteed that this callback will be called after at least one Tracking
callback and before a TrackingLost
callback with the same stream_id
and track_id
(track_id = sample->getID()
). To subscribe to this callback, use the VideoWorker.addTemplateCreatedCallbackU
method. To unsubscribe from this callback, use the VideoWorker.removeTemplateCreatedCallback
method by providing the callback_id
that you received from the VideoWorker.addTemplateCreatedCallbackU
method.
Recognize Faces
Face tracking requires template creation and matching with the database. Therefore, VideoWorker
must be created with matching_thread>0
and processing_thread>0
using the Video Engine Extended license. To create Video Engine Extended for one stream, specify the parameters streams_count=1
, processing_threads_count=1
, matching_threads_count=1
.
To setup or change the database, use the VideoWorker.setDatabase
member function. This function can be called at any time.
VideoWorker.MatchFoundCallbackU
returns the result of matching with the database. When a template is created for the tracked face, it is compared with each template from the database, and if the distance to the closest element is less than distance_threshold
specified in this element, then a match is fixed. This callback is called after N
consecutive matches with the elements belonging to the same person.
You can set the <not_found_match_found_callback>
tag to 1
to enable this callback after N
sequential not-found hits (i.e. when the closest element is beyond its distance_threshold
.) In this case match_result
of the first element in VideoWorker.MatchFoundCallbackData.search_results
will be at zero distance, and the person_id
and element_id
identifiers will be equal to VideoWorker.MATCH_NOT_FOUND_ID
. The N
number can be set in the configuration file in the <consecutive_match_count_for_match_found_callback>
tag.
It is guaranteed that this callback will be called after at least one Tracking
callback and before a TrackingLost
callback with the same stream_id
and track_id
(track_id = sample->getID()
). To subscribe to this callback, use the VideoWorker.addMatchFoundCallbackU
method. To unsubscribe from this callback, use the VideoWorker.removeMatchFoundCallback
method by providing the callback_id
that you received from the VideoWorker.addMatchFoundCallbackU
method. The maximum number of elements returned in the VideoWorker.MatchFoundCallbackData.search_results
is set in the configuration file in the search_k
tag. This number can be changed by the FacerecService.Config.overrideParameter
object, for example: video_worker_config.overrideParameter("search_k", 3);
.
Estimation of age, gender, and emotions
To estimate age and gender, specify the parameter age_gender_estimation_threads_count > 0
. To estimate emotions, specify the parameter emotions_estimation_threads_count > 0
. The information about age, gender, and emotions is returned in VideoWorker.TrackingCallbackU
. The information about emotions is constantly updated. The information about age and gender is updated only if there is a sample of better quality. By default the estimation of age, gender, and emotions is enabled after you create VideoWorker
.
To disable estimation of age, gender, and emotions on a specified stream, use the following methods:
VideoWorker.disableAgeGenderEstimationOnStream
(age and gender)VideoWorker.disableEmotionsEstimationOnStream
(emotions)
To enable estimation of age, gender, and emotions on a specified stream again, use the following methods:
VideoWorker.enableAgeGenderEstimationOnStream
(age and gender)VideoWorker.enableEmotionsEstimationOnStream
(emotions)
Liveness Estimation
Active Liveness
To enable this type of liveness estimation, set the enable_active_liveness
parameter in the VideoWorker configuration file to 1
. All faces for identification will then be subjected to several checks. The following checks are available (set in the LivenessChecks
structure):
SMILE
: smileBLINK
: blinkTURN_UP
: turn your head upTURN_DOWN
: turn your head downTURN_RIGHT
: turn your head to the rightTURN_LEFT
: turn your head to the leftPERSPECTIVE
: face position check (move your face closer to the camera)
Note: to use the SMILE
check, specify the number of streams in the emotions_estimation_threads_count
parameter of the VideoWorker
object.
Between the checks, the face must be in a neutral position (in front of the camera). The order of the checks can be random (this mode is selected by default), or set during initialization (by creating a list of non-repeating checks). See an example of setting a checklist in the video_recognition_demo
sample (C++/C#/Java/Python).
The check status is returned in the active_liveness_result
attribute of TrackingCallback
. This attribute contains the following fields:
verdict
: status of the current check (theActiveLiveness.Verdict
object)type
: type of check (theActiveLiveness.LivenessChecks
object, see the description above)progress_level
: degree of confidence for check passing - a number in the range of [0,1]
The ActiveLiveness.Verdict
object contains the following fields:
ALL_CHECKS_PASSED
: all checks passedCURRENT_CHECK_PASSED
: current check passedCHECK_FAIL
: check failedWAITING_FACE_ALIGN
: waiting for neutral face positionNOT_COMPUTED
: liveness is not estimatedIN_PROGRESS
: check is in progress
Short Time Identification
Short time identification (STI) is used to recognize a track as a person who has been in front of a camera not long ago. STI is used even if this person is not in the database and even if matching is disabled. For example, if a person is detected, tracked, lost, and then detected and tracked again within, for example, one minute, this person will be considered as the same person.
If short time identification is enabled, VideoWorker
matches the tracks, where a face is lost, with other tracks, where a face was lost not longer than sti_outdate_time
seconds ago. Matched tracks are grouped as sti_person
. ID of this group (sti_person_id
) is returned in VideoWorker.TrackingLostCallbackU
. The value of sti_person_id
is equal to the track_id
value of the first element that formed the group sti_person
. When a specific group sti_person
exceeds the specified period sti_outdate_time
, then VideoWorker.StiPersonOutdatedCallbackU
is called.
If the face from STI group is currently tracked, then VideoWorker.StiPersonOutdatedCallbackU
is not called, and the timer for this group is reset.
Short time identification does not affect the license usage. To use this function, there must be at least one thread for template creation (processing_thread>0
).
Detailed Info about VideoWorker Configuration Parameters
Click here to see the list of parameters from the configuration file that can be changed with the FacerecService.Config.overrideParameter method
max_processed_width
andmax_processed_height
– Limits the size of the image that is submitted to the internal detector of new faces.min_size
andmax_size
– Minimum and maximum face size for detection (the size is defined for the image already downscaled undermax_processed_width
andmax_processed_height
). You can specify relative values for the REFA detector, then the absolute values will be values relative to the image width.min_neighbors
– An integer detector parameter. Note that large values require greater detection confidence. You can change this parameter based on the situation. For example, increase the value if a large number of false detections is observed, and decrease the value if a large number of faces is not detected. Do not change this setting if you aren't sure.min_detection_period
– A real number that means the minimum time (in seconds) between two runs of the internal detector. A zero value means ‘no restrictions’. It is used to reduce the processor load. Large values increase the latency in detecting new faces.max_detection_period
– An integer that means the max time (in frames) between two runs of the internal detector. A zero value means ‘no restrictions’. For example, if you process a video offline, you can set the value to1
so as not to miss a single person.consecutive_match_count_for_match_found_callback
– An integer that means the number of consecutive matches of a track with the same person from the database to consider this match valid (see alsoVideoWorker.MatchFoundCallbackU
).recognition_yaw_min_threshold
,recognition_yaw_max_threshold
,recognition_pitch_min_threshold
andrecognition_pitch_max_threshold
– Real numbers that mean the restrictions on the face orientation to be used for recognition.min_tracking_face_size
– A real number that means the minimum size of a tracked face.max_tracking_face_size
– A real number that means the maximum size of a tracked face, non-positive value removes this limitation.min_template_generation_face_size
– A real number that means the minimum face size for template generation, faces of lower size will be marked asweak = true
.single_match_mode
– an integer,1
means that the single match mode is enabled,0
means that the single match is disabled. If this mode is on, the track that is matched with the person from the database will never generate a template and be matched with the database once again.delayed_samples_in_tracking_callback
– An integer,1
meansenabled
,0
meansdisabled
. The fact is that the internal detector, which detects new faces, does not have time to work on all the frames. Therefore, some samples may be received with a delay. If the value1
is selected, all delayed samples will be transmitted toTrackingCallbackFunc
. Otherwise, the delayed samples appear only if the callback order is violated (i.e. a track sample must appear at least once inTrackingCallback
before callingTrackingLostCallback
for this track). To determine what frame a sample actually belongs to, use theRawSample.getFrameID
method.weak_tracks_in_tracking_callback
– An integer,1
meansenabled
,0
meansdisabled
. By default this flag is disabled and samples with the flag ofweak = true
are not passed to theTracking
callback if at the time of their creation there are no samples with the sametrack_id
(track_id = sample.getID()
) with the flag ofweak=false
. Ifweak_tracks_in_tracking_callback
is enabled, all samples are passed to theTracking
callback, so theTrackingLost
callback can be called withbest_quality_sample = NULL
.search_k
– An integer, which means the maximum number of elements returned to theVideoWorker.MatchFoundCallbackU
callback, i.e. this is thek
parameter, passed internally in theRecognizer.search
method.processing_queue_size_limit
– An integer that means the max count of samples in the queue for template creation. The integer is used to limit the memory consumption in cases, where new faces appear faster than templates are created.matching_queue_size_limit
– An integer that means the max count of templates in the queue for matching with database. The integer is used to limit the memory consumption in cases, where new templates appear faster than they are matched with the database (this can happen in case of a very large database).recognizer_processing_less_memory_consumption
– An integer that will be used as a value of theprocessing_less_memory_consumption
flag in theFacerecService.createRecognizer
method to create an internal recognizer.not_found_match_found_callback
– An integer,1
meansenabled
,0
meansdisabled
, callsVideoWorker.MatchFoundCallbackU
after N consecutive mismatches.depth_data_flag
– An integer value,1
turns on depth frame processing to confirm face liveness (means that it belongs to a real person) during face recognition,0
turns off this mode. All overriden parameters with thedepth_liveness.
name prefix are forwarded to theDepthLivenessEstimator
config with the prefix removed. So to overrideparamname
param for depth liveness, overridedepth_liveness.paramname
forVideoWorker
. SeeFacerecService.Config.overrideParameter
.timestamp_distance_threshold_in_microsecs
– Maximum allowed distance between the timestamps of a color image and a corresponding depth frame in microseconds; used ifdepth_data_flag
is set.max_frames_number_to_synch_depth
– Maximum queue size when synchronizing the depth data; used ifdepth_data_flag
is set.max_frames_queue_size
– Maximum queue size of frames used by the tracker; whendepth_data_flag
is set, the recommended value is max(3, rgbFPS / depthFPS).offline_work_i_e_dont_use_time
– An integer,1
meansenabled
,0
meansdisabled
. Default value isdisabled
. When enabled, the check withmax_detector_confirm_wait_time
is not performed. Alsomax_occlusion_count_wait
will be used istead ofmax_occlusion_time_wait
.max_occlusion_time_wait
– A real number in seconds. When the tracker detects face occlusion, it holds the face position and tries to track it on new frames during this time.max_occlusion_count_wait
– An integer. Means the same asmax_occlusion_time_wait
but in this case time is measured in frames instead of seconds. Used only whenoffline_work_i_e_dont_use_time
is enabled.fda_max_bad_count_wait
– An integer. Whenfda_tracker
detects decline in the face quality, it tries to track that face with the general purpose tracker (instead of the fda method designed and tuned for faces) during at mostfda_max_bad_count_wait
frames.base_angle
– An integer:0
,1
,2
, or3
. Set camera orientation:0
meansstandard
(default),1
means+90 degrees
,2
means-90 degrees
,3
means180 degrees
.fake_detections_cnt
– An integer. Number of start positions to search a face usingvideo_worker_fdatracker_fake_detector.xml
.fake_detections_period
– An integer. Each start position will be used once infake_detections_period frames
.fake_rect_center_xN
,fake_rect_center_yN
,fake_rect_angleN
,fake_rect_sizeN
– Real numbers. Parameters of start positions. N is from0
tofake_detections_cnt – 1
including.fake_rect_center_xN
– x coordinate of a center relative to the image width.fake_rect_center_yN
– y coordinate of a center relative to the image height.fake_rect_angleN
– roll angle in degrees.fake_rect_sizeN
– size relative to max(image width, image height).downscale_rawsamples_to_preferred_size
– An integer,1
meansenabled
,0
meansdisabled
. The default value isenabled
. When enabled,VideoWorker
downscales each sample to the suitable size (seeRawSample.downscaleToPreferredSize
) in order to reduce memory consumption. However, it decreases the performance. It is recommended to disabledownscale_rawsamples_to_preferred_size
and useRawSample.downscaleToPreferredSize
manually forRawSamples
that you need to save or keep in RAM for a long time.squeeze_match_found_callback_groups
– An integer,1
meansenabled
,0
meansdisabled
. Default value isdisabled
. When the track gets the first N consecutive matches with the same person from the database, all N matches will be reported (where N = value ofconsecutive_match_count_for_match_found_callback
), i.e. there will be N consecutiveVideoWorker.MatchFoundCallbackU
calls. But ifsqueeze_match_found_callback_groups
is enabled, only one match will be reported – the one with the minimal distance to the database.debug_log_enabled
– An integer,1
meansenabled
,0
meansdisabled
. Default value isdisabled
. This can also be enabled by setting the environment variableFACE_SDK_VIDEOWORKER_DEBUG_LOG_ENABLED
to1
. If enabled,VideoWorker
logs the result of its work instd::clog
.need_stable_results
– An integer,1
meansenabled
,0
meansdisabled
. Default value isdisabled
. The parameter is used to produce the same results when working with the same data. It disables few optimizations that can produce slightly different results due to unstable order of multithreaded computations. Also it enablesoffline_work_i_e_dont_use_time
andset max_detection_period
value to1
. Also it disables the frame skip (disables the limit set bymax_frames_queue_size
).