Benchmarks
Modes:
- Create a Configuration File
- Detection
- Processing
- 1:1 Recognition Test
- 1:N Recognition Test
- Search Speed Test
Create a Dataset Configuration File
- Dataset Configuration File is a text file with 3 lines per each image (
<person_id> <image_id> <path_to_image>
).
Example:
person0_id
image0_id
path_to_image0
person0_id
image1_id
path_to_image1
...
Example of creating an image dataset configuration file
Collect a list of image files
find -type f | sort > ../lfw_simple_format.txt
lfw_simple_format.txt
is a text file with the lines in the following format:<path/to/person/dir/image_file>
. Images with the samepath/to/person/dir
belong to the same person. Images for one person must be in a row.
Example:
person1_dir/image1
person1_dir/image2
person2_dir/image1
...
The paths in the configuration files must be relative to the directory passed through the dataset_root_dir
parameter.
Convert a list to a Configuration File
In this mode the program converts the config file from list format
to dataset format
(used in other modes):
Launch parameters:
mode
– program mode (convert_config_format)result_dataset_config
– config file of the image datasetFILE
– one file that contains a list of image files
Example of launch:
./test_sdk \
--mode convert_config_format \
--result_dataset_config dataset_config.txt \
lfw_simple_format.txt
The conversion result is a config file of the image dataset.
Detection
In this mode the program detects faces in images with the id in the range of [begin_image_id, end_image_id)
. Each image should contain only one person. The images, containing more than one person detected, will be discarded.
Note: when using a GPU, it is recommended to run no more than one of any test at the same time.
Before starting the program, you need to get the database file with images (see Create a Configuration File)
Launch parameters:
mode
– program mode (detection
)dll_path
– path to thelibfacerec.so
orfacerec.dll
library filesdk_config_dir
– path to the conf/facerec directorydataset_config
– config file of the image dataset (see Create a Configuration File)capturer_config
– capturer config file namedataset_root_dir
– path to the dataset directorydetection_result_file
– file for storing the detection results[begin_image_id]
– index of the image, from which the detection begins (the default setting is 0)[end_image_id]
– index of the image, to which detection is performed (by default, the processing is performed to the end of the file)[use_cpu_cores_count]
– number of cores used for detection (the default setting is 1)
Example of launch from the bin directory:
./test_sdk \
--mode detection \
--dll_path ../lib/libfacerec.so \
--sdk_config_dir ../conf/facerec \
--capturer_config common_capturer4_lbf.xml \
--dataset_config dataset_config.txt \
--dataset_root_dir /path/to/data \
--detection_result_file \
detection_result.bin
The detection result is a text file with the lines in the following format: <image_id> <points_count> <points>
Processing
In this mode the program creates templates from the faces detected in the images with the id in the range of [begin_image_id, end_image_id)
.
Before starting the program, you need to get the detections file for the recognizer under test (see Detection)
Launch parameters:
mode
– program mode (processing
)dll_path
– path to thelibfacerec.so
orfacerec.dll
library filesdk_config_dir
– path to the conf/facerec directorydataset_config
– config file of the image dataset (see Create a Configuration File)dataset_root_dir
– path to the dataset directoryrecognizer_config
– recognizer config file nameprocessing_result_file
– file for storing the resulting templates[begin_image_id]
– index of the image, from which processing starts (the default setting is 0)[end_image_id]
– index of the image, to which processing is performed (by default, the processing is performed at the end of the file)[use_cpu_cores_count]
– number of cores used for processing (the default setting is 1)FILES
– file(s) with the detection results
Note: a trial license allows to run the test only when [use_cpu_cores_count]
is set to 1
.
Example of launch from the bin directory:
./test_sdk \
--mode processing \
--dll_path ../lib/libfacerec.so \
--sdk_config_dir ../conf/facerec \
--dataset_config dataset_config.txt \
--dataset_root_dir /path/to/data \
--recognizer_config method6v7_recognizer.xml \
--processing_result_file ./templates_6v7.bin \
detection_result.bin
The processing result is a binary file, which contains one record per each generated template. Each record is a 64-bit unsigned integer (image_id
) followed by a serialized template.
1:1 Recognition Test
In this mode the program performs the 1:1 recognition test using the templates generated from the images with the id
in the range of [begin_image_id, end_image_id)
.
Before running the test, it is necessary to calculate the template file for the recognizer under test (see Processing)
Launch parameters:
mode
– program mode (recognition_test_11
)dll_path
– path to thelibfacerec.so
orfacerec.dll
library filesdk_config_dir
– path to the conf/facerec directorydataset_config
– config file of the image dataset (see Create a Configuration File)recognizer_config
– recognizer config file nameresult_roc_file
– file to save the ROC curveresult_closest_mismatches_file
– result file with the closest mismatches[begin_image_id]
– index of the first image used in the test (the default setting is 0)[end_image_id]
– index of the first image afterbegin_image_id
not used in the test (by default use all frombegin_image_id
to the end of the file)[use_cpu_cores_count]
– number of cores used for matching (the default setting is 1)FILES
– file(s) with the processing results
Note: a trial license allows to run the test only when [use_cpu_cores_count]
is set to 1
.
Example of launch from the bin directory:
./test_sdk \
--mode recognition_test_11 \
--dll_path ../lib/libfacerec.so \
--sdk_config_dir ../conf/facerec \
--dataset_config dataset_config.txt \
--recognizer_config method6v7_recognizer.xml \
--result_roc_file ./roc11_6v7.txt \
--result_closest_mismatches_file ./closest_mismatches_file.txt \
templates_6v7.bin
1:1 Recognition of test results:
- A text file, which contains a ROC curve; the file line describes the curve point in the following format:
<far> <tar> <distance>
. - A text file with the lines in the following format:
<distance> <image_id1> <image_id2> <path_to_image1> <path_to_image2>
. This file contains pairs of images marked as belonging to different people, but having a minimum distance between the templates. We recommend you to generate this file using the best method and view the images from the top of it to check the errors of the dataset annotation.
To draw a ROC curve, you need to run the draw_roc_curves utility (see Building a ROC curve)
1:N Recognition Test
In this mode the program performs the 1:N recognition test using the templates generated from the images with the id
in the range of [begin_image_id, end_image_id)
.
Before running the test, it is necessary to calculate the template file for the recognizer under test (see Processing)
Launch parameters:
mode
– program mode (recognition_test_1N
)dll_path
– path to thelibfacerec.so
orfacerec.dll
library filesdk_config_dir
– path to the conf/facerec directorydataset_config
– config file of the image dataset (see Create a Configuration File)recognizer_config
– recognizer config file nameresult_roc_file
– file to save the ROC curve.[begin_image_id]
– index of the first image used in the test (the default setting is 0)[end_image_id]
– index of the first image afterbegin_image_id
not used in the test (by default use all frombegin_image_id
to the end of the file)[use_cpu_cores_count]
– number of cores used for searching (the default setting is 1)[acceleration]
– search acceleration type (the default setting is 0)0
– search withpbio::Recognizer::search
with thepbio::Recognizer::SearchAccelerationType::NO_SEARCH_ACCELERATION
acceleration1
– search withpbio::Recognizer::search
with thepbio::Recognizer::SearchAccelerationType::SEARCH_ACCELERATION_1
acceleration-1
– search withpbio::Recognizer::verifyMatch
in a single thread (independent ofsearch_threads_count
)FILES
– file(s) with the processing results
Note: a trial license allows to run the test only when [use_cpu_cores_count]
is set to 1
. In addition, it may be needed to specify value for the [end_image_id]
parameter so that "gallery_templates size
" does not exceed 1000 (printed to the console after running the test).
Example of launch from the bin directory:
./test_sdk \
--mode recognition_test_1N \
--dll_path ../lib/libfacerec.so \
--sdk_config_dir ../conf/facerec \
--dataset_config dataset_config.txt \
--recognizer_config method6v7_recognizer.xml \
--result_roc_file ./roc1N_6v7.txt \
--acceleration 1 \
templates_6v7.bin
The result of the 1:N recognition test is a text file, which contains a ROC curve. The file line describes the curve point in the following format: <far> <tar> <distance>
.
To draw a ROC curve, you need to run the draw_roc_curves utility (see Building a ROC curve)
Search Speed Test
In this mode the program performs the search speed test using the templates generated in processing mode or using the utility (see Template Generator).
Launch parameters:
mode
– program mode (search_speed_test
).dll_path
– path to thelibfacerec.so
orfacerec.dll
library filesdk_config_dir
– path to the conf/facerec directoryrecognizer_config
– recognizer config file name[templates_count]
– number of templates used (by default, processing is performed until the end of the file)[queries_count]
– number of queries (the default setting is 1)[query_k_nearest]
– number of the nearest templates for searching (the default setting is 1)[search_threads_count]
– number of threads used for searching (the default setting is 1)[acceleration]
– search acceleration type (the default setting is 0)0
– search withpbio::Recognizer::search
with thepbio::Recognizer::SearchAccelerationType::NO_SEARCH_ACCELERATION
acceleration1
– search withpbio::Recognizer::search
with thepbio::Recognizer::SearchAccelerationType::SEARCH_ACCELERATION_1
acceleration-1
– search withpbio::Recognizer::verifyMatch
in a single thread (independent ofsearch_threads_count
)FILES
– file(s) obtained from the processing mode or by using the utility
Note: if a trial license is used, the value of [templates_count]
cannot exceed 1000.
Example of launch from the bin directory:
./test_sdk \
--mode search_speed_test \
--dll_path ../lib/libfacerec.so \
--sdk_config_dir ../conf/facerec \
--recognizer_config method6v7_recognizer.xml \
templates_6v7.bin
The result of the search speed test is a message with test results.
Utilities
Template Generator
A utility for creating random templates.
Launch parameters:
recognizer_version
– recognizer version (select one of [11v1000, 10v30, 10v100, 10v1000, 9v30, 9v300, 9v1000, 8v7, 8v6, 7v7, 7v6, 7v3, 7v2, 7, 6v7, 6v6, 6v5, 6v4, 6v3, 6v2, 6])templates_count
– number of generated templatesresult_file
– a result binary file, which contains the generated random templates
Example of launch:
./template_generator \
6v7 \
100000 \
random_templates.bin
The template generator result is a binary file, which contains the generated random templates of the same format as in the processing mode.
Source code: examples/cpp/test_sdk
ROC-curve construction
A utility for constructing a ROC curve from a complete set of points.
- Python 3 required to run the app
- Before running the program, you need to install the matplotlib package:
pip3 install matplotlib
Launch parameters:
rocs_folder
- path to the folder where the files with the calculated set of points for the ROC curve are storedroc_points_files
– names of files for which it is necessary to build a curve (it is possible to specify several files). The file name will be displayed as the name of the ROC curve on the graph
Run example:
python3 ./draw_roc_curves.py\
./base_roc \
roc11_11v1000.txt \
roc11_10v100.txt \
roc11_10v30.txt \
roc11_9v300.txt
Source code: examples/python/draw_roc_curves.py