Database usage
Loading a face database
To upload the database of face images, use the script from the utils/upload_script folder of OMNI Platform distribution kit. The script accepts a file that contains paths to dataset images, and sends requests to OMNI Platform to create profiles from these images.
The result is a list of profiles in OMNI Platform and a list of files in the script_files folder:
result.txt: a file with a number that indicates the number of images processed the last time the script was run. This number also includes images processed with errors. The number is updated while the script is running.
errors.txt: a file with image upload errors written in the format:
<path to image>:<error>trace_errors.txt: a file with a detailed description of the image upload error in the format:
<path to the image>:<error> <traceback>. You can use this file by yourself when you try to fix an error, or attach it to your email to the technical support team.
System requirements
- Ubuntu 20.04
- Python v3.8
- pip v22.0.4
All subsequent steps are performed from the upload_script folder.
Preparation
Make sure you have a file that contains the paths to the face images. The path to each image must start on a new line without indentation.
For example:
/home/stranger/Downloads/1.jpeg
/home/stranger/Downloads/2.jpeg
/home/stranger/Downloads/3.jpegIf you don't have this file, generate it using a command that will recursively find all jpg or jpeg files in the dataset and display the absolute paths to them in the file:
find <absolute path to the dataset with images> -type f -name "*.jpg" -o -name "*.jpeg" > image_paths.txt
-name <regular expression>specifies the image format, or rather the search condition.-ospecifies a logical OR relationship between two conditions.
Generate a virtual environment to run the image upload script:
python3 -m venv generate_persons_envConnect to the generated environment:
source generate_persons_env/bin/activateInstall the necessary dependencies in the virtual environment:
pip3 install -r requirements.txt
Operation
Launching a scipt
The script is launched only from the virtual environment created at the preparation stage by executing the generate_persosn.py file:
python generate_persons.py
You can start creating profiles from a previously generated list of image paths using the following command:
python generate_persons.py -p ./image_paths.txt -H <platform url> -t <platform token> -P <profile group id>
Required arguments:
-p --paths: path to a file with paths to face images
-H --host: URL of OMNI Platform instance where images are to be uploaded. Make sure the URL is spelled without a slash in the end.
-t --token: user access token/authorization key that grants access to upload images to OMNI Platform. The token is available on the home page of OMNI Platform web interface after authorization.
Optional arguments:
-T --throughput: a parameter determines the maximum number of profile creation requests that the server executes during script running. It is recommended not to change this setting without calculating OMNI Platform performance on your server machine configuration.
-l --limit <limit>: a parameter limits the number of uploaded images if the dataset contains more images than required for uploading.
P --profile_group: uuid4 of profile group where all profiles created from uploaded images should be added to. Adding to a profile group is required to configure notifications for created profiles. uuid4 can be obtained from OMNI Platform web interface by clicking on the five digits in the upper left corner of the Watchlist card.
-c --clear-prev: a parameter ignores the previous results of script running and starts uploading images again by clearing the result.txt file. Note that in this case the errors.txt and trace_errors.txt files will also be cleared. To keep the logs, copy these files to another location.
You can get a full description of the script parameters by running the following command:
python3 generate_persons.py -h
The script is running successfully when new profiles from dataset images appear in OMNI Platform as well as console messages are obtained in the format: Handled <number of processed images> images
Stopping a script
You can stop the script with Ctrl+C. All previous processing will be saved and, if necessary, you will be able to resume the uploading.
Image upload errors
A message about the failed image uploading will be printed to the console, while running the script. The message itself will be saved to the errors.txt file, more detailed error information will be written to the trace_errors.txt file.
- sort: cannot read: ./local_im: No such file or directory
Incorrect path to the file that contains absolute paths to dataset images. Check that the file is located at the given path.
- <class 'aiohttp.client_exceptions.ClientResponseError'>|404, message='Not Found', url=URL('http://sadasdasd/api/v2/')
<class 'aiohttp.client_exceptions.ClientConnectorError'>|Cannot connect to host asdasd:80 ssl:default [Name or service not known]
<class 'aiohttp.client_exceptions.InvalidURL'>|http:///asdasd/api/v2/
Incorrect URL of OMNI Platform instance. Check that OMNI Platform is indeed located at the specified URL.
- <class 'Exception'>|[{'message': 'User has no access or id is incorrect'}]
The user access token is incorrect. Check that you copied the token correctly.
- <class 'Exception'>|[{'message': 'One or several profiles_groups does not exist', 'code': '0x573bkd35'}]
<class 'Exception'>|[{'message': '“%(value)s” is not a valid UUID.'}]
No profile group found by the passed id. Check that you copied the profile group id correctly.
- <class 'Exception'>|[{'message': 'No faces found', 'code': '0x95bg42fd'}]
<class 'Exception'>|[{'message': 'Low quality photo', 'code': '0x86bd49dh'}]
No faces found in the image, or poor image quality. In this case, the profile for this image will not be created in OMNI Platform.
Database backup and restore
Not supported for external databases.
Commands are executed from the setup directory.
Database backup
To back up the database, run the command below:
./cli.sh platform db-backup - <postgres_user> <dump_path>
The <root_postgres_user> name can be found in the ./cfg/platform.secrets.json file in postgres-root-credentials.user
Database restore
Restoring data from a backup must be performed on the same version of the platform from which the backup was created.
To restore the database, run the following commands:
Stop image-api:
./cli.sh image-api uninstallStop matcher:
./cli.sh matcher uninstallRequest a list of services:
kubectl get deploymentStop all services except platform-postgres-dep:
kubectl scale deployment <SERVICE_NAME_1> <SERVICE_NAME_2> --replicas=0Restore data from a backup:
./cli.sh platform db-restore - <dump_path>Run matcher:
./cli.sh matcher installRestart stopped services
kubectl scale deployment <SERVICE_NAME> --replicas <COUNT>or redeploy OMNI Platform:
./cli.sh platform installRun image-api:
./cli.sh image-api install