Scheduled data cleanup configuration

This section describes the principles of the planned cleanup system and its proper configuration at the service level.

The cleanup system is responsible for deleting outdated data and allows you to configure the cleanup schedule using cron expressions.

The management point is the baf-dashboard service, while the actual data deletion is performed by the platform-backend and video-recorder services.

platform-backend

The service is responsible for deleting outdated authorization attempt photos. Registration attempt photos are only deleted when the applicant is deleted.

All cleanup configuration parameters are located in the platform.values.yaml file.

Cleanup schedule and parameters

The cleanup procedure runs on a schedule configured via the following variables:

• backend.planned_retention_policy_day_of_week - day of the week (cron)
• backend.planned_retention_policy_hour - hour (cron)
• backend.planned_retention_policy_minute - minute (cron)

In standard settings, the period is set to the 59th minute of every hour, meaning cleanup runs hourly.

Data deletion is performed in batches to distribute the load:

• backend.delete_sample_batch_size - number of frames deleted per operation
• backend.delete_sample_batch_sleep - pause in seconds between deletion operations

The number of parallel cleanup processes is configured via backend.retention_process_count. Increasing the number of processes allows parallel cleanup of data from different accounts, while data from one account is always processed by a single process.

note

Do not set delete_sample_batch_sleep to more than 5 seconds - this directly affects transaction duration in the system. To reduce load, it's better to decrease the batch size.

Backward compatibility mode

When starting the service on a clean database or on BAF version 1.14.0 and higher, the procedure operates in backward compatibility mode. In this mode, photos from attempts created earlier than backend.sample_ttl from the procedure's start time are deleted.

The procedure transitions to the new operation mode after someone visits the cleanup settings page in the dashboard.

note

The mode operates individually for each account - transitioning to the new mode in one account does not affect other accounts.

Planned cleanup mode

After transitioning to planned cleanup mode, the procedure checks whether cleanup should be performed at the current moment. If the scheduled time has not arrived or cleanup is disabled, the procedure completes without taking action. If the conditions are met, data cleanup is launched from the planned date, calculated from the cleanup settings.

In standard settings, the physical start time for data deletion may be shifted by up to 1 hour, as the procedure runs hourly.

video-recorder

The service is responsible for deleting outdated video data from attempts.

All cleanup configuration parameters are located in the video-recorder.values.yaml file.

Cleanup schedule and parameters

The procedure runs similarly to the platform-backend service, but the check frequency is configured via the variable:

• video_recorder.retention_check_frequency - frequency of cleanup check procedure execution in seconds (maximum period - 1 hour)

In standard settings, the frequency is set to 1200 seconds, meaning cleanup runs every 20 minutes.

Data deletion is performed in batches to distribute the load:

• video_recorder.retention_batch_size - number of records deleted per operation
• video_recorder.retention_batch_pause - pause in seconds between deletion operations

The number of parallel cleanup processes is configured via video_recorder.retention_process_count. Increasing the number of processes speeds up data cleanup for different accounts, while data from one account is always processed by a single process.

note

Do not set video_recorder.retention_batch_pause to more than 5 seconds - this directly affects transaction duration in the system. To reduce load, it's better to decrease the batch size.

Working with MinIO features

S3 storage does not delete data immediately, but moves it to the .trash folder, which is cleaned up every minio.delete_cleanup_interval seconds.

It's important that the volume of data deleted by video-recorder during this period does not cause an I/O storm. When changing the batch size or pause duration, it's necessary to adjust minio.delete_cleanup_interval accordingly.

Backward compatibility mode

The service does not have a backward compatibility mode. When upgrading BAF from version 1.14.0 and higher, data cleanup via MinIO is disabled. Until planned cleanup is enabled, data is stored indefinitely.

note

Erroneous content is still deleted via MinIO - data older than minio.error_data_retention_days is automatically deleted.

Planned cleanup mode

The logic of planned cleanup operation is similar to the platform-backend service. After transitioning to this mode, the procedure checks whether cleanup should be performed. If the scheduled time has not arrived or cleanup is disabled, the procedure completes without taking action. If the conditions are met, data cleanup is launched from the calculated planned date.

In standard settings, the physical start time for data deletion may be shifted by up to 20 minutes, as the procedure runs every 20 minutes.