Processes and events

OMNI Agent converts the recognition and tracking data into the formats of processes and events for subsequent sending to OMNI Platform or third-party services.

Processes

A data format that represents recognition and tracking results within time intervals.

Each process contains the following set of data:

• Process ID
• Process time interval (start and end of the process)
• Process type (track, human, face, body, emotion etc.)
• Detection object with face attributes (object is a human in the camera’s field of view)
• Best shot

Process specification

Process structure

Description of process structure fields

General structure

{
  "processes": [ { ... } ]
}

processes is a list of all processes recorded by the system. Each process is described according to one of the schemas in "$defs/process".

Basic definitions

process_base

• id – process UUID (unique identifier).

• type – defines the type of process (track, emotion, action, etc.).

• time_interval – time interval of the process (start and end in ISO8601).

• finalized – indicates whether the process is completed.

object_base

• object – the source (object) for which the process is created:

  • id – object UUID.

  • class – object type: "human", "face", "body".

child_process_base

• parent – UUID of the parent process to which the current one belongs (for example, a face process is linked to a human process).

numerical_base

• numerical_id – internal numerical process identifier.

object_source_base

• object.source – source that created the object (e.g., camera).

process_source_base

• source – unique identifier of the camera observing the process.

crop_base_base64 and crop_base_binary

• best_shot – saved frame (best image of the object), which may be:

  • encoded in Base64 (crop_base_base64),

  • binary with metadata (crop_base_binary).

__bsm (Binary Sample with Meta)

• blob – binary data (base64).

• format – storage format (e.g., NDARRAY).

• dtype – data type (uint8_t, float, etc.).

• shape – array dimensions.

• color_format – color space (RGB, BGR, GRAY, etc.).

Specific processes

human_process: Human tracking process.

Includes:

• reference to source (source),

• basic tracking attributes (process_base_track),

• object description (object_base),

• optionally – match_data:

  • profileGroups – profile groups where the person was found.

  • distance, fr_r, fa_r, score – recognition metrics.

face_process: Face tracking process.

object:

• class = "face",

• age – age,

• gender – gender (MALE, FEMALE, UNKNOWN),

• quality – detection quality,

• May include best_shot (face frame).

body_process: Body tracking process.

object:

• class = "body",

• bbox – body bounding box coordinates (x, y),

• score – probability of correct detection.

emotion_process: Emotion detection based on face.

• emotion – string with the emotion name.

• type = "emotion".

attention_process: Attention detection (e.g., whether a person is looking at the screen).

• type = "attention".

joint_process: Skeleton keypoint tracking.

object:

• class = "joint",

• joint – keypoint name (e.g., "nose", "left_wrist"),

• bbox – coordinates,

• score – confidence metric.

presence_process: Presence detection in a region of interest (ROI).

• object.class = "roi",

• object.name – region name,

• action = "roi_presence".

trigger_crossing_process: Line-crossing event.

• object.class = "trigger",

• object.name – line name,

• action = "trigger_crossing",

• direction – direction ("in" or "out").

action_recognition_process: Action recognition process.

• action – action name (e.g., "fall").

external_process: External process (e.g., video playback).

• object.class = "media",

• name – video name,

• type = "external",

• repeats – whether it repeats,

• duration – duration,

• internal_time_interval – internal interval.

Auxiliary fields

• __uuid – UUID (unique identifier).

• timestamp – time in ISO8601.

• positive_integer – positive number.

• __dtype – data type (for binary arrays).

Events

A data format that represents recognition and tracking results as facts. OMNI Agent can generate the following types of events:

• Face events:
  • Identification events (a person from the camera is found in the database).
  • Non-identification events (a person from the camera is not found in the database).
  • Face hide control events (a person is within a Region of Interest (ROI) or crosses a marked virtual line, while hiding their face to avoid recognition).
• HAR (Human Action Recognition) events:
  • Falling
  • Fighting
  • Lying
  • Sitting
• Regions events:
  • ROI events (a person is within or exits a Region of Interest).
  • Line crossing events (a person crosses a marked virtual line in the video stream in either direction).

Events are sent to OMNI Platform via HTTP protocol. Then, you can configure the transmission of these events from OMNI Platform to external services through websockets.

Important

Events are available only for OMNI Agent Online.

Event specification

Event structure

Description of event structure fields

Structure of Identification Event:

• type – identification
• date – event time
• id – event identifier
• object (the object that generated the event):
  • class – object class (for example, human)
  • id – object identifier
• parents (event parent, usually a process):
  • id – event parent identifier
  • type – type of the event parent (for example, human, face)
• source – video stream identifier
• trigger_source:
  • id
• image – cropped image
• image_bbox — normalized detection coordinates.
  • [0] - x1 coordinate
  • [1] - y1 coordinate
  • [2] - x2 coordinate
  • [3] - у2 coordinate
• original_image – full frame (optinally)
• identification_data – array of identification candidates (profiles):
  • profile_id – profile identifier
  • group_ids – identifier of the group to which the profile was added
  • score – degree of similarity of the face from the frame with the face from the profile from 0 (0%) to 1 (100%)
  • far – False Acceptance Rate when the system mistakes images of different people as images of the same person
  • frr – False Rejection Rate when the system mistakes two images of the same person as images of different people
  • distance – distance between compared template vectors. The smaller the value, the higher the confidence in correct recognition

Structure of Non-Identification Event:

• type – non_identification
• date – event time
• id – event identifier
• object (the object that generated the event):
  • class – object class (for example, human)
  • id – object identifier
• parents (event parent, usually a process):
  • id – event parent identifier
  • type – type of the event parent (for example, human, face)
• source – video stream identifier
• trigger_source:
  • id
• image – cropped image
• image_bbox — normalized detection coordinates.
  • [0] - x1 coordinate
  • [1] - y1 coordinate
  • [2] - x2 coordinate
  • [3] - у2 coordinate
• original_image – full frame (optinally)

Structure of ROI Event:

• type – roi
• date – event time
• id – event identifier
• parents (event parent, usually a process):
  • id – event parent identifier
  • type – type of the event parent (for example, human, body)
• source – video stream identifier
• trigger_source:
  • id
• image – cropped image
• image_bbox — normalized detection coordinates.
  • [0] - x1 coordinate
  • [1] - y1 coordinate
  • [2] - x2 coordinate
  • [3] - у2 coordinate
• original_image – full frame (optinally)
• roi_data:
  • direction (a person is in/out of ROI) – in/out

Structure of Line Crossing Event:

• type – crossing
• date – event time
• id – event identifier
• parents (event parent, usually a process):
  • id – event parent identifier
  • type – type of the event parent (for example, human, body)
• source – video stream identifier
• trigger_source:
  • id
• image – cropped image
• image_bbox — normalized detection coordinates.
  • [0] - x1 coordinate
  • [1] - y1 coordinate
  • [2] - x2 coordinate
  • [3] - у2 coordinate
• original_image – full frame (optinally)
• crossing_data:
  • direction (a person crossed the line the forward/reverse direction) – in/out

Structure of HAR Event:

• type – har
• date – event time
• id – event identifier
• parents (event parent, usually a process):
  • id – event parent identifier
  • type – type of the event parent (for example, human, body)
• source – video stream identifier
• trigger_source:
  • id
• image – cropped image
• image_bbox — normalized detection coordinates.
  • [0] - x1 coordinate
  • [1] - y1 coordinate
  • [2] - x2 coordinate
  • [3] - у2 coordinate
• original_image – full frame (optinally)
• har_data:
  • action – fight, fall, sit, lie

Structure of Face Hide Control Event:

• type (event type) — face_hide_control.
• date — event time.
• face_hide_control_data:
  • group_ids — identifier of the group to which the profile is added.
  • match_event_id — identifier of the identification event.
  • no_suitable_face — flag indicating whether a face that needs to be found in the database was detected.
  • profile_id — identifier of the profile from the database.
  • status — status of the control check.
• id — event identifier.
• image — detection crop.
• image_bbox — normalized coordinates of the detection.
  • [0] - x1 coordinate
  • [1] - y1 coordinate
  • [2] - x2 coordinate
  • [3] - у2 coordinate
• parents (parent event, usually a process):
  • id — identifier of the parent event.
  • type — type of the parent event (e.g., human or body).
• source — video stream identifier.
• trigger_source (trigger data that initiated the event, e.g., added line crossing or ROI):
  • id — trigger identifier.
  • name — trigger name.