Overview

The Face Recognition Edge node recognizes faces (by a match to faces provided by LDAP, JSON or CSV config, or a Lumeo Watchlist) within a specified region in live video stream, and adds face metadata to the frame metadata.

To use this node, you specify face images via LDAP, a JSON/CSV configuration, or a Lumeo Watchlist, and configure this node to perform lookup against those registered faces. You can also add metadata to those faces which this node will retrieve and display or color the face with.

This node makes it easy to build common use cases such as access control, customer experience improvements, etc.

This node requires a AI Model Node and a Track Objects Node before it in order to function properly.

Inputs & Outputs

Inputs : 1, Media Format : Raw Video
Outputs : 1, Media Format: Raw Video
Output Metadata : Face Information

Properties

Property	Description	Type	Default	Required
`roi_labels`	Regions of interest labels	`string`	—	No
`rois`	Look up faces only in this region. Conditional on `roi_labels`. Format: comma-separated normalized `x,y` coordinate pairs; separate multiple polygons with semicolons (for example, `0.1,0.1,0.9,0.1,0.9,0.9`).	`string`	`null`	No
`known_face_source`	Source from which to fetch images and metadata of known faces to lookup against. Options: LDAP (`ldap`); JSON (`json`); CSV (`csv`); Watchlist Items (`trackables`).	`enum`	`ldap`	No
`json_source`	Raw JSON or URL from which to fetch a JSON containing metadata and URLs for known faces. See documentation for format. Conditional on `known_face_source` being `json`.	`string`	`null`	Yes
`csv_source`	Raw CSV or URL from which to fetch a CSV containing metadata and URLs for known faces. See documentation for format. Conditional on `known_face_source` being `csv`.	`string`	`null`	Yes
`trackables_lists_ids`	Watchlist IDs that contain the reference faces to load. Conditional on `known_face_source` being `trackables`. Format: comma-separated trackable list UUIDs.	`string`	`null`	Yes
`trackables_items_ids`	Watchlist items IDs from the previously selected watchlists. Leave blank to load all items. Conditional on `known_face_source` being `trackables`. Format: comma-separated trackable item UUIDs.	`string`	`null`	No
`known_faces_refresh_interval`	Seconds between checks of the face source (Watchlist, LDAP, or JSON/CSV URL) for added, updated or removed faces after startup. Set to 0 to only load faces at startup. Unit: seconds.	`number`	`300`	No
`ldap_source_server`	LDAP server host and port. Conditional on `known_face_source` being `ldap`.	`string`	`null`	Yes
`ldap_source_use_ssl`	LDAP use SSL? Conditional on `known_face_source` being `ldap`.	`bool`	`true`	Yes
`ldap_source_user`	LDAP user. Conditional on `known_face_source` being `ldap`.	`string`	`null`	No
`ldap_source_password`	LDAP password. Conditional on `known_face_source` being `ldap`.	`string`	`null`	No
`ldap_cert`	LDAP certificate. Conditional on `known_face_source` being `ldap`.	`string`	`null`	No
`ldap_cert_key`	LDAP certificate Key. Conditional on `known_face_source` being `ldap`.	`string`	`null`	No
`ldap_search_base`	LDAP search base. Conditional on `known_face_source` being `ldap`.	`string`	`null`	Yes
`ldap_attributes`	Extra LDAP attributes. Conditional on `known_face_source` being `ldap`.	`string`	`null`	No
`ldap_photo_attrib`	Attribute name that stores the photo or photo URL in your LDAP configuration. Conditional on `known_face_source` being `ldap`.	`string`	`null`	Yes
`ldap_photo_type`	Format of the LDAP photo attribute. Options: Raw (`raw`); URL (`url`). Conditional on `known_face_source` being `ldap`.	`enum`	`raw`	No
`display_roi`	Display ROI?	`bool`	`true`	No
`display_faceinfo`	Display face info on video? Options: Disabled (`disabled`); Bottom left (`bottom_left`); Bottom right (`bottom_right`); Top left (`top_left`); Top right (`top_right`).	`enum`	`bottom_left`	No
`display_userdata`	Display face metadata?	`bool`	`false`	No
`face_label_attributes`	Comma-separated list of attributes from LDAP or `user_data` JSON/CSV field to generate the face's label.	`string`	`null`	No
`min_confidence`	Minimum confidence level for a face to be considered a match. Lower values are more permissive. Range: minimum `0`, maximum `1`. Step: `0.01`.	`float`	`0.4`	No
`min_face_size_pixels`	Min. width and height of a face to attempt looking up.	`number`	`64`	No
`face_model`	Face model to use. Options: High Accuracy (`Facenet512`): Most accurate, but slow; Balanced (`large`): Balance of accuracy and speed; Fast Performance (`small`): Fastest with some loss in accuracy.	`enum`	`small`	No
`face_lookup_interval`	Minimum time in seconds between successive lookups of the same face. Supports decimals; set 0 to lookup every frame. Unit: seconds.	`float`	`1.0`	No
`max_lookups_per_face`	Max. lookups per face before marking it unrecognized. Unit: lookups.	`number`	`10`	No
`front_face_lookups`	If enabled, improves accuracy by only attempting to recognize faces looking at the camera.	`bool`	`true`	No
`reset_cache`	Reset face cache at startup?	`bool`	`false`	No
`face_match_use_float16`	Store the known-face matching matrix as float16 instead of float32, halving its memory for very large face lists on memory-constrained devices, with negligible accuracy impact. Matching may be slower in float16, so enable only when memory, not match latency, is the constraint.	`bool`	`false`	No
`debug`	Log debugging information?	`bool`	`false`	No

Face Label Attributes

When used with a Watchlist, the following attributes are available to use to set face_label_attributes field: trackable_name, trackable_labels

Output Metadata

The fields below are declared by this node's metadata schema; the JSON values are representative examples.

Path	Type	Description
`nodes.<node_id>.rois.<roi_label>.recognized_face_count`	`integer`	Number of recognized faces in this frame
`nodes.<node_id>.rois.<roi_label>.recognized_face_delta`	`integer`	Number of newly recognized faces in this frame
`nodes.<node_id>.rois.<roi_label>.unrecognized_face_count`	`integer`	Number of unrecognized faces in this frame
`nodes.<node_id>.rois.<roi_label>.unrecognized_face_delta`	`integer`	Number of newly unrecognized faces in this frame
`nodes.<node_id>.rois.<roi_label>.coords`	`array`	Array of pixel coordinates defining the configured ROI or line.
`nodes.<node_id>.rois.<roi_label>.recognized_faces[].id`	`integer`	Tracked face object ID.
`nodes.<node_id>.rois.<roi_label>.recognized_faces[].person_id`	`string`	Identifier of the matched person.
`nodes.<node_id>.rois.<roi_label>.recognized_faces[].confidence`	`number`	Confidence of the face match.
`nodes.<node_id>.rois.<roi_label>.recognized_faces[].user_data`	`any JSON value`	JSON-compatible value containing user data stored with the matched identity.
`nodes.<node_id>.rois.<roi_label>.recognized_faces[].label`	`string`	Display label of the matched identity.
`nodes.<node_id>.rois.<roi_label>.recognized_faces[].description`	`string`	Human-readable summary of the face match.
`nodes.<node_id>.rois.<roi_label>.unrecognized_face_ids`	`array of strings`	Tracking IDs for unrecognized Faces.
`nodes.<node_id>.rois.<roi_label>.objects_of_interest_keys`	`array`	Array of metadata keys that contain object IDs relevant to downstream integrations.
`nodes.<node_id>.rois.<roi_label>.description`	`string`	Human-readable summary of the node’s current result.
`nodes.<node_id>.rois.<roi_label>.recognized_faces[].user_data.example`	`string`	Value of example.
`nodes.<node_id>.type`	`string`	Identifies the node type that produced this metadata.

JSON example

{
  "nodes": {
    "face_rec1": {
      "type": "face_rec",
      "rois": {
        "face_area": {
          "coords": [
            [0, 0],
            [960, 0],
            [1920, 0],
            [1920, 540],
            [1920, 1080],
            [960, 1080],
            [0, 1080],
            [0, 540],
            [0, 0]
          ],
          "recognized_faces": [
            {
              "id": 6.9322489671912e+18,
              "person_id": "019ef1b5-18f7-7f13-9157-cb2089c2496f:019ef1b5-19bf-7fa3-a72c-dcefb5aef13a",
              "confidence": 0.441994607448578,
              "user_data": {
                "trackable_id": "019ef1b5-18f7-7f13-9157-cb2089c2496f",
                "trackable_list_id": "019eb516-8c99-7be2-8d89-fd36839cb2fe",
                "trackable_name": "E1",
                "trackable_description": "",
                "trackable_labels": "employee"
              },
              "label": "E1 employee",
              "description": "E1 employee 44%"
            }
          ],
          "unrecognized_face_ids": [],
          "recognized_face_count": 1,
          "recognized_face_delta": 1,
          "unrecognized_face_count": 0,
          "unrecognized_face_delta": 0,
          "objects_of_interest_keys": [
            "recognized_faces"
          ],
          "description": "face_area : recognizedFaces(1) unrecognizedFaces(0)"
        }
      }
    }
  }
}

Additional Info

When used with a Lumeo Watchlist, the userdata field contains the following elements:

trackable_id : Watchlist Item ID
trackable_list_id : Watchlist ID
trackable_name : Watchlist Item name
trackable_description : Watchlist Item description
trackable_labels : Comma separated Watchlist Item labels

Object labels and attributes

Object labels/classes added: None.
Object attribute labels/classes added: Recognized person labels and unknown use class 10200. Configured face metadata values use sequential classes beginning at 10201; trackable item and list identifiers use 10403 and 10402; yaw values, when emitted, use 10204.

    "objects": [
      {
        "id": 6932248967191218,
        "source_node_id": "model_inference1",
        "model_id": "8d5d001f-b8b2-49f8-a43f-e3f63f36e095",
        "label": "face",
        "class_id": 0,
        "rect": {
          "left": 170,
          "top": 311,
          "width": 114,
          "height": 162
        },
        "probability": 0.760648369789124,
        "attributes": [
          {
            "label": "E1 employee",
            "class_id": 10200,
            "probability": 0.441994607448578
          },
          {
            "label": "E1",
            "class_id": 10201,
            "probability": 1
          },
          {
            "label": "employee",
            "class_id": 10202,
            "probability": 1
          },
          {
            "label": "019ef1b5-18f7-7f13-9157-cb2089c2496f",
            "class_id": 10403,
            "probability": 1
          },
          {
            "label": "019eb516-8c99-7be2-8d89-fd36839cb2fe",
            "class_id": 10402,
            "probability": 1
          }
        ],
        "corr_id": "e91a84ae-42d3-4708-96ec-993d4720e9e8"
      }
    ],

JSON or CSV Format

When using known_face_source set to json or csv, the specified configuration must follow the format below.

Fields

Field	Requirement	Description
`person_id`	Mandatory	Unique Identifier for a face.
`image_url`	Mandatory	Must point to a Face image and be accessible from the internet. ex. https://assets.lumeo.com/media/demos/facerec/8672.jpg
`label`	Mandatory	Label to be displayed on the video for the face.
`user_data`		You can associate additional attributes along with each registered face/person (aka `user_data`). Lumeo can extract and display these attributes for matched faces. This node will use the following `user_data` attributes, if present. These (and any other) `user_data` properties will be added to `recognized_faces.user_data` field within Lumeo metadata.
`user_data.modifyTimestamp`	Optional	Node will use this to cache face signatures to speed up subsequent startup times if this value has not changed.

`user_data.color`	Optional	Node will display the recognized face with a bounding box in this color.
`user_data.aux_label`	Optional	Node will display this label along with the recognized face on the video
`user_data.<your_attribute_label>`	Optional

JSON

[
    {
        "image_url": "https://assets.lumeo.com/media/demos/facerec/8672.jpg",
        "person_id": "person1",
        "label": "person_name",
        "user_data": {
            "modifyTimestamp": "2022-01-02T12:12:12Z",
            "color": "00ffff",
            "aux_label": "FastPass Valid",
            "attrib1": 12345,
            "attrib2": "test"
        }
    },
    {
        "image_url": "https://assets.lumeo.com/media/demos/facerec/8674.jpg",
        "person_id": "person2",
        "label": "person_name2",
        "user_data": {
            "modifyTimestamp": "2022-01-02T12:12:12Z",
            "color": "00ffff",
            "aux_label": "FastPass Invalid",
            "attrib1": 12345,
            "attrib2": "test"
        }
    },
    {
       ...
    }
]

CSV

person_id,label,image_url,user_data.modifyTimestamp,user_data.aux_label,user_data.color,user_data.add_your_attributes_here
person1,person name,https://assets.lumeo.com/media/demos/facerec/8674.jpg,1/14/2023 8:53:22,access denied,0xff0000,testing

Google Sheets (CSV)

CSV template : https://docs.google.com/spreadsheets/d/1u4t1Fg3nMWQjP902xyLKk-H2ASw_DLNCr4nZ3tX8NAU/edit#gid=0

A neat trick is to enable Publish to Web feature in Google sheets which generates a live updating CSV link that you can configure in this node's settings. This lets you manage your face database in Google Sheets and have Lumeo automatically pull from it every time a deployment is started.

Here's how :

Go to File -> Share -> Publish to Web in Google Sheets menu

Go to File -> Share -> Publish to Web in Google Sheets menu

Select Link, Sheet and CSV options. Also enable "Automatically republish ... " checkbox.

Copy the resulting link into this node's properties.