AI Model Node

Run an AI Model on the video stream

Overview

Use the Model inference node to run an AI model on the video stream. The node adds inference outputs (object bounding boxes, labels, probabilities) and raw tensor outputs from the Model to the metadata.

Use the Overlay Node after the Inference Node to display the output on the video stream, and/or Function Node if you need to process the output and use the metadata in your app.

Labels for off-the-shelf models can be found under Model Labels

Detection Models

Detection Models identify objects in a stream, and provide you with the object label along with the coordinates for the object's bounding box. Detection Models can be run on an entire frame or on the output of another Detection Model.

Examples of Detection Models : Face detector, Traffic Detector

Classifier Models

Classifier Models (Such as a Mask Classifier, Vehicle Type Classifier, etc.) operate on the output of a Detector Model and identify the type of the object detected by the Detector.

In order to use a Classifer Model, add a 2nd Model Inference Node after the one that runs the Detector Model, and use the infer_on_node property to configure the 2nd node to operate on the output of the 1st one.

Inputs & Outputs

  • Inputs : 1, Media Format : Raw or Multiplexed Video
  • Outputs : Raw or Multiplexed Video (same as Input format)
  • Output Metadata : Objects, Bounding boxes, classes and Raw tensors.

Properties

Property

Value

model

The Marketplace or BYO model to use for inference.

See AI Models for details on how to upload your own model.

infer_on_node

Select another model inference node from this pipeline to infer only on objects previously detected by that node.

Typically used for Classifier models that classify the objects detected by a Detector model.

Leave this empty if you want the model to run on the entire frame.

inference_interval

Frequency, in Frames, with which the model should run.

1 Frame means every frame, 2 means every other frame, etc. Use this property to tune how many streams you can process on your device.

Metadata

Access this metadata using the Function Node or using the API, from the snapshot or clip saved downstream of this node.

See Function examples here : https://docs.lumeo.com/docs/run-custom-function-node#accessing-metadata

Metadata Property

Description

objects

List of Objects detected by the Model inference nodes. See table below for the format of this property.

'objects': [{
    'id': 5750484150146564100,
    'label': 'car',
    'class_id': 0,
    'probability': -0.10000000149011612,
    'rect': {
        'width': 47.60007858276367,
        'top': 201.2154541015625,
        'left': 656.13134765625,
        'height': 25.11772346496582
    },
    'attributes': [{
        'label': 'white',
        'probability': 0.9437278509140015,
        'class_id': 10
    }]
}]

Object Properties

Key

Type

Description

id

Long Int, Optional

If present, a unique ID assigned to this object by a Track Objects Node

label

String

Label for this object as outlined in Model Labels

class_id

Int

Unique id for this object's label.

probability

Float

Probability of detection

rect

Dictionary

Bounding box for this object. Contains:
left
top
width
height

attributes

Dictionary

Object's classes, as identified by Classifier models chained to this Object Detection model.

Contains:
label : Class label
probability : Probability
class_id : Unique id for the class' label


Did this page help you?