Overview

Use the Model inference node to run an AI model on the video stream. The node adds inference outputs (object bounding boxes, labels, probabilities) and raw tensor outputs from the Model to the metadata.

Use the Overlay Node after the Inference Node to display the output on the video stream, and/or Function Node if you need to process the output and use the metadata in your app.

Labels for off-the-shelf models can be found under Model Labels

Detection Models

Detection Models identify objects in a stream, and provide you with the object label along with the coordinates for the object's bounding box. Detection Models can be run on an entire frame or on the output of another Detection Model.

Examples of Detection Models : Face detector, Traffic Detector

Classifier Models

Classifier Models (Such as a Mask Classifier, Vehicle Type Classifier, etc.) operate on the output of a Detector Model and identify the type of the object detected by the Detector.

In order to use a Classifer Model, add a 2nd Model Inference Node after the one that runs the Detector Model, and use the infer_on_node property to configure the 2nd node to operate on the output of the 1st one.

Inputs & Outputs

Inputs : 1, Media Format : Raw or Multiplexed Video
Outputs : Raw or Multiplexed Video (same as Input format)
Output Metadata : Objects, Bounding boxes, classes and Raw tensors.

Properties

Property	Value
`model`	The Marketplace or BYO model to use for inference. See AI Models for details on how to upload your own model.
`infer_on_node`	Select another model inference node from this pipeline to infer only on objects previously detected by that node. Typically used for Classifier models that classify the objects detected by a Detector model. Leave this empty if you want the model to run on the entire frame.
`inference_interval`	Frequency, in Frames, with which the model should run. 1 Frame means every frame, 2 means every other frame, etc. Use this property to tune how many streams you can process on your device.
`class_properties`	Optional key-value map in the format <String, ClassInferenceProperties>. Maps each class label to the ClassInferenceProperties set. Use `""` as key to specify global properties that should affect all the classes of the model. Or a single class name to specify properties for an unique model class. See the fields of `ClassInferenceProperties` in the table below. (optional field*)

ClassInferenceProperties Property	Value
`min_inference_threshold`	Minimum inference threshold, should be set on the [0.0, 1.0] interval. (optional field)
`eps`	Relative difference between sides of the rectangles to merge them into a group. Used in OpenCV groupRectangles function and DBSCAN algorithm. Increase the threshold to reduce potential duplicate detections of a single object. (optional field)
`object_min_size`	Minimum size (`Width x Height` format) in pixels to consider a detected object. (optional field)
`object_max_size`	Maximum size (`Width x Height` format) in pixels to consider a detected object. (optional field)

Metadata

Access this metadata using the Function Node or using the API, from the snapshot or clip saved downstream of this node.

See Function examples here : https://docs.lumeo.com/docs/custom-function-node#accessing-metadata

Metadata Property	Description
`objects`	List of Objects detected by the Model inference nodes. See table below for the format of this property.

'objects': [{
    'id': 5750484150146564100,
    'label': 'car',
    'class_id': 0,
    'probability': 0.70000000149011612,
    'rect': {
        'width': 47,
        'top': 201,
        'left': 656,
        'height': 25
    },
    'attributes': [{
        'label': 'white',
        'probability': 0.9437278509140015,
        'class_id': 10
    }]
}]

Object Properties

Key	Type	Description
`id`	Long Int, Optional	If present, a unique ID assigned to this object by a Track Objects Node
`label`	String	Label for this object as outlined in Model Labels
`class_id`	Int	Unique id for this object's label.
`probability`	Float	Probability of detection
`rect`	Dictionary	Bounding box for this object. Contains: `left`, `top`, `width`, `height`
`attributes`	Dictionary	Object's classes, as identified by Classifier models chained to this Object Detection model. Contains: `label` : Class label, `probability` : Probability, `class_id` : Unique id for the class' label