AI Model

Run an AI Model inference on input frames, and add detected objects to metadata.

Overview

Use the Model inference node to run an AI model on the video stream. The node adds inference outputs (object bounding boxes, labels, probabilities) and raw tensor outputs from the Model to the metadata.

Use the Overlay Node after the Inference Node to display the output on the video stream, and/or Function Node if you need to process the output and use the metadata in your app.

Labels for off-the-shelf models can be found under Model Labels

Detection Models

Detection Models identify objects in a stream, and provide you with the object label along with the coordinates for the object's bounding box. Detection Models can be run on an entire frame or on the output of another Detection Model.

Examples of Detection Models : Face detector, Traffic Detector

Classifier Models

Classifier Models (Such as a Mask Classifier, Vehicle Type Classifier, etc.) operate on the output of a Detector Model and identify the type of the object detected by the Detector.

In order to use a Classifer Model, add a 2nd Model Inference Node after the one that runs the Detector Model, and use the infer_on_node property to configure the 2nd node to operate on the output of the 1st one.

Inputs & Outputs

  • Inputs : 1, Media Format : Raw or Multiplexed Video
  • Outputs : Raw or Multiplexed Video (same as Input format)
  • Output Metadata : Objects, Bounding boxes, classes and Raw tensors.

Properties

PropertyValue
modelThe Marketplace or BYO model to use for inference. See AI Models for details on how to upload your own model.
infer_on_nodeSelect another model inference node from this pipeline to infer only on objects previously detected by that node. Typically used for Classifier models that classify the objects detected by a Detector model. Leave this empty if you want the model to run on the entire frame.
inference_intervalFrequency, in Frames, with which the model should run. 1 Frame means every frame, 2 means every other frame, etc. Use this property to tune how many streams you can process on your device.
class_propertiesOptional key-value map in the format <String, ClassInferenceProperties>. Maps each class label to the ClassInferenceProperties set. Use "*" as key to specify global properties that should affect all the classes of the model. Or a single class name to specify properties for an unique model class. See the fields of ClassInferenceProperties in the table below. (optional field)
ClassInferenceProperties PropertyValue
min_inference_thresholdMinimum inference threshold, should be set on the [0.0, 1.0] interval. (optional field)
epsRelative difference between sides of the rectangles to merge them into a group. Used in OpenCV groupRectangles function and DBSCAN algorithm. Increase the threshold to reduce potential duplicate detections of a single object. (optional field)
object_min_sizeMinimum size (Width x Height format) in pixels to consider a detected object. (optional field)
object_max_sizeMaximum size (Width x Height format) in pixels to consider a detected object. (optional field)

Metadata

Access this metadata using the Function Node or using the API, from the snapshot or clip saved downstream of this node.

See Function examples here : https://docs.lumeo.com/docs/custom-function-node#accessing-metadata

Metadata PropertyDescription
objectsList of Objects detected by the Model inference nodes. See table below for the format of this property.
'objects': [{
    'id': 5750484150146564100,
    'label': 'car',
    'class_id': 0,
    'probability': 0.70000000149011612,
    'rect': {
        'width': 47,
        'top': 201,
        'left': 656,
        'height': 25
    },
    'attributes': [{
        'label': 'white',
        'probability': 0.9437278509140015,
        'class_id': 10
    }]
}]

Object Properties

KeyTypeDescription
idLong Int, OptionalIf present, a unique ID assigned to this object by a Track Objects Node
labelStringLabel for this object as outlined in Model Labels
class_idIntUnique id for this object's label.
probabilityFloatProbability of detection
rectDictionaryBounding box for this object. Contains: left, top, width, height
attributesDictionaryObject's classes, as identified by Classifier models chained to this Object Detection model. Contains: label : Class label, probability : Probability, class_id : Unique id for the class' label