AI Models

AI Models allow you to extract information from video such as detecting people, vehicles, identifying if a person is wearing a mask etc. Lumeo provides you with a set of curated Models for common use cases, or you can upload your own models in a variety of popular formats (TensorFlow, PyTorch, ONNX, etc.) and use them within a Pipeline.

Overview

AI Models are used in Pipelines to infer on video, and the inference output can be visualized on the stream or sent to your application using other Pipeline Nodes. Lumeo provides Ready-to-use models and allows you to bring your own models to use for inferencing.

Using a Model

You can run an AI Model (BYO or one from the Analytics Library) using the AI Model Node in the Pipeline. To do so, add the Model Inference Node and select the model you'd like to use from Node properties.

The Model inference node adds inference output (detected objects, labels, etc.) for supported (ie non-Custom) model architectures & capabilities to the Pipeline metadata. This can be displayed on the video stream using the Overlay Node and accessed within the Function Node for processing.

For Custom model architectures and capabilities, Lumeo does not extract any metadata, but adds raw inference output tensors to the Pipeline metadata which you can extract and parse using the Function Node.

Ready-to-use Models

The Ready-to-use Models section in the Analytics Library lists a set of curated models that you can use within your Pipelines right away. Lumeo manages these models and keeps your Solution up to date with new versions of these models whenever they are updated.

15811581

A set of curated models available in the Marketplace

Bring-your-own (BYO) AI Models

Lumeo allows you to easily upload and use your own / custom models within Pipelines. Lumeo keeps your Solution up to date with new versions of these models whenever you update them in the Console.

14661466

Easily upload your own vision model and use it in a Pipeline

BYO Model Types

Lumeo supports the following capabilities, architectures and formats of models :

Capability defines the nature of inputs, outputs and what the model does. Lumeo supports models with the following capabilities:

  • Detection : Given an image, detects specific objects in the image & outputs bounding boxes + detection probabilities.
  • Classification : Given an image, outputs a list of categories with probabilities.
  • Custom : For models with "Custom" capabilities, Lumeo will run the model but does not attempt to parse the result to extract an object or a category. You will need to write a custom parser using the Function Node to process the result and extract any relevant information.

Architecture or Topology refers to the structure of the model (# of layers, operations, interconnects, inputs, outputs, output formats, etc.). Lumeo supports the following architectures :

  • Mobilenet
  • Resnet
  • YoloV2, YoloV2 Tiny, YoloV3, YoloV3 Tiny, YoloV4, YoloV4 Tiny
  • SSD
  • Custom : For models with "Custom" architecture, Lumeo will run the model but does not attempt to parse the result to extract an object or a category. You will need to write a custom parser using the Function Node to process the result and extract any relevant information. See Guide here : Custom Model Parser

Format refers to how models are represented & stored, often tied to frameworks used to generate them. Lumeo supports the following model formats:

  • Caffe
  • ONNX
  • UFF
  • Darknet / Yolo Native

Coming soon: TensorFlow, PyTorch native support.

In the Model Conversion guide you can find instructions how to convert the most popular formats to ONNX. If you have a custom model you'd like to use with Lumeo, please contact us and we will get it working with you.

Model Format

Weights File

Metadata File

Caffe

.caffemodel file

.prototxt

ONNX

.onnx file

None

UFF

.uff file

None

Darknet / Yolo Native

.weights file

Yolo Config file.

Must be renamed in this format:
cfg.<model_format>

Ex. cfg.yolov3 , cfg.yolov4-tiny

Model Parameters

Every BYO model requires 2 sets of parameters in order for you to use it with Lumeo : Labels and Configuration, specified under Additional Parameters (JSON).

Labels

Parameter Key : labels
Parameter Value : List of labels, in one of these formats.

Classifier Model Labels

Single line containing labels separated by semicolons ex. label1;label2;label3;label4;

Detection Model Labels

Each label in a new line.

675675

Configuration

Parameter Key : config
Parameter Value : Model properties, from the following table.

Property

Meaning

Type and Range

Example Notes

Network Types

num-detected-classes

Number of classes detected by the network

Integer, >0

num-detected-classes=­91

Detector

net-scale-factor

Pixel normalization factor

Float, >0.0

net-scale-factor=­0.031

All

model-file

Pathname of the caffemodel file

String

model-file=<model_uuid>.caffemodel

All

proto-file

Pathname of the prototxt file

String

proto-file=<model_uuid>.prototxt

All

int8-calib-file

Pathname of the INT8 calibration file for dynamic range adjustment with an FP32 model

String

int8-calib-file=<model_uuid>.­int8

All

batch-size

Number of frames or objects to be inferred together in a batch

Integer, >0

batch-size=30

All

uff-file

Pathname of the UFF model file

String

uff-file=
<model_uuid>.uff

All

onnx-file

Pathname of the ONNX model file

String

onnx-file=<model_uuid>.onnx

All

labelfile-path

Pathname of a text file containing the labels for the model

String

labelfile-path=<model_uuid>.labels

Detector & classifier

mean-file

Pathname of mean data file (PPM format)

String

mean-file=­<model_uuid>.ppm

All

input-object-min-width

Secondary Inference Node infers only on objects with this minimum width

Integer, >0

put-object-min-width=40

All
(Just applies to secondary inference nodes)

input-object-min-height

Secondary Inference Node infers only on objects with this minimum height

Integer, >0

input-object-min-height=40

All
(Just applies to secondary inference nodes)

input-object-max-width

Secondary Inference Node infers only on objects with this maximum width

Integer, >0

input-object-max-width=256

0 disables the threshold

All
(Just applies to secondary inference nodes)

input-object-max-height

Secondary Inference Node infers only on objects with this maximum height

Integer, >0

input-object-max-height=256

All
(Just applies to secondary inference nodes)

network-mode

Data format to be used by inference

Integer
0: FP32
1: INT8
2: FP16

network-mode=0

All

offsets

Array of mean values of color components to be subtracted from each pixel. Array length must equal the number of color components in the frame. The plugin multiplies mean values by net-scale-factor.

Semicolon delimited float array, all values ≥0

offsets=77.5;21.2;11.8

All

output-blob-names

Array of output layer names

Semicolon delimited string array

For detector: output-blob-names=coverage;bbox

For multi-label classifiers: output-blob-names=coverage_attrib1;coverage_attrib2

All

parse-bbox-func-name

Name of the custom bounding box parsing function. If not specified, uses the internal function for a resnet model

String

parse-bbox-func-name=
parse_bbox_resnet

Detector

parse-bbox-instance-mask-func-name

Name of the custom instance segmentation parsing function. It is mandatory for instance segmentation network as there is no internal function.

String

parse-bbox-instance-mask-func-name=NvDsInferParseCustomMrcnnTLT

Segmentation

custom-lib-path

Absolute pathname of a library containing custom method implementations for custom models

String

custom-lib-path=/opt/lumeo/lib/objectdetector_fasterrcnn.so

or

custom-lib-path=/opt/lumeo/lib/objectdetector_ssd.so

or

custom-lib-path=/opt/lumeo/lib/objectdetector_yolo.so

All

model-color-format

Color format required by the model

Integer 0: RGB 1: BGR 2: GRAY

model-color-format=0

All

classifier-async-mode

Enables inference on detected objects and asynchronous metadata attachments. Works only when tracker-ids are attached. Pushes buffer downstream without waiting for inference results. Attaches metadata after the inference results are available to next Gst Buffer in its internal queue.

Boolean

classifier-async-mode=1

Classifier

classifier-threshold

Minimum threshold label probability. The Inference Node outputs the label having the highest probability if it is greater than this threshold

Float, ≥0

classifier-threshold=0.4

Classifier

uff-input-blob-name

Name of the input blob in the UFF file

String

uff-input-blob-name=Input_1

All

enable-dla

Indicates whether to use the DLA engine for inferencing. Note: DLA is supported only on NVIDIA® Jetson AGX Xavier™. Currently work in progress.

Boolean

enable-dla=1

All

network-type

Type of network

Integer

0: Detector
1: Classifier
2: Segmentation
3: Instance Segmentation

network-type=1

All

maintain-aspect-ratio

Indicates whether to maintain aspect ratio while scaling input. Currently the inference engine supports asymmetric padding only.

Boolean

maintain-aspect-ratio=1

All

custom-network-config

Pathname of the configuration file for custom networks available in the custom interface for creating CUDA engines.

String

custom-network-config=<model_uuid>.config

All

tlt-encoded-model

Pathname of the Transfer Learning Toolkit (TLT) encoded model.

String

tlt-encoded-model=­
­<model_uuid>.etlt

All

segmentation-threshold

Confidence threshold for the segmentation model to output a valid class for a pixel. If confidence is less than this threshold, class output for that pixel is −1.

Float, ≥0.0

segmentation-threshold=0.3

Segmentation

segmentation-output-order

Segmentation network output layer order

Integer 0: NCHW 1: NHWC

segmentation-output-order=1

Segmentation

workspace-size

Workspace size to be used by the engine, in MB

Integer, >0

workspace-size=45

All

force-implicit-batch-dim

When a network supports both implicit batch dimension and full dimension, force the implicit batch dimension mode.

Boolean

force-implicit-batch-dim=1

All

infer-dims

Binding dimensions to set on the image input layer.

channel;

infer-dims=3;224;224

All

uff-input-order

UFF input layer order

Integer 0: NCHW 1: NHWC 2: NC

uff-input-order=1

All

cluster-mode

Clustering algorithm to use. Refer to the next table for configuring the algorithm specific parameters.

Integer
0: OpenCV groupRectangles()
1: DBSCAN
2: Non Maximum Suppression
3: DBSCAN + NMS Hybrid
4: No clustering

cluster-mode=2

cluster-mode=4 for instance segmentation

Detector

filter-out-class-ids

Filter out detected objects belonging to specified class-ids

Semicolon delimited integer array

filter-out-class-ids=1;2

Detector

scaling-filter

The filter to use for scaling frames / object crops to network resolution

Integer, refer to enum NvBufSurfTransform_Inter in nvbufsurftransform.h for valid values

scaling-filter=1

All

scaling-compute-hw

Compute hardware to use for scaling frames / object crops to network resolution

Integer
0: Platform default – GPU (dGPU), VIC (Jetson)
1: GPU
2: VIC (Jetson only)

scaling-compute-hw=2

All

output-io-formats

Specifies the data type and order for bound output layers. For layers not specified, defaults to FP32 and CHW

Semi-colon separated list of format. ::;::

data-type should be one of [fp32, fp16, int32, int8]

order should be one of [chw, chw2, chw4, hwc8, chw16, chw32]

output-io-formats=conv2d_bbox:fp32:chw;conv2d_cov/Sigmoid:fp32:chw

All

layer-device-precision

Specifies the device type and precision for any layer in the network

Semi-colon separated list of format. <layer1-name>:<precision>:<device-type>;<layer2-name>:<precision>:<device-type>;

precision should be one of [fp32, fp16, int8]

Device-type should be one of [gpu, dla]

layer-device-precision=output_cov/Sigmoid:fp32:gpu;output_bbox/BiasAdd:fp32:gpu;

All

File Name in Configuration

Please note the onnx-file should follow <model_unique_id>.<weights_file_extension> format.
By its turn, labelfile-path uses <model_unique_id>.labels format.

You can find the model's unique ID in the top left corner of the page, after saving the model.

550550

Classifier Example

For example, consider an ONNX classifier model with the Weights file named: car-colors-classifier.onnx

A possible configuration for this model is shown bellow:

879879

Additional Parameters (JSON)

config

[property]
input-dims=3;500;500;0
model-color-format=0
onnx-file=100e2157-0ce1-4fca-9087-5e4eccab3fdf.onnx
labelfile-path=100e2157-0ce1-4fca-9087-5e4eccab3fdf.labels
network-mode=2
is-classifier=1
classifier-threshold=0.4
output-blob-names=sequential_3

labels

black;blue;brown;gold;green;grey;maroon;orange;red;silver;white;yellow

Detector example

For a YOLO detector, besides the weights file you will need to upload the Metadata File containing the network architecture, whose extension should match the YOLO version you're using (in this case yolov4-tiny)

10051005

Additional Parameters (JSON)

config

[property]

gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
custom-network-config=b6861749-5251-4e55-b6ef-1419edcce5b7.yolov4-tiny
model-file=b6861749-5251-4e55-b6ef-1419edcce5b7.weights
labelfile-path=b6861749-5251-4e55-b6ef-1419edcce5b7.labels
batch-size=1
network-mode=2
num-detected-classes=80
interval=0
gie-unique-id=1
process-mode=1
network-type=0
cluster-mode=4
maintain-aspect-ratio=0
parse-bbox-func-name=NvDsInferParseYolo
custom-lib-path=/opt/lumeo/lib/objectdetector_yolo.so
engine-create-func-name=NvDsInferYoloCudaEngineGet

[class-attrs-all]
pre-cluster-threshold=0.25

labels

person
bicycle
car
motorbike
aeroplane
bus
train
(...)

API Reference

See models


Did this page help you?