Overview

The Vision Language Model Lookup node is designed to perform lookups using Vision Language Models (using Lumeo Cloud, Google, OpenAI, Anthropic, Nvidia, AWS Bedrock) on objects within a Region of Interest (ROI), or on a ROI within the frame. This functionality is useful for applications requiring advanced recognition and understanding of objects or areas within a video feed.

Inputs & Outputs

• Inputs: 1, Media Format: Raw Video
• Outputs: 1, Media Format: Raw Video
• Output Metadata: nodes.node_id, recognized_objs, recognized_obj_ids, recognized_obj_count, recognized_obj_delta, value_changed_delta, unrecognized_obj_count, unrecognized_obj_delta

Properties

Prompt Examples

Generate scene description.

Analyze the scene and provide a concise description of any unique, interesting, or noteworthy elements that would be suitable for a push notification alert. Focus on key details that capture the essence of what's happening or what's important in the image.

Attribute Examples

Providing explicit attributes lets the model return structured output that will be added as ROI or object attributes.

Each attribute is a key-value pair. Key is the attribute name, and value is the instruction for the model to extract the attribute.
The model will return the extracted attribute value as a string, which will be added as an attribute to the object/ROI.

Describe the image

{"description": "Describe the image briefly."}

Describe the image and add attributes for vehicle type and numbers

{"description": "Describe the image briefly. Return null if no vehicle is present.", "vehicle_type": "Comma separated list of vehicle types: car\|bus\|van", "vehicle_numbers": "Comma separated list of vehicle numbers"}

Add an alert flag in the metadata
{"alert": "Is this person wearing a pink shirt?"}

Publishing attributes as metrics

The Publish Metrics and Publish to BigQuery nodes can publish VLM ROI attributes. They read nodes.<vlm_node_id>.rois.<roi_label>.attributes and send one Node Metadata record or BigQuery row for each non-null attribute. For example, "vehicle_type": "car" is published as:

{
  "node_type": "vlm",
  "roi_label": "roi1",
  "node_meta": {
    "roi_label": "roi1",
    "attributes": {
      "name": "vehicle_type",
      "value": "car"
    }
  }
}

Attribute values retain their string, number, or boolean type; null values are omitted. With no explicit trigger on the publisher, the VLM's label_changed_delta fields ensure that only frames containing changed results are collected.

Objects to Extract Examples

Certain models support extracting bounding boxes by describing the objects you are looking to detect (this is highly experimental at the moment).
For those models, you can specify objects_to_extract as a JSON with each key being the object label and value being the description, as follows:

{"person_ppe_violation": "Person violating ppe protocol. Properties: vest=vest/no_vest, helmet=helmet/no_helmet, goggles=goggles/no_goggles"}

This will insert objects with label person_ppe_detection that match the description, and add the properties as object attributes.

Custom Model Names

Output Metadata

The fields below are declared by this node's metadata schema; the JSON values are representative examples.

JSON example

{
  "nodes": {
    "vlm1": {
      "alert": false,
      "alert_message": "value",
      "alert_obj_ids": [],
      "label_changed_obj_delta": 0,
      "objects_of_interest_keys": [],
      "recognized_obj_count": 0,
      "recognized_obj_delta": 0,
      "recognized_obj_ids": [],
      "rois": {
        "roi1": {
          "attributes": {
            "attribute_name": "value"
          },
          "label": "example",
          "label_available": false,
          "label_changed_delta": false
        }
      },
      "search_text": "value",
      "type": "vlm",
      "unrecognized_obj_count": 0,
      "unrecognized_obj_delta": 0
    }
  }
}

Object labels and attributes

• Object labels/classes added: ROI mode adds the configured ROI label with class 10600. Object extraction adds model-returned labels, falling back to vlm_object, with class 9998.
• Object attribute labels/classes added: ROI objects receive lvm_roi (10600); configured response values use 10602; extracted objects receive vlm_extracted (9998) and extracted property values use 9999; successful results add vlm_results (10601); generated object descriptions use 10604.

{
  "objects": [{
    "id": 2775161862,
    "source_node_id": null,
    "model_id": null,
    "label": "roi2",
    "class_id": 10600,
    "rect": {
        "left": 128,
        "top": 72,
        "width": 512,
        "height": 575
    },
    "probability": 1.0,
    "attributes": [{
        "label": "unblocked",
        "class_id": 10602,
        "probability": 1.0
    }, {
        "label": "lvm_results",
        "class_id": 10601,
        "probability": 1.0
    }, {
        "label": "lvm_roi",
        "class_id": 10600,
        "probability": 1.0
    }],
    "corr_id": "75f5141e-020a-4f27-af26-cf17b32c2544"
  }]
}