Overview

The Local Vision Language Model Lookup (VLM) node performs visual question answering on objects within a Region of Interest (ROI), or on a ROI within the frame. It uses local inference with models like InternVL2-1B (small) and InternVL2-2B (large) to analyze and answer questions about image content.

Inputs & Outputs

• Inputs: 1, Media Format: Raw Video
• Outputs: 1, Media Format: Raw Video
• Output Metadata: nodes.node_id, recognized_objs, recognized_obj_ids, recognized_obj_count, recognized_obj_delta, label_changed_obj_delta

Properties

Prompt Examples

Generate scene description.

Analyze the scene and provide a concise description of any unique, interesting, or noteworthy elements that would be suitable for a push notification alert. Focus on key details that capture the essence of what's happening or what's important in the image.

Attribute Examples

Providing explicit attributes lets the model return structured output that will be added as ROI or object attributes.

Each attribute is a key-value pair. Key is the attribute name, and value is the instruction for the model to extract the attribute.
The model will return the extracted attribute value as a string, which will be added as an attribute to the object/ROI.

Describe the image

{"description": "Describe the image briefly."}

Describe the image and add attributes for vehicle type and numbers

{"description": "Describe the image briefly. Return null if no vehicle is present.", "vehicle_type": "Comma separated list of vehicle types: car\|bus\|van", "vehicle_numbers": "Comma separated list of vehicle numbers"}

Add an alert flag in the metadata
{"alert": "Is this person wearing a pink shirt?"}

Publishing attributes as metrics

The Publish Metrics and Publish to BigQuery nodes can publish VLM ROI attributes. They read nodes.<vlm_node_id>.rois.<roi_label>.attributes and send one Node Metadata record or BigQuery row for each non-null attribute. For example, "vehicle_type": "car" is published as:

{
  "node_type": "vlm",
  "roi_label": "roi1",
  "node_meta": {
    "roi_label": "roi1",
    "attributes": {
      "name": "vehicle_type",
      "value": "car"
    }
  }
}

Attribute values retain their string, number, or boolean type; null values are omitted. With no explicit trigger on the publisher, the VLM's label_changed_delta fields ensure that only frames containing changed results are collected.

Model Types

InternVL2_5-1B (Small)

• Faster inference
• Good for basic scene description and object detection
• Lower memory requirements
• Default model: OpenGVLab/InternVL2_5-1B
• Model size: 2.0 GB download
• GPU Memory: Requires ~2.5GB of GPU memory

InternVL2_5-2B (Large)

• More detailed and nuanced responses
• Better understanding of complex scenes
• Higher memory requirements
• Default model: OpenGVLab/InternVL2_5-2B
• Model size: 4.5 GB download
• GPU Memory: Requires ~5.0GB of GPU memory

Output Metadata

The fields below are declared by this node's metadata schema; the JSON values are representative examples.

JSON example

{
  "nodes": {
    "local_vlm1": {
      "alert": false,
      "alert_message": "value",
      "alert_obj_ids": [],
      "label_changed_obj_delta": 0,
      "objects_of_interest_keys": [],
      "recognized_obj_count": 0,
      "recognized_obj_delta": 0,
      "recognized_obj_ids": [],
      "rois": {
        "roi1": {
          "attributes": {
            "attribute_name": "value"
          },
          "label": "example",
          "label_available": false,
          "label_changed_delta": false
        }
      },
      "type": "local_vlm",
      "unrecognized_obj_count": 0,
      "unrecognized_obj_delta": 0
    }
  }
}

Object labels and attributes

• Object labels/classes added: ROI mode adds the configured ROI label with class 10600. Object extraction adds model-returned labels, falling back to vlm_object, with class 9998.
• Object attribute labels/classes added: ROI objects receive lvm_roi (10600); configured response values use 10602; extracted objects receive vlm_extracted (9998); successful results add vlm_results (10601).

{
  "objects": [{
    "id": 2775161862,
    "source_node_id": null,
    "model_id": null,
    "label": "roi2",
    "class_id": 10600,
    "rect": {
        "left": 128,
        "top": 72,
        "width": 512,
        "height": 575
    },
    "probability": 1.0,
    "attributes": [{
        "label": "unblocked",
        "class_id": 10602,
        "probability": 1.0
    }, {
        "label": "lvm_results",
        "class_id": 10601,
        "probability": 1.0
    }, {
        "label": "lvm_roi",
        "class_id": 10600,
        "probability": 1.0
    }],
    "corr_id": "75f5141e-020a-4f27-af26-cf17b32c2544"
  }]
}