Overview

The Vision Language Model Lookup node is designed to perform lookups using Vision Language Models (using Lumeo Cloud, Google, OpenAI, Anthropic, Nvidia, AWS Bedrock) on objects within a Region of Interest (ROI), or on a ROI within the frame. This functionality is useful for applications requiring advanced recognition and understanding of objects or areas within a video feed.

Inputs & Outputs

• Inputs: 1, Media Format: Raw Video
• Outputs: 1, Media Format: Raw Video
• Output Metadata: nodes.node_id, recognized_objs, recognized_obj_ids, recognized_obj_count, recognized_obj_delta, value_changed_delta, unrecognized_obj_count, unrecognized_obj_delta

Properties

Prompt Examples

Generate scene description.

Analyze the scene and provide a concise description of any unique, interesting, or noteworthy elements that would be suitable for a push notification alert. Focus on key details that capture the essence of what's happening or what's important in the image.

Attribute Examples

Providing explicit attributes lets the model return structured output that will be added as ROI or object attributes.

Each attribute is a key-value pair. Key is the attribute name, and value is the instruction for the model to extract the attribute.
The model will return the extracted attribute value as a string, which will be added as an attribute to the object/ROI.

Describe the image

{"description": "Describe the image briefly."}

Describe the image and add attributes for vehicle type and numbers

{"description": "Describe the image briefly. Return null if no vehicle is present.", "vehicle_type": "Comma separated list of vehicle types: car\|bus\|van", "vehicle_numbers": "Comma separated list of vehicle numbers"}

Custom Model Names

Metadata

Example JSON

{
    "nodes": {
        "lvm1": {
            "type": "lvm",
            "rois": {
                "roi2": {
                    "label_changed_delta": true,
                    "label_available": true,
                    "label": "The scene contains 2 vehicles that seem to be in an accident.",
                    "attributes": {
                        "vehicle_type": "car, bus",
                        "vehicle_numbers": "1234, 5678"
                    }
                }
            },
            "recognized_obj_ids": ["2775161862"],
            "recognized_obj_count": 1,
            "recognized_obj_delta": 1,
            "label_changed_obj_delta": 1,
            "unrecognized_obj_count": 0,
            "unrecognized_obj_delta": 0,
            "objects_of_interest_keys": ["recognized_obj_ids"]
        }
    },
    "objects": [{
        "id": 2775161862,
        "source_node_id": null,
        "model_id": null,
        "label": "roi2",
        "class_id": 10600,
        "rect": {
            "left": 128,
            "top": 72,
            "width": 512,
            "height": 575
        },
        "probability": 1.0,
        "attributes": [{
            "label": "unblocked",
            "class_id": 10602,
            "probability": 1.0
        }, {
            "label": "lvm_results",
            "class_id": 10601,
            "probability": 1.0
        }, {
            "label": "lvm_roi",
            "class_id": 10600,
            "probability": 1.0
        }],
        "corr_id": "75f5141e-020a-4f27-af26-cf17b32c2544"
    }]
}