Overview

The Google Gemini Lookup node is designed to perform lookups using the powerful Gemini model on objects within a Region of Interest (ROI), or on a ROI within the frame. This functionality is useful for applications requiring advanced recognition and understanding of objects or areas within a video feed.

Inputs & Outputs

• Inputs: 1, Media Format: Raw Video
• Outputs: 1, Media Format: Raw Video
• Output Metadata: nodes.node_id, recognized_objs, recognized_obj_ids, recognized_obj_count, recognized_obj_delta, value_changed_delta, unrecognized_obj_count, unrecognized_obj_delta

Properties

Prompt Examples

Generate scene description.

Analyze the scene and provide a concise description of any unique, interesting, or noteworthy elements that would be suitable for a push notification alert. Focus on key details that capture the essence of what's happening or what's important in the image.

Attribute Examples

Providing explicit attributes lets the model return structured output that will be added as ROI or object attributes.

Each attribute is a key-value pair. Key is the attribute name, and value is the instruction for the model to extract the attribute.
The model will return the extracted attribute value as a string, which will be added as an attribute to the object/ROI.

Describe the image
{"description": "Describe the image briefly."}

Describe the image and add attributes for vehicle type and numbers
{"description": "Describe the image briefly. Return null if no vehicle is present.", "vehicle_type": "Comma separated list of vehicle types: car\|bus\|van", "vehicle_numbers": "Comma separated list of vehicle numbers"}

Output Metadata

The fields below are declared by this node's metadata schema; the JSON values are representative examples.

JSON example

{
  "nodes": {
    "gemini1": {
      "label_changed_obj_delta": 0,
      "objects_of_interest_keys": [],
      "recognized_obj_count": 0,
      "recognized_obj_delta": 0,
      "recognized_obj_ids": [],
      "rois": {
        "roi1": {
          "attributes": {
            "door_status": "unblocked"
          },
          "label": "example",
          "label_available": false,
          "label_changed_delta": false
        }
      },
      "type": "gemini",
      "unrecognized_obj_count": 0,
      "unrecognized_obj_delta": 0
    }
  }
}

Object labels and attributes

• Object labels/classes added: In ROI mode, the configured ROI label is added as an object with class 10600.
• Object attribute labels/classes added: ROI objects receive lvm_roi (10600); configured response values use class 10602; successful results add lvm_results (10601).

{
    "objects": [{
        "id": 2775161862,
        "source_node_id": null,
        "model_id": null,
        "label": "roi2",
        "class_id": 10600,
        "rect": {
            "left": 128,
            "top": 72,
            "width": 512,
            "height": 575
        },
        "probability": 1.0,
        "attributes": [{
            "label": "unblocked",
            "class_id": 10602,
            "probability": 1.0
        }, {
            "label": "lvm_results",
            "class_id": 10601,
            "probability": 1.0
        }, {
            "label": "lvm_roi",
            "class_id": 10600,
            "probability": 1.0
        }],
        "corr_id": "75f5141e-020a-4f27-af26-cf17b32c2544"
    }]
}