Google Gemini Lookup

Perform a lookup with Gemini on objects in an ROI, or on a ROI in the frame.

Overview

The Google Gemini Lookup node is designed to perform lookups using the powerful Gemini model on objects within a Region of Interest (ROI), or on a ROI within the frame. This functionality is useful for applications requiring advanced recognition and understanding of objects or areas within a video feed.

Inputs & Outputs

  • Inputs: 1, Media Format: Raw Video
  • Outputs: 1, Media Format: Raw Video
  • Output Metadata: nodes.node_id, recognized_objs, recognized_obj_ids, recognized_obj_count, recognized_obj_delta, value_changed_delta, unrecognized_obj_count, unrecognized_obj_delta

Properties

PropertyDescriptionTypeDefaultRequired
roi_labelsRegions of interest labelshiddenYes
roisRegions of interest. Type: polygon. Default: null. Conditional on roi_labels.polygonnullYes
processing_modeProcessing mode. Options: ROIs, at Interval (rois_interval), ROIs, upon Trigger (rois_trigger), Objects in an ROI (objects).enumrois_intervalYes
intervalLookup interval. Collect objects or ROIs for lookup at least this many seconds apart.float1No
batch_sizeNumber of images to process in each Gemini request. Set to more than 1 to use prompts that reference multiple images.number1No
triggerQueue ROI for lookup when this condition evaluates to true. Conditional on processing_mode being rois_trigger.trigger-conditionnullNo
enable_prebufferIf true, samples ROI at interval to fill batch, and performs lookup when trigger is met. Else performs lookup when batch is full. Conditional on processing_mode being rois_trigger.boolfalseNo
objects_to_processObject types to process. ex. car,person,car.red. Conditional on processing_mode being objects.model-labelnullNo
min_obj_size_pixelsMin. width and height of an object. Conditional on processing_mode being objects.number64No
obj_lookup_size_change_thresholdObject size ratio change threshold. Slider min: 0.1, max: 2.0, step: 0.2. Conditional on processing_mode being objects.slider0.1No
max_lookups_per_objMax. attempts per object. Conditional on processing_mode being objects.number5No
gemini_api_keyGemini API KeystringnullYes
promptGemini PrompttextnullYes
detail_levelImage Detail Level. Options: Low, High.enumlowNo
max_tokensMaximum number of tokens to return for each request.number500No
post_processingGemini Results Post Processing. Options: Text, JSON.enumtextNo
post_proc_attribs_pathJSON path to extract object/ROI attributes from Gemini response. ex. object_color. Conditional on post_processing being json.stringnullNo
display_roiDisplay ROI on video?booltrueNo
display_objinfoDisplay results on video? Options: Disabled, Bottom left, Bottom right, Top left, Top right.enumbottom_leftNo
debugLog debugging information?boolfalseNo

Metadata

Metadata PropertyDescription
nodes.[node_id].rois.[roi_id].label_changed_deltaIndicates if there has been a change in the label of the ROI.
nodes.[node_id].rois.[roi_id].label_availableIndicates if a label is available for the ROI.
nodes.[node_id].rois.[roi_id].labelLabel contents for the ROI. String if post_processing property is text, else JSON object.
nodes.[node_id].recognized_obj_countThe count of recognized objects.
nodes.[node_id].recognized_obj_deltaThe change in the count of recognized objects.
nodes.[node_id].label_changed_obj_deltaThe change in the count of objects with changed labels.
nodes.[node_id].unrecognized_obj_countThe count of unrecognized objects.
nodes.[node_id].unrecognized_obj_deltaThe change in the count of unrecognized objects.

Example JSON

{
    "nodes": {
        "gemini1": {
            "type": "gemini",
            "rois": {
                "roi2": {
                    "label_changed_delta": true,
                    "label_available": true,
                    "label": {
                        "door_status": "unblocked"
                    }
                }
            },
            "recognized_obj_ids": ["2775161862"],
            "recognized_obj_count": 1,
            "recognized_obj_delta": 1,
            "label_changed_obj_delta": 1,
            "unrecognized_obj_count": 0,
            "unrecognized_obj_delta": 0,
            "objects_of_interest_keys": ["recognized_obj_ids"]
        }
    },
    "objects": [{
        "id": 2775161862,
        "source_node_id": null,
        "model_id": null,
        "label": "roi2",
        "class_id": 10600,
        "rect": {
            "left": 128,
            "top": 72,
            "width": 512,
            "height": 575
        },
        "probability": 1.0,
        "attributes": [{
            "label": "unblocked",
            "class_id": 10602,
            "probability": 1.0
        }, {
            "label": "lvm_results",
            "class_id": 10601,
            "probability": 1.0
        }, {
            "label": "lvm_roi",
            "class_id": 10600,
            "probability": 1.0
        }],
        "corr_id": "75f5141e-020a-4f27-af26-cf17b32c2544"
    }]
}