Overview

The Google Gemini Lookup node is designed to perform lookups using the powerful Gemini model on objects within a Region of Interest (ROI), or on a ROI within the frame. This functionality is useful for applications requiring advanced recognition and understanding of objects or areas within a video feed.

Inputs & Outputs

Inputs: 1, Media Format: Raw Video
Outputs: 1, Media Format: Raw Video
Output Metadata: nodes.node_id, recognized_objs, recognized_obj_ids, recognized_obj_count, recognized_obj_delta, value_changed_delta, unrecognized_obj_count, unrecognized_obj_delta

Properties

Property	Description	Type	Default	Required
`roi_labels`	Regions of interest labels	hidden		Yes
`rois`	Regions of interest. Type: polygon. Default: null. Conditional on `roi_labels`.	polygon	null	Yes
`processing_mode`	Processing mode. Options: ROIs, at Interval (rois_interval), ROIs, upon Trigger (rois_trigger), Objects in an ROI (objects).	enum	rois_interval	Yes
`interval`	Lookup interval. Collect objects or ROIs for lookup at least this many seconds apart.	float	1	No
`batch_size`	Number of images to process in each Gemini request. Set to more than 1 to use prompts that reference multiple images.	number	1	No
`trigger`	Queue ROI for lookup when this condition evaluates to true. Conditional on `processing_mode` being `rois_trigger`.	trigger-condition	null	No
`enable_prebuffer`	If true, samples ROI at interval to fill batch, and performs lookup when trigger is met. Else performs lookup when batch is full. Conditional on `processing_mode` being `rois_trigger`.	bool	false	No
`objects_to_process`	Object types to process. ex. `car`,`person`,`car.red`. Conditional on `processing_mode` being `objects`.	model-label	null	No
`min_obj_size_pixels`	Min. width and height of an object. Conditional on `processing_mode` being `objects`.	number	64	No
`obj_lookup_size_change_threshold`	Object size ratio change threshold. Slider min: 0.1, max: 2.0, step: 0.2. Conditional on `processing_mode` being `objects`.	slider	0.1	No
`max_lookups_per_obj`	Max. attempts per object. Conditional on `processing_mode` being `objects`.	number	5	No
`gemini_api_key`	Gemini API Key	string	null	Yes
`prompt`	Gemini Prompt	text	null	Yes
`detail_level`	Image Detail Level. Options: Low, High.	enum	low	No
`max_tokens`	Maximum number of tokens to return for each request.	number	500	No
`post_processing`	Gemini Results Post Processing. Options: Text, JSON.	enum	text	No
`post_proc_attribs_path`	JSON path to extract object/ROI attributes from Gemini response. ex. `object_color`. Conditional on `post_processing` being `json`.	string	null	No
`display_roi`	Display ROI on video?	bool	true	No
`display_objinfo`	Display results on video? Options: Disabled, Bottom left, Bottom right, Top left, Top right.	enum	bottom_left	No
`debug`	Log debugging information?	bool	false	No

Metadata

Metadata Property	Description
`nodes.[node_id].rois.[roi_id].label_changed_delta`	Indicates if there has been a change in the label of the ROI.
`nodes.[node_id].rois.[roi_id].label_available`	Indicates if a label is available for the ROI.
`nodes.[node_id].rois.[roi_id].label`	Label contents for the ROI. String if `post_processing` property is `text`, else JSON object.
`nodes.[node_id].recognized_obj_count`	The count of recognized objects.
`nodes.[node_id].recognized_obj_delta`	The change in the count of recognized objects.
`nodes.[node_id].label_changed_obj_delta`	The change in the count of objects with changed labels.
`nodes.[node_id].unrecognized_obj_count`	The count of unrecognized objects.
`nodes.[node_id].unrecognized_obj_delta`	The change in the count of unrecognized objects.

Example JSON

{
    "nodes": {
        "gemini1": {
            "type": "gemini",
            "rois": {
                "roi2": {
                    "label_changed_delta": true,
                    "label_available": true,
                    "label": {
                        "door_status": "unblocked"
                    }
                }
            },
            "recognized_obj_ids": ["2775161862"],
            "recognized_obj_count": 1,
            "recognized_obj_delta": 1,
            "label_changed_obj_delta": 1,
            "unrecognized_obj_count": 0,
            "unrecognized_obj_delta": 0,
            "objects_of_interest_keys": ["recognized_obj_ids"]
        }
    },
    "objects": [{
        "id": 2775161862,
        "source_node_id": null,
        "model_id": null,
        "label": "roi2",
        "class_id": 10600,
        "rect": {
            "left": 128,
            "top": 72,
            "width": 512,
            "height": 575
        },
        "probability": 1.0,
        "attributes": [{
            "label": "unblocked",
            "class_id": 10602,
            "probability": 1.0
        }, {
            "label": "lvm_results",
            "class_id": 10601,
            "probability": 1.0
        }, {
            "label": "lvm_roi",
            "class_id": 10600,
            "probability": 1.0
        }],
        "corr_id": "75f5141e-020a-4f27-af26-cf17b32c2544"
    }]
}