Google Gemini Lookup
Perform a lookup with Gemini on objects in an ROI, or on a ROI in the frame.
Overview
The Google Gemini Lookup node is designed to perform lookups using the powerful Gemini model on objects within a Region of Interest (ROI), or on a ROI within the frame. This functionality is useful for applications requiring advanced recognition and understanding of objects or areas within a video feed.
Inputs & Outputs
- Inputs: 1, Media Format: Raw Video
- Outputs: 1, Media Format: Raw Video
- Output Metadata:
nodes.node_id
,recognized_objs
,recognized_obj_ids
,recognized_obj_count
,recognized_obj_delta
,value_changed_delta
,unrecognized_obj_count
,unrecognized_obj_delta
Properties
Property | Description | Type | Default | Required |
---|---|---|---|---|
roi_labels | Regions of interest labels | hidden | Yes | |
rois | Regions of interest. Type: polygon. Default: null. Conditional on roi_labels . | polygon | null | Yes |
processing_mode | Processing mode. Options: ROIs, at Interval (rois_interval), ROIs, upon Trigger (rois_trigger), Objects in an ROI (objects). | enum | rois_interval | Yes |
interval | Lookup interval. Collect objects or ROIs for lookup at least this many seconds apart. | float | 1 | No |
batch_size | Number of images to process in each Gemini request. Set to more than 1 to use prompts that reference multiple images. | number | 1 | No |
trigger | Queue ROI for lookup when this condition evaluates to true. Conditional on processing_mode being rois_trigger . | trigger-condition | null | No |
enable_prebuffer | If true, samples ROI at interval to fill batch, and performs lookup when trigger is met. Else performs lookup when batch is full. Conditional on processing_mode being rois_trigger . | bool | false | No |
objects_to_process | Object types to process. ex. car ,person ,car.red . Conditional on processing_mode being objects . | model-label | null | No |
min_obj_size_pixels | Min. width and height of an object. Conditional on processing_mode being objects . | number | 64 | No |
obj_lookup_size_change_threshold | Object size ratio change threshold. Slider min: 0.1, max: 2.0, step: 0.2. Conditional on processing_mode being objects . | slider | 0.1 | No |
max_lookups_per_obj | Max. attempts per object. Conditional on processing_mode being objects . | number | 5 | No |
gemini_api_key | Gemini API Key | string | null | Yes |
prompt | Gemini Prompt | text | null | Yes |
detail_level | Image Detail Level. Options: Low, High. | enum | low | No |
max_tokens | Maximum number of tokens to return for each request. | number | 500 | No |
post_processing | Gemini Results Post Processing. Options: Text, JSON. | enum | text | No |
post_proc_attribs_path | JSON path to extract object/ROI attributes from Gemini response. ex. object_color . Conditional on post_processing being json . | string | null | No |
display_roi | Display ROI on video? | bool | true | No |
display_objinfo | Display results on video? Options: Disabled, Bottom left, Bottom right, Top left, Top right. | enum | bottom_left | No |
debug | Log debugging information? | bool | false | No |
Metadata
Metadata Property | Description |
---|---|
nodes.[node_id].rois.[roi_id].label_changed_delta | Indicates if there has been a change in the label of the ROI. |
nodes.[node_id].rois.[roi_id].label_available | Indicates if a label is available for the ROI. |
nodes.[node_id].rois.[roi_id].label | Label contents for the ROI. String if post_processing property is text , else JSON object. |
nodes.[node_id].recognized_obj_count | The count of recognized objects. |
nodes.[node_id].recognized_obj_delta | The change in the count of recognized objects. |
nodes.[node_id].label_changed_obj_delta | The change in the count of objects with changed labels. |
nodes.[node_id].unrecognized_obj_count | The count of unrecognized objects. |
nodes.[node_id].unrecognized_obj_delta | The change in the count of unrecognized objects. |
Example JSON
{
"nodes": {
"gemini1": {
"type": "gemini",
"rois": {
"roi2": {
"label_changed_delta": true,
"label_available": true,
"label": {
"door_status": "unblocked"
}
}
},
"recognized_obj_ids": ["2775161862"],
"recognized_obj_count": 1,
"recognized_obj_delta": 1,
"label_changed_obj_delta": 1,
"unrecognized_obj_count": 0,
"unrecognized_obj_delta": 0,
"objects_of_interest_keys": ["recognized_obj_ids"]
}
},
"objects": [{
"id": 2775161862,
"source_node_id": null,
"model_id": null,
"label": "roi2",
"class_id": 10600,
"rect": {
"left": 128,
"top": 72,
"width": 512,
"height": 575
},
"probability": 1.0,
"attributes": [{
"label": "unblocked",
"class_id": 10602,
"probability": 1.0
}, {
"label": "lvm_results",
"class_id": 10601,
"probability": 1.0
}, {
"label": "lvm_roi",
"class_id": 10600,
"probability": 1.0
}],
"corr_id": "75f5141e-020a-4f27-af26-cf17b32c2544"
}]
}
Updated 10 days ago