OpenAI GPT4 Lookup

Perform a lookup with GPT4 on objects in an ROI, or on a ROI in the frame.

Overview

The OpenAI GPT4 Lookup node is designed to perform lookups using the powerful GPT4 model on objects within a Region of Interest (ROI), or on a ROI within the frame. This functionality is useful for applications requiring advanced recognition and understanding of objects or areas within a video feed.

Inputs & Outputs

  • Inputs: 1, Media Format: Raw Video
  • Outputs: 1, Media Format: Raw Video
  • Output Metadata: nodes.node_id, recognized_objs, recognized_obj_ids, recognized_obj_count, recognized_obj_delta, value_changed_delta, unrecognized_obj_count, unrecognized_obj_delta

Properties

PropertyDescriptionTypeDefaultRequired
roi_labelsRegions of interest labelshiddenYes
roisRegions of interest. Type: polygon. Default: null. Conditional on roi_labels.polygonnullYes
processing_modeProcessing mode. Options: ROIs, at Interval (rois_interval), ROIs, upon Trigger (rois_trigger), Objects in an ROI (objects).enumrois_intervalYes
intervalLookup interval. Collect objects or ROIs for lookup at least this many seconds apart.float1No
batch_sizeNumber of images to process in each GPT request. Set to more than 1 to use prompts that reference multiple images.number1No
triggerQueue ROI for lookup when this condition evaluates to true. Conditional on processing_mode being rois_trigger.trigger-conditionnullNo
enable_prebufferIf true, samples ROI at interval to fill batch, and performs lookup when trigger is met. Else performs lookup when batch is full. Conditional on processing_mode being rois_trigger.boolfalseNo
objects_to_processObject types to process. ex. car,person,car.red. Conditional on processing_mode being objects.model-labelnullNo
min_obj_size_pixelsMin. width and height of an object. Conditional on processing_mode being objects.number64No
obj_lookup_size_change_thresholdObject size ratio change threshold. Slider min: 0.1, max: 2.0, step: 0.2. Conditional on processing_mode being objects.slider0.1No
max_lookups_per_objMax. attempts per object. Conditional on processing_mode being objects.number5No
openai_api_keyOpenAI API KeystringnullYes
promptGPT PrompttextnullYes
detail_levelGPT Detail Level. Options: Low, High.enumlowNo
max_tokensMaximum number of tokens to return for each request.number500No
post_processingGPT Results Post Processing. Options: Text, JSON.enumtextNo
post_proc_attribs_pathJSON path to extract object/ROI attributes from GPT response. ex. object_color. Conditional on post_processing being json.stringnullNo
display_roiDisplay ROI on video?booltrueNo
display_objinfoDisplay results on video? Options: Disabled, Bottom left, Bottom right, Top left, Top right.enumbottom_leftNo
debugLog debugging information?boolfalseNo

Metadata

Metadata PropertyDescription
nodes.[node_id].rois.[roi_id].label_changed_deltaIndicates if there has been a change in the label of the ROI.
nodes.[node_id].rois.[roi_id].label_availableIndicates if a label is available for the ROI.
nodes.[node_id].recognized_obj_countThe count of recognized objects.
nodes.[node_id].recognized_obj_deltaThe change in the count of recognized objects.
nodes.[node_id].label_changed_obj_deltaThe change in the count of objects with changed labels.
nodes.[node_id].unrecognized_obj_countThe count of unrecognized objects.
nodes.[node_id].unrecognized_obj_deltaThe change in the count of unrecognized objects.

Example JSON