Overview

The Anthropic Claude Lookup node is designed to perform lookups using the powerful Claude model on objects within a Region of Interest (ROI), or on a ROI within the frame. This functionality is useful for applications requiring advanced recognition and understanding of objects or areas within a video feed.

Inputs & Outputs

Inputs: 1, Media Format: Raw Video
Outputs: 1, Media Format: Raw Video
Output Metadata: nodes.node_id, recognized_objs, recognized_obj_ids, recognized_obj_count, recognized_obj_delta, value_changed_delta, unrecognized_obj_count, unrecognized_obj_delta

Properties

Property	Description	Type	Default	Required
`roi_labels`	Regions of interest labels	hidden		Yes
`rois`	Regions of interest. Type: polygon. Default: null. Conditional on `roi_labels`.	polygon	null	Yes
`processing_mode`	Processing mode. Options: ROIs, at Interval (`rois_interval`), ROIs, upon Trigger (`rois_trigger`), Objects in an ROI (`objects`).	enum	`rois_interval`	Yes
`interval`	Lookup interval. Collect objects or ROIs for lookup at least this many seconds apart.	float	1	No
`batch_size`	Number of images to process in each Claude request. Set to more than 1 to use prompts that reference multiple images when `processing_mode` is `rois_interval` or `rois_trigger`.	number	1	No
`trigger`	Queue ROI for lookup when this condition evaluates to true. Conditional on `processing_mode` being `rois_trigger`.	trigger-condition	null	No
`enable_prebuffer`	If true, samples ROI at interval to fill batch, and performs lookup when trigger is met. Else performs lookup when batch is full. Conditional on `processing_mode` being `rois_trigger`.	bool	false	No
`objects_to_process`	Object types to process. ex. `car`,`person`,`car.red`. Conditional on `processing_mode` being `objects`.	model-label	null	No
`min_obj_size_pixels`	Min. width and height of an object. Conditional on `processing_mode` being `objects`.	number	64	No
`obj_lookup_size_change_threshold`	Object size ratio change threshold. Slider min: 0.1, max: 2.0, step: 0.2. Conditional on `processing_mode` being `objects`.	slider	0.1	No
`max_lookups_per_obj`	Max. attempts per object. Conditional on `processing_mode` being `objects`.	number	5	No
`anthropic_api_key`	Anthropic API Key. Get it here.	string	null	Yes
`prompt`	Claude Prompt	text	null	Yes
`detail_level`	Claude Detail Level. Options: Low, High.	enum	low	No
`max_tokens`	Maximum number of tokens to return for each request.	number	500	No
`post_processing`	Claude Results Post Processing. Options: Text, JSON.	enum	text	No
`post_proc_attribs_path`	JSON path to extract object/ROI attributes from Claude response. ex. `object_color`. Conditional on `post_processing` being `json`.	string	null	No
`display_roi`	Display ROI on video?	bool	true	No
`display_objinfo`	Display results on video? Options: Disabled, Bottom left, Bottom right, Top left, Top right.	enum	bottom_left	No
`debug`	Log debugging information?	bool	false	No

Prompt Examples

Generate scene description. Use with post_processing set to text.

Analyze the scene and provide a concise description of any unique, interesting, or noteworthy elements that would be suitable for a push notification alert. Focus on key details that capture the essence of what's happening or what's important in the image.

Generate structured output that can be added as ROI or object attributes.
Use with post_processing set to json, and remember to set post_proc_attribs_path to the attribute you want to add (ex. intersection_state).

Analyze these images from a traffic intersection. Return a JSON object with the following attributes:
- scene_description: A concise description of the scene.
- intersection_state: "jammed", "flowing", "empty"

Metadata

Metadata Property	Description
`nodes.[node_id].rois.[roi_id].label_changed_delta`	Indicates if there has been a change in the label of the ROI.
`nodes.[node_id].rois.[roi_id].label_available`	Indicates if a label is available for the ROI.
`nodes.[node_id].rois.[roi_id].label`	Label contents for the ROI. String if `post_processing` property is `text`, else JSON object.
`nodes.[node_id].recognized_obj_count`	The count of recognized objects.
`nodes.[node_id].recognized_obj_delta`	The change in the count of recognized objects.
`nodes.[node_id].label_changed_obj_delta`	The change in the count of objects with changed labels.
`nodes.[node_id].unrecognized_obj_count`	The count of unrecognized objects.
`nodes.[node_id].unrecognized_obj_delta`	The change in the count of unrecognized objects.

Example JSON

{
    "nodes": {
        "claude1": {
            "type": "claude",
            "rois": {
                "roi2": {
                    "label_changed_delta": true,
                    "label_available": true,
                    "label": {
                        "door_status": "unblocked"
                    }
                }
            },
            "recognized_obj_ids": ["2775161862"],
            "recognized_obj_count": 1,
            "recognized_obj_delta": 1,
            "label_changed_obj_delta": 1,
            "unrecognized_obj_count": 0,
            "unrecognized_obj_delta": 0,
            "objects_of_interest_keys": ["recognized_obj_ids"]
        }
    },
    "objects": [{
        "id": 2775161862,
        "source_node_id": null,
        "model_id": null,
        "label": "roi2",
        "class_id": 10600,
        "rect": {
            "left": 128,
            "top": 72,
            "width": 512,
            "height": 575
        },
        "probability": 1.0,
        "attributes": [{
            "label": "unblocked",
            "class_id": 10602,
            "probability": 1.0
        }, {
            "label": "lvm_results",
            "class_id": 10601,
            "probability": 1.0
        }, {
            "label": "lvm_roi",
            "class_id": 10600,
            "probability": 1.0
        }],
        "corr_id": "75f5141e-020a-4f27-af26-cf17b32c2544"
    }]
}