Overview

Bulk processing runs one pipeline over many clips — for video search and indexing, backfilling analytics, generating alerts from archived footage, or benchmarking a model across a test set.

There are three steps:

1. Build and test the pipeline that processes a single clip
2. Set up gateways with enough capacity to process them
3. Queue the clips, using the lumeo-bulk-deploy script, the deployment_queues API, or Universal Bridge

Which feed mechanism to use:

1. Build and Test Your Pipeline

Any pipeline can process a clip, so this is no different from building a regular pipeline — see the Getting Started Guide.

👍

Test on one clip before running a batch

Queue a single file, confirm the output is what you expect, then run the full set. A pipeline problem discovered on clip 400 of 500 costs you the entire run.

2. Set Up Gateways

The preferred way to process a large number of clips is to run Lumeo Gateways in the cloud as a Kubernetes cluster, so you can scale capacity up and down to meet demand. You can also request Lumeo-managed Cloud Gateways in your account instead of running your own. Any gateway in your workspace will work if you are just experimenting.

To set up gateways as a Kubernetes cluster, follow the Gateway Setup Guide or the GCP - Kubernetes guide.

📘

Recommendation: use a separate workspace for clip processing

The deployment queue assigns work to the first available gateway in the workspace — edge or cloud, regardless of what else it is doing. A dedicated workspace keeps a large backfill from landing on a gateway that is running live cameras.

Throughput is controlled per gateway by the Max deployments property in gateway settings. That is the knob to turn: the queue itself does not rate limit, it fills whatever capacity exists.

3. Process Clips

Using the lumeo-bulk-deploy script

The script uploads or references your media, optionally associates it with a virtual camera, and queues it for processing.

Install

pipx install "lumeo[scripts]"

⚠️

Include the scripts extra

The lumeo package ships with no base dependencies, so a plain pip install lumeo produces commands that fail on startup. Use pip install "lumeo[scripts]" or pipx install "lumeo[scripts]". Requires Python 3.8+, and ffmpeg on your PATH for large-file splitting.

A first run

lumeo-bulk-deploy \
  --app_id 'd413586b-0ccb-4aaa-9fdf-3df7404f716d' \
  --token 'xxxxxxx' \
  --pipeline_id 'ee55c234-b3d5-405f-b904-cfb2bd6f2e06' \
  --pattern '/Users/username/media/lumeo-*.mp4' \
  --tag 'bulk-uploads/2026-07-25'

Every run needs three things: a source, a target (a pipeline or a camera), and your workspace credentials.

👍

Always tag your run

--tag labels the uploaded files and streams, and --deployment_prefix labels the resulting deployments. Without them, finding the output of a 500-clip run means guessing at timestamps.

Check progress at any time:

lumeo-bulk-deploy --app_id '<app-id>' --token '<token>' --queue_size

Choosing a source

Specify exactly one source. If you pass several, the script picks one silently, in this order: --csv_file, --file_list, --pattern, --s3_bucket, --tag.

Local files matching a pattern — uploads each file to Lumeo:

lumeo-bulk-deploy --app_id '<app-id>' --token '<token>' \
  --pipeline_id '<pipeline-id>' \
  --pattern '/Users/username/media/lumeo-*.mp4'

A list of URLs — creates streams that point at your URLs, with no upload, so nothing counts toward your Lumeo storage:

lumeo-bulk-deploy --app_id '<app-id>' --token '<token>' \
  --pipeline_id '<pipeline-id>' \
  --file_list 'https://assets.lumeo.com/media/parking_lot/mall-parking-1.mp4,https://assets.lumeo.com/media/sample/sample-people-car-traffic.mp4'

A CSV manifest — when different clips need different pipelines or configs:

lumeo-bulk-deploy --app_id '<app-id>' --token '<token>' --csv_file ./manifest.csv

The CSV has fixed, positional columns and requires a header row (the first line is always skipped):

file_uri,camera_external_id,camera_id,pipeline_id,deployment_config
/Users/username/warehouse2.mp4,,,ee55c234-b3d5-405f-b904-cfb2bd6f2e06,
https://assets.lumeo.com/media/parking_lot/mall-parking-1.mp4,,,ee55c234-b3d5-405f-b904-cfb2bd6f2e06,
https://storage.googleapis.com/lumeo-public-media/demos/warehouse5.mp4,,,ee55c234-b3d5-405f-b904-cfb2bd6f2e06,"{""overlay_meta2"": {""text"": ""my-test-run"",""show_frame_count"":true}}"

Blank cells fall back to the equivalent command-line argument, so you can put the common pipeline on the command line and override it per row. Embedded JSON follows normal CSV quoting: wrap the field in double quotes and double any quotes inside it.

An S3 bucket — creates streams from signed URLs, so nothing is uploaded to Lumeo:

lumeo-bulk-deploy --app_id '<app-id>' --token '<token>' \
  --pipeline_id '<pipeline-id>' \
  --s3_bucket 'lumeo-test' \
  --s3_access_key_id 'xxxx' --s3_secret_access_key 'xxxx' \
  --s3_endpoint_url 'https://sfo2.digitaloceanspaces.com' \
  --s3_prefix 'incoming/2026-07' \
  --tag 's3-uploads/2026-07-25'

Specify either --s3_region (AWS) or --s3_endpoint_url (S3-compatible storage such as DigitalOcean Spaces, MinIO or Cloudflare R2). --s3_prefix narrows the run to one path within the bucket.

Files already in Lumeo — reprocess everything carrying a tag, for example to re-run a new model over a saved test set:

lumeo-bulk-deploy --app_id '<app-id>' --token '<token>' \
  --pipeline_id '<pipeline-id>' \
  --tag 'test-bench/person-model-testing'

When --tag is the only source it selects files rather than applying a tag. Use it with --pipeline_id; it cannot be combined with a camera.

Overriding pipeline configuration

--deployment_config passes node overrides as JSON. The video source is always replaced by the clip being processed:

lumeo-bulk-deploy --app_id '<app-id>' --token '<token>' \
  --pipeline_id '<pipeline-id>' \
  --pattern '/Users/username/media/lumeo-*.mp4' \
  --deployment_config '{"overlay_meta2": {"text": "my-test-run","show_frame_count":true}}'

Using a virtual camera instead of a pipeline

--camera_id or --camera_external_id associates the clips with a camera and takes the pipeline and configuration from that camera. --camera_external_id creates the virtual camera if it does not exist, which is useful when you are grouping clips by a site or device id from your own system.

lumeo-bulk-deploy --app_id '<app-id>' --token '<token>' \
  --camera_external_id 'store-114-entrance' \
  --pattern '/Users/username/media/store114/*.mp4'

⚠️

A camera overrides your pipeline arguments

When you specify a camera, --pipeline_id and --deployment_config are ignored entirely. If the camera has no pipeline configured, the clips upload but nothing is queued — so configure the camera first.

Useful options

🚧

--delete_processed is not an on/off flag

It takes a value, and any value turns it on — including --delete_processed false. Omit it entirely unless you want local files deleted after successful processing. For split files it removes the chunks, not your original.

🚧

Keep your token out of shell history

The script accepts credentials only as command-line arguments, so tokens are recorded in your shell history and visible in process listings. Use a scoped token, and prefix the command with a space if your shell is configured to skip such lines.

Version note

--s3_bucket, --tag used on its own, and the live-stream options below require a release newer than 0.1.36. On 0.1.36 they exit immediately with an error. Check your version with pip show lumeo, and upgrade with pipx upgrade lumeo or pip install -U "lumeo[scripts]".

Processing live streams

The same script can deploy a pipeline against live RTSP streams instead of files. Because live streams run continuously, these deployments are created directly on a gateway you name rather than queued:

lumeo-bulk-deploy --app_id '<app-id>' --token '<token>' \
  --pipeline_id '<pipeline-id>' \
  --gateway_id '<gateway-id>' \
  --stream_urls 'rtsp://camera-1/stream,rtsp://camera-2/stream'

Use --stream_id for one existing stream, or --stream_tag to deploy against every RTSP stream carrying a tag. A --pipeline_id is required — cameras are not consulted in this mode, and file options are ignored.

Using the API

Each workspace has a default deployment queue that creates deployments from queued entries first-in-first-out, on the least utilized gateway. See the deployment_queues API reference for full details.

GET  /v1/apps/{application_id}/deployment_queues
POST /v1/apps/{application_id}/deployment_queues/{queue_id}/entries
GET  /v1/apps/{application_id}/deployment_queues/{queue_id}/entries

Post one entry per clip with a pipeline_id, deployment_name and deployment_configuration. To pin work to a specific gateway and start it immediately, bypass the queue and create the deployment directly with POST /v1/apps/{application_id}/deployments, passing gateway_id and "state": "running".

Pipelines that generate alerts do so as each deployment runs. Clips and images produced by the Save Clip Node and Save Snapshot Node are available through the files API, or in your own S3 bucket if the pipeline is configured to write there.

Using Universal Bridge

Universal Bridge lets you upload clips to Lumeo's cloud over SMTP, FTP, or a script. Each upload creates a virtual camera, which you configure in the Console with a pipeline and camera-specific overrides; that pipeline is then deployed for every new clip uploaded to it.

This is the right choice when clips arrive continuously rather than as a one-off batch.

Learn more: Universal Bridge

Billing and Performance Considerations

Clips uploaded to Lumeo's cloud, and any media generated by your pipeline, count toward your Lumeo cloud storage usage. For large backfills, referencing files by URL (--file_list or a CSV of URLs) avoids storing the source clips, since Lumeo streams them from your origin — pipeline outputs are still stored.

Processing throughput is bounded by total gateway capacity and each gateway's Max deployments setting, not by the queue.

Appendix: all lumeo-bulk-deploy options

Run lumeo-bulk-deploy --help for the authoritative list from your installed version. The lumeo package also ships lumeo-bulk-delete, lumeo-media-download, lumeo-model-management, lumeo-load-test, camera importers for Avigilon, Verkada, Hanwha Wave and HikCentral, and lumeo-rhombus-copy-footage. Run lumeo-scripts to list them all.