Overview

This node is publishes pipeline metadata to Elasticsearch periodically or when a trigger condition is met. You can use it for reporting and dashboards.

Inputs & Outputs

Inputs : 1, Media Format : Raw Video
Outputs : 1, Media Format: Raw Video
Output Metadata : None

Pipeline usage

Place this node after the analytics whose metadata should be indexed. Limit node_ids_to_publish and avoid snapshots unless the use case needs them, because full frame metadata and base64 images can increase index size quickly.

Custom Metrics

Custom Function nodes can publish custom metrics and metadata for specific objects by storing them under its node metadata. Refer to Custom Function for how to do so.

Properties

Property	Description	Type	Default	Required
`trigger_type`	Collect metadata. Options: Custom trigger (`custom_trigger`); Auto trigger (`auto_trigger`): When significant node properties change; Interval (`periodic`); Scheduled (`scheduled`).	`enum`	`auto_trigger`	Yes
`trigger`	Publish when this condition evaluates to true. Conditional on `trigger_type` being `custom_trigger`.	trigger-condition	`null`	No
`interval`	Set to 0 to publish every time it is collected. Unit: seconds.	`float`	`5`	No
`schedule_publishing`	If set, ignore the min. interval and schedule publishing at a synchronized time. Options: Disabled (`null`); Daily (`day`); Hourly (`hour`); Every 30 minutes (`30min`); Every 15 minutes (`15min`). Conditional on `trigger_type` being `scheduled`.	`enum`	`null`	No
`node_ids_to_publish`	Publish metadata only for the specified node IDs. If not specified, publish metadata from all upstream nodes.	`string`	`null`	No
`snapshot_type`	Send snapshots? Options: None (`null`); Small (320px) (`small`): Send snapshot as base64-encoded JPEG to Elasticsearch. Significantly incrases Elasticsearch storage consumption; Medium (640px) (`medium`): Send snapshot as base64-encoded JPEG to Elasticsearch. Significantly incrases Elasticsearch storage consumption.	`enum`	`null`	No
`es_uri`	URI for your Elasticsearch Server. ex. `https://lumeo1.elastic-cloud.com:9243` or `https://user:[email protected]:9243`	`string`	`null`	Yes
`es_verify_certs`	Turn off to allow self-signed certificates (Warning: insecure).	`bool`	`true`	No
`es_api_key`	Base10-encoded API key for your Elasticsearch server. Leave blank if credentials are specified in the URI.	`string`	`null`	No
`es_index`	Name of the index to which metadata will be published. Must be all-lowercase.	`string`	`lumeo`	Yes
`es_max_retries`	Number of times to retry publishing if connection to Elasticsearch server fails, before giving up and discarding the data.	`number`	`3`	Yes
`use_stream_clock`	Use timestamps from stream?	`bool`	`false`	No

Output Metadata

Node does not insert any output metadata; upstream metadata continues through the pipeline

Elasticsearch Configuration

Viewing Snapshots in Kibana

Head to Stack Management -> Index Patterns -> lumeo-<application_id> and search for snapshot field. This field will appear once atleast one shapshot has been sent to ES from Lumeo.

Click "Edit Field" (pencil icon on the right end of the field name)
Enable "Set Format" and configure it with the following settings:
- Format: Url
- Type: Image
- URL Template: data:image/jpg;base64,{{rawValue}}
- Label Template: Image

Setup a new Elastic offline instance

Follow this guide to setup a new single-node Elastic search + Kibana instance to use as an offline data warehouse with Lumeo.

Prerequisites for machine:

Ubuntu 24
Atleast 16GB RAM, 4 CPU cores (increase if larger data size)
Atleast 128GB disk, more if you plan to store more data.

1. Adjust kernel settings on Linux

The vm.max_map_count kernel setting must be set to at least 262144

How you set vm.max_map_count depends on your platform.
Open the ‘/etc/sysctl.conf’ file in a text editor with root privileges. You can use the following command

sudo nano /etc/sysctl.conf

Navigate to the end of the file or search for the line containing vm.max_map_count, If the line exists, modify it to set the desired value

vm.max_map_count=262144

If the line doesn’t exist, add it at the end of the file. Save the file and exit the text editor. Apply the changes by running the following command

sudo sysctl -p

This command reloads the sysctl settings from the configuration file. Now, the value of vm.max_map_count should be updated to 262144.

2. Prepare Environment Variables

Create or navigate to an empty directory for the project.

Inside this directory, create a .env file and set up the necessary environment variables.

Copy the following content and paste it into the .env file.

# Password for the 'elastic' user (at least 6 characters, no special characters)
ELASTIC_PASSWORD=lumeovision

# Password for the 'kibana_system' user (at least 6 characters, no special characters)
KIBANA_PASSWORD=lumeovision

# Lumeo Elasticsearch Configuration. Deletes data after 180 days. Adjust retention duration as needed, and set to 0 to disable deletes.
LUMEO_INDEX_PREFIX=lumeo
LUMEO_ILM_DELETE_DAYS=180

# Version of Elastic products
STACK_VERSION=9.2.4

# Set the cluster name
CLUSTER_NAME=docker-cluster

# Set to 'basic' or 'trial' to automatically start the 30-day trial of advanced features
LICENSE=basic

# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200

# Port to expose Kibana to the host
KIBANA_PORT=443

# Increase or decrease based on the available host memory (in bytes)
MEM_LIMIT=2147483648

In the .env file, specify a password for the ELASTIC_PASSWORD and KIBANA_PASSWORD variables.

The passwords must be alphanumeric and can’t contain special characters, such as ! or @. The bash script included in the compose.yml file only works with alphanumeric characters.

3. Create Docker Compose Configuration

Now, create a compose.yml file in the same directory and copy the following content and paste it into the compose.yml file.

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - ./api-keys:/usr/share/elasticsearch/api-keys
    user: "0"
    command: >
      bash -c '
        if [ x$${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x$${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:$${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"$${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        
        echo "Setting up Lumeo indices and ILM policies";
        INDEX_PREFIX="${LUMEO_INDEX_PREFIX:-lumeo}";
        ILM_DELETE_DAYS="${LUMEO_ILM_DELETE_DAYS:-30}";
        
        if [ "$$ILM_DELETE_DAYS" -gt 0 ]; then
          echo "Creating ILM policy: $${INDEX_PREFIX}-ilm-policy";
          curl -s -X PUT --cacert config/certs/ca/ca.crt \
            -u "elastic:$${ELASTIC_PASSWORD}" \
            -H "Content-Type: application/json" \
            https://es01:9200/_ilm/policy/$${INDEX_PREFIX}-ilm-policy \
            -d "{
              \"policy\": {
                \"phases\": {
                  \"hot\": {
                    \"actions\": {
                      \"rollover\": {
                        \"max_age\": \"1d\"
                      }
                    }
                  },
                  \"delete\": {
                    \"min_age\": \"$${ILM_DELETE_DAYS}d\",
                    \"actions\": {
                      \"delete\": {}
                    }
                  }
                }
              }
            }";
          
          echo "Creating index template: $${INDEX_PREFIX}-template";
          curl -s -X PUT --cacert config/certs/ca/ca.crt \
            -u "elastic:$${ELASTIC_PASSWORD}" \
            -H "Content-Type: application/json" \
            https://es01:9200/_index_template/$${INDEX_PREFIX}-template \
            -d "{
              \"index_patterns\": [\"$${INDEX_PREFIX}-*\"],
              \"template\": {
                \"settings\": {
                  \"index.lifecycle.name\": \"$${INDEX_PREFIX}-ilm-policy\",
                  \"index.lifecycle.rollover_alias\": \"$${INDEX_PREFIX}\"
                },
                \"mappings\": {
                  \"properties\": {
                    \"snapshot\": {
                      \"type\": \"text\",
                      \"index\": false,
                      \"store\": false,
                      \"norms\": false,
                      \"fielddata\": false,
                      \"eager_global_ordinals\": false,
                      \"index_phrases\": false
                    }
                  }
                }
              }
            }";
        fi;
        
        echo "Creating initial index: $${INDEX_PREFIX}-000001";
        curl -s -X PUT --cacert config/certs/ca/ca.crt \
          -u "elastic:$${ELASTIC_PASSWORD}" \
          -H "Content-Type: application/json" \
          https://es01:9200/$${INDEX_PREFIX}-000001 \
          -d "{
            \"aliases\": {
              \"$${INDEX_PREFIX}\": {
                \"is_write_index\": true
              }
            },
            \"mappings\": {
              \"properties\": {
                \"snapshot\": {
                  \"type\": \"text\",
                  \"index\": false,
                  \"store\": false,
                  \"norms\": false,
                  \"fielddata\": false,
                  \"eager_global_ordinals\": false,
                  \"index_phrases\": false
                }
              }
            }
          }";
        
        echo "Creating API key for $${INDEX_PREFIX}";
        API_KEY_RESPONSE=$$(curl -s -X POST --cacert config/certs/ca/ca.crt \
          -u "elastic:$${ELASTIC_PASSWORD}" \
          -H "Content-Type: application/json" \
          https://es01:9200/_security/api_key \
          -d "{
            \"name\": \"$${INDEX_PREFIX}-writer-key\",
            \"role_descriptors\": {
              \"$${INDEX_PREFIX}_writer\": {
                \"cluster\": [\"monitor\"],
                \"indices\": [{
                  \"names\": [\"$${INDEX_PREFIX}\", \"$${INDEX_PREFIX}-*\"],
                  \"privileges\": [\"write\", \"create_index\", \"auto_configure\", \"view_index_metadata\"]
                }]
              }
            }
          }");
        
        echo "$$API_KEY_RESPONSE" | grep -q "\"id\"";
        if [ $$? -eq 0 ]; then
          API_KEY_ID=$$(echo "$$API_KEY_RESPONSE" | grep -o "\"id\":\"[^\"]*\"" | cut -d\" -f4);
          API_KEY=$$(echo "$$API_KEY_RESPONSE" | grep -o "\"api_key\":\"[^\"]*\"" | cut -d\" -f4);
          ENCODED=$$(echo "$$API_KEY_RESPONSE" | grep -o "\"encoded\":\"[^\"]*\"" | cut -d\" -f4);
          
          echo "============================================================";
          echo "API KEY CREATED";
          echo "============================================================";
          echo "ID: $$API_KEY_ID";
          echo "API Key: $$API_KEY";
          echo "Encoded: $$ENCODED";
          echo "============================================================";
          
          mkdir -p /usr/share/elasticsearch/api-keys;
          echo "LUMEO_ES_API_KEY=$$ENCODED" > /usr/share/elasticsearch/api-keys/$${INDEX_PREFIX}-api-key.env;
          echo "LUMEO_ES_INDEX=$$INDEX_PREFIX" >> /usr/share/elasticsearch/api-keys/$${INDEX_PREFIX}-api-key.env;
          echo "";
          echo "API key saved to: ./api-keys/$${INDEX_PREFIX}-api-key.env";
          echo "Add this to your application environment:";
          echo "  LUMEO_ES_API_KEY=$$ENCODED";
          echo "  LUMEO_ES_INDEX=$$INDEX_PREFIX";
        else
          echo "Failed to create API key";
          echo "$$API_KEY_RESPONSE";
        fi;
        
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120
    environment:
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - KIBANA_PASSWORD=${KIBANA_PASSWORD}
      - LUMEO_INDEX_PREFIX=${LUMEO_INDEX_PREFIX:-lumeo}
      - LUMEO_ILM_DELETE_DAYS=${LUMEO_ILM_DELETE_DAYS:-0}

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - SERVER_PUBLICBASEURL=http://localhost:5601
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata:
    driver: local
  kibanadata:
    driver: local

4. Start Docker Compose

Now you can start Elasticsearch and Kibana using Docker Compose. Run the following command from your project directory

docker compose up -d

Check status:

docker compose ps -a

5. Retrieve Elastic API key

The init container will save API key for you to use in the Publish to Elasticsearch node settings.

This key can be found in api-keys/lumeo-api-key.env file after the init container finishes it's work (typically a few minutes)

6. Access Elasticsearch and Kibana

Once Docker Compose has started the services, you can access Elasticsearch at https://localhost:9200 and Kibana at https://localhost:443 in your web browser.