diff --git a/docs/developer-guide/useful-commands-and-queries.md b/docs/developer-guide/useful-commands-and-queries.md index 5b55f3a..e6c7ded 100644 --- a/docs/developer-guide/useful-commands-and-queries.md +++ b/docs/developer-guide/useful-commands-and-queries.md @@ -41,8 +41,10 @@ docker cp _postgresql_1:/deployment/datapoints.csv ./ ### Backup/Restore OpenRemote DB * Create backup: `docker exec or-postgresql-1 pg_dump -Fc openremote -f /tmp/db.bak` +* Optional: Exclude datapoint records from the backup using the following command: `docker exec or-postgresql-1 pg_dump -Fc openremote -f /tmp/db.bak --exclude-table-data='_timescaledb_internal._hyper_*'` * Copy to the Docker host: `docker cp or-postgresql-1:/tmp/db.bak ~/` -* SCP the backup off the source server onto the destination server +* Remove the backup from within the container: `docker exec or-postgresql-1 rm /tmp/db.bak` +* SCP the backup off the source server onto the destination server: e.g. `scp :~/db.bak .` * On the destination server stop the manager and Keycloak containers and any project specific containers that are using the DB: `docker stop or-manager-1 or-keycloak-1` * Copy backup into the postgresql container: `docker cp db.bak or-postgresql-1:/tmp/` * Drop existing DB: `docker exec or-postgresql-1 dropdb openremote` @@ -139,6 +141,13 @@ mv -v data/old/* $PGDATA rm -r data/new data/old ``` +## Bash +### SSH tunnel for proxy stats +HAProxy stats web page is only accessible on localhost in our default config, this can be tunnelled to your local machine to allow access at http://localhost:8404/stats: +```shell +ssh -L 8404:localhost:8404 +``` + ## Queries ### Get DB table size info diff --git a/docs/user-guide/metrics.md b/docs/user-guide/metrics.md new file mode 100644 index 0000000..de31448 --- /dev/null +++ b/docs/user-guide/metrics.md @@ -0,0 +1,431 @@ +# Metrics + +Prometheus formatted metrics endpoints can be configured for each container (including the OpenRemote Manager), you will either need Prometheus server running to scrape these endpoints or use a cloud provider service; here's an example using AWS Cloudwatch: + +```mermaid +graph LR + %% Styling Definitions + classDef greenStyle fill:#d4edda,stroke:#28a745,stroke-width:2px,color:#000; + classDef innerGreenStyle fill:#e8f5e9,stroke:#28a745,stroke-width:1px,color:#000; + classDef orangeStyle fill:#ffe0b2,stroke:#f57c00,stroke-width:2px,color:#000; + classDef redStyle fill:#ffcdd2,stroke:#c62828,stroke-width:2px,color:#000; + classDef purpleStyle fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px,color:#000; + + subgraph AWS [AWS] + direction TB + CW[Cloudwatch]:::orangeStyle + DB["Dashboard
or-default
- Standard dashboard uses variables to update widgets
- Instance names list generated from list of or_attributes_total metrics
- Metrics are sampled over configurable period
- Sampling configuration can be tricky"]:::purpleStyle + end + + subgraph EC2 [EC2 Instance] + + subgraph CWAgent [Cloudwatch Agent] + direction TB + CWConfig["Cloudwatch Config
/opt/aws/amazon-cloudwatch-agent/var/config.json
- Cloudwatch agent is able to act like Prometheus server
- Regex filters and matchers to select metrics
- Cloudwatch maps metric types to its own"]:::innerGreenStyle + PromScrape["Prometheus Scrape Config
/opt/aws/amazon-cloudwatch-agent/var/prometheus.yaml
- Defines the scrape configs for each container
- Be aware some functionality does not work"]:::innerGreenStyle + end + + subgraph Docker [Docker Containers] + direction TB + Manager["Manager
http://localhost:8404/metrics
- Micrometer with Prometheus Registry
- Runs on own embedded web server port 8404
- OR_METRICS_ENABLED: true/false"]:::greenStyle + HAProxy["HA Proxy
http://localhost:8404/metrics
- Uses prometheus-exporter
- Runs on own embedded web server port 8404
- Configured via haproxy.cfg"]:::greenStyle + Keycloak["Keycloak
http://localhost:8080/metrics
- Built in prometheus metrics support
- KC_METRICS_ENABLED: true/false
- Do not publicly expose"]:::orangeStyle + PostgreSQL["PostgreSQL
- No metrics at present could use postgresql-exporter"]:::redStyle + end + end + + %% Connections + PromScrape --> Manager + PromScrape --> HAProxy + CWAgent --> CW + CW --> DB + +``` + +Refer to the website of each container app for details of metrics exposed and their meaning; here's an overview of the OpenRemote Manager metrics. + +## OpenRemote Manager + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ComponentMetric nameTypeLabelsDescription
                                                                               
Artemisartemis_activegaugebroker: localhostIf the server is active
Artemisartemis_address_memory_usagegaugebroker: localhostMemory used by all the addresses on broker for in-memory messages
Artemisartemis_address_memory_usage_percentagegaugebroker: localhostMemory used by all the addresses on broker as a percentage of the global-max-size
Artemisartemis_address_sizegaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
The number of estimated bytes being used by all the queue(s) bound to this address; used to control paging and blocking
Artemisartemis_authentication_countgaugebroker: localhost
result: failure | success
Number of successful authentication attempts
Artemisartemis_authorization_countgaugebroker: localhost
result: failure | success
Number of successful authorization attempts
Artemisartemis_connection_countgaugebroker: localhostNumber of clients connected to this server
Artemisartemis_consumer_countgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of consumers consuming messages from this queue
Artemisartemis_delivering_durable_message_countgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of durable messages that this queue is currently delivering to its consumers
Artemisartemis_delivering_durable_persistent_sizegaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Persistent size of durable messages that this queue is currently delivering to its consumers
Artemisartemis_delivering_message_countgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of messages that this queue is currently delivering to its consumers
Artemisartemis_delivering_persistent_sizegaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Persistent size of messages that this queue is currently delivering to its consumers
Artemisartemis_disk_store_usagegaugebroker: localhostFraction of total disk store used
Artemisartemis_durable_message_countgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of durable messages currently in this queue (includes scheduled, paged, and in-delivery messages)
Artemisartemis_durable_persistent_sizegaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Persistent size of durable messages currently in this queue (includes scheduled, paged, and in-delivery messages)
Artemisartemis_limit_percentgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
The % of memory limit (global or local) that is in use by this address
Artemisartemis_message_countgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of messages currently in this queue (includes scheduled, paged, and in-delivery messages)
Artemisartemis_messages_acknowledgedgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of messages acknowledged from this queue since it was created
Artemisartemis_messages_addedgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of messages added to this queue since it was created
Artemisartemis_messages_expiredgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of messages expired from this queue since it was created
Artemisartemis_messages_killedgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of messages removed from this queue since it was created due to exceeding the max delivery attempts
Artemisartemis_number_of_pagesgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
Number of pages used by this address
Artemisartemis_persistent_sizegaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Persistent size of all messages (including durable and non-durable) currently in this queue (includes scheduled, paged, and in-delivery messages)
Artemisartemis_replica_syncgaugebroker: localhostIf the initial replication synchronization process is complete
Artemisartemis_routed_message_countgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
Number of messages routed to one or more bindings
Artemisartemis_scheduled_durable_message_countgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of durable scheduled messages in this queue
Artemisartemis_scheduled_durable_persistent_sizegaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Persistent size of durable scheduled messages in this queue
Artemisartemis_scheduled_message_countgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Number of scheduled messages in this queue
Artemisartemis_scheduled_persistent_sizegaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
queue: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
Persistent size of scheduled messages in this queue
Artemisartemis_session_countgaugebroker: localhostNumber of sessions on this server
Artemisartemis_total_connection_countgaugebroker: localhostTotal number of clients which have connected to this server since it was started
Artemisartemis_total_session_countgaugebroker: localhostTotal number of sessions created on this server since it was started
Artemisartemis_unrouted_message_countgaugeaddress: \*.\*.writeattribute.# | \*.\*.writeattributevalue.# | provisioning.\*.request
broker: localhost
Number of messages not routed to any bindings
Executorsexecutor_active_threadsgaugename: ContainerExecutor | ContainerScheduledExecutorThe approximate number of threads that are actively executing tasks
Executorsexecutor_completed_tasks_totalcountername: ContainerExecutor | ContainerScheduledExecutorThe approximate total number of tasks that have completed execution
Executorsexecutor_idle_secondssummaryname: ContainerExecutor | ContainerScheduledExecutorIdle time of executor
Executorsexecutor_idle_seconds_maxgaugename: ContainerExecutor | ContainerScheduledExecutorMaximum idle time of executor
Executorsexecutor_pool_core_threadsgaugename: ContainerExecutor | ContainerScheduledExecutorThe core number of threads for the pool
Executorsexecutor_pool_max_threadsgaugename: ContainerExecutor | ContainerScheduledExecutorThe maximum allowed number of threads in the pool
Executorsexecutor_pool_size_threadsgaugename: ContainerExecutor | ContainerScheduledExecutorThe current number of threads in the pool
Executorsexecutor_queue_remaining_tasksgaugename: ContainerExecutor | ContainerScheduledExecutorThe number of additional elements that this queue can ideally accept without blocking
Executorsexecutor_queued_tasksgaugename: ContainerExecutor | ContainerScheduledExecutorThe approximate number of tasks that are queued for execution
Executorsexecutor_scheduled_once_totalcountername: ContainerExecutorTotal tasks scheduled once
Executorsexecutor_scheduled_repetitively_totalcountername: ContainerScheduledExecutorTotal tasks scheduled repetitively
Executorsexecutor_secondssummaryname: ContainerExecutor | ContainerScheduledExecutorMeasures executor task execution time
Executorsexecutor_seconds_maxgaugename: ContainerExecutor | ContainerScheduledExecutorMaximum execution time of executor tasks
Eventsor_attributes_totalcountersource: AgentService | AttributeLinkingService | EnergyOptimisationService | GatewayService | RulesEngine | none | ...Total attributes processed by source
Eventsor_attributes_secondssummary(none)Total time spent processing attribute events
Eventsor_attributes_seconds_maxgauge(none)Maximum time spent processing an attribute event
Eventsor_provisioning_secondssummary(none)Total time spent processing provisioning requests
Eventsor_provisioning_seconds_maxgauge(none)Maximum time spent processing provisioning requests
Rulesor_rules_secondssummary(none)Total time spent processing rules
Rulesor_rules_seconds_maxgauge(none)Maximum time spent processing rules