Skip to main content

OSS Temporal Service metrics reference

OSS Temporal Service metrics

The information on this page is relevant to open source Temporal Service deployments.

See Cloud metrics for metrics emitted by Temporal Cloud.

See SDK metrics for metrics emitted by the SDKs.

A Temporal Service emits a range of metrics to help operators get visibility into the Temporal Service's performance and to set up alerts.

All metrics emitted by the Temporal Service are listed in metric_defs.go.

For details on setting up metrics in your Temporal Service configuration, see the Temporal Service configuration reference.

The dashboards repository contains community-driven Grafana dashboard templates that can be used as a starting point for monitoring the Temporal Service and SDK metrics. You can use these templates as references to build your own dashboards. For any metrics that are missing in the dashboards, use metric_defs.go as a reference.

Note that, apart from these metrics emitted by the Temporal Service, you should also monitor infrastructure-specific metrics like CPU, memory, and network for all hosts that are running Temporal Service services.

Common metrics​

Temporal emits metrics for each gRPC service request. These metrics are emitted with type, operation, and namespace tags, which provide visibility into Service usage and show the request rates across Services, Namespaces, and Operations.

  • Use the operation tag in your query to get request rates, error rates, or latencies per operation.
  • Use the service_name tag with the service role tag values to get details for the specific service.

All common tags that you can add in your query are defined in the metric_defs.go file.

For example, to see service requests by operation on the Frontend Service, use the following:

sum by (operation) (rate(service_requests{service_name="frontend"}[2m]))

Note: All metrics queries in this topic are Prometheus queries.

The following list describes some metrics you can get started with.

service_requests​

Shows service requests received per Task Queue. Example: Service requests by operation sum(rate(service_requests{operation=\"AddWorkflowTask\"}[2m]))

service_latency​

Shows latencies for all Client request operations. Usually these are the starting point to investigate which operation is experiencing high-latency issues. Example: P95 service latency by operation for the Frontend Service histogram_quantile(0.95, sum(rate(service_latency_bucket{service_name="frontend"}[5m])) by (operation, le))

service_error_with_type​

(Available only in v1.17.0+) Identifies errors encountered by the service. Example: Service errors by type for the Frontend Service sum(rate(service_errors_with_type{service_name="frontend"}[5m])) by (error_type)

client_errors​

An indicator for connection issues between different Server roles. Example: Client errors sum(rate(client_errors{service_name="frontend",service_role="history"}[5m]))

In addition to these, you can define some service-specific metrics to get performance details for each service. Start with the following list, and use metric_defs.go to define additional metrics as required.

Matching Service metrics​

poll_success​

Shows for Tasks that are successfully matched to a poller. Example: sum(rate(poll_success{}[5m]))

poll_timeouts​

Shows when no Tasks are available for the poller within the poll timeout. Example: sum(rate(poll_timeouts{}[5m]))

asyncmatch_latency​

Measures the time from creation to delivery for async matched Tasks. The larger this latency, the longer Tasks are sitting in the queue waiting for your Workers to pick them up. Example: histogram_quantile(0.95, sum(rate(asyncmatch_latency_bucket{service_name=~"matching"}[5m])) by (operation, le))

no_poller_tasks​

Emitted whenever a task is added to a task queue that has no poller, and is a counter metric. This is usually an indicator that either the Worker or the starter programs are using the wrong Task Queue. Use no_poller_tasks_per_tl to get data per Task Queue.

History Service metrics​

A History Task is an internal Task in Temporal that is created as part of a transaction to update Workflow state and is processed by the Temporal History service. It is critical to ensure that the History Task processing system is healthy. The following key metrics can be used to monitor the History Service health:

task_requests​

Emitted on every Task process request. Example: sum(rate(task_requests{service="$service",operation=~"TransferActive.*"}[1m]))

task_errors​

Emitted on every Task process error. Example: sum(rate(task_errors{operation=~"TransferActive.*"}[1m]))

task_attempt​

Number of attempts on each Task Execution. A Task is retried forever, and each retry increases the attempt count. Example: histogram_quantile($percentile, sum(rate(task_attempt_bucket{service="$service",operation=~"TransferActive.*"}[1m])) by (operation, le))

task_latency_processing​

Shows the processing latency per attempt. Example: histogram_quantile($percentile, sum(rate(task_latency_processing_bucket{operation=~"TransferActive.*",service="$service", service_name="history"}[1m])) by (operation, le))

task_latency​

Measures the in-memory latency across multiple attempts.

task_latency_queue​

Measures the duration, end-to-end, from when the Task should be executed (from the time it was fired) to when the Task is done.

task_latency_load​

(Available only in v1.18.0+) Measures the duration from Task generation to Task loading (Task schedule to start latency for persistence queue).

task_latency_schedule​

(Available only in v1.18.0+) Measures the duration from Task submission (to the Task scheduler) to processing (Task schedule to start latency for in-memory queue).

queue_latency_schedule​

(Available only in v1.18.0+) Measures the time to schedule 100 Tasks in one Task channel in the host-level Task scheduler. If fewer than 100 Tasks are in the Task channel for 30 seconds, the latency is scaled to 100 Tasks upon emission. Note: This is still an experimental metric and is subject to change.

task_latency_userlatency​

Shows the latency introduced because of Workflow logic. For example, if you have one Workflow scheduling many Activities or Child Workflows at the same time, it can cause a per-Workflow lock contention. The wait period for the per-Workflow lock is counted as userlatency.

The operation tag contains details about Task type and Active versus Standby statuses, and can be used to get request rates, error rates, or latencies per operation, which can help identify issues caused by database problems.

Persistence metrics​

Temporal Server emits metrics for every persistence database read and write. Some of the most important ones are the following:

persistence_requests​

Emitted on every persistence request. Examples:

  • Prometheus query for getting the total number of persistence requests by operation for the History Service: sum by (operation) (rate(persistence_requests{service="$service",service_name="history"}[1m]))
  • Prometheus query for getting the total number of persistence requests by operation for the Matching Service: sum by (operation) (rate(persistence_requests{cluster="$cluster",service_name="matching"}[5m]))

persistence_errors​

Shows all persistence errors. This metric is a good indicator for connection issues between the Temporal Service and the persistence store. Example:

  • Prometheus query for getting all persistence errors by service (history) sum (rate(persistence_errors{service="$service",service_name="history"}[1m]))

persistence_error_with_type​

Shows all errors related to the persistence store with type, and contain an error_type tag.

  • Prometheus query for getting persistence errors with type by (history) and by error type: sum(rate(persistence_error_with_type{service="$service",service_name="history"}[1m])) by (error_type)

persistence_latency​

Shows the latency on persistence operations. Example:

  • Prometheus query for getting latency by percentile: histogram_quantile($percentile, sum(rate(persistence_latency_bucket{service="$service" service_name="history"}[1m])) by (operation, le))

Schedule metrics​

Temporal emits metrics that track the performance and outcomes of these Scheduled Executions.

Below are additional metrics that can help you monitor and optimize your Scheduled Workflow Executions.

schedule_buffer_overruns​

Indicates instances where the buffer for holding Scheduled Workflows exceeds its maximum capacity. This scenario typically occurs when schedules with a buffer_all overlap policy have their average run length exceeding the average schedule interval.

Example: To monitor buffer overruns.

sum(rate(schedule_buffer_overruns{namespace="$namespace"}[5m]))

schedule_missed_catchup_window​

Tracks occurrences when the system fails to execute a Scheduled Action within the defined catchup window. Missed catchup windows can result from extended outages beyond the configured catchup period.

Example: To identify missed catchup opportunities.

sum(rate(schedule_missed_catchup_window{namespace="$namespace"}[5m]))

schedule_rate_limited​

Reflects instances where the creation of Workflows by a Schedule is throttled due to rate limiting policies within a Namespace. This metric is crucial for identifying scheduling patterns that frequently hit rate limits, potentially causing missed catchup windows.

Example: To assess the impact of rate limiting on Scheduled Executions.

sum(rate(schedule_rate_limited{namespace="$namespace"}[5m]))

schedule_action_success​

Measures the successful execution of Workflows as per their schedules or through manual triggers. This metric is confirms that Workflows are running as expected without delays or errors.

Example: To track the success rate of Scheduled Workflow Executions.

sum(rate(schedule_action_success{namespace="$namespace"}[5m]))

Workflow metrics​

These metrics pertain to Workflow statistics.

workflow_cancel​

Number of Workflows canceled before completing execution.

workflow_continued_as_new​

Number of Workflow Executions that were Continued-As-New from a past execution.

workflow_failed​

Number of Workflows that failed before completion.

workflow_success​

Number of Workflows that successfully completed.

workflow_timeout​

Number of Workflows that timed out before completing execution.

Nexus metrics​

These metrics pertain to Nexus Operations.

Nexus Machinery in the History Service​

See architecture document for more info.

In-Memory Buffer​

dynamic_worker_pool_scheduler_enqueued_tasks: A counter that is incremented when a task is enqueued to the buffer.

dynamic_worker_pool_scheduler_dequeued_tasks: A counter that is incremented when a task is dequeued from the buffer.

dynamic_worker_pool_scheduler_rejected_tasks: A counter that is incremented when the buffer is full and adding the task is rejected.

dynamic_worker_pool_scheduler_buffer_size: A gauge that periodically samples the size of the buffer.

Concurrency Limiter​

dynamic_worker_pool_scheduler_active_workers: A gauge that periodically samples the number of running goroutines.

Rate Limiter​

rate_limited_task_runnable_wait_time: A histogram representing the time a task spends waiting for the rate limiter.

Circuit Breaker​

circuit_breaker_executable_blocked: A counter that is incremented every time a task execution is blocked by the circuit breaker.

Task Executors​

nexus_outbound_requests: A counter representing the number of Nexus outbound requests made by the history service.

nexus_outbound_latency: A histogram representing the latency of outbound Nexus requests made by the history service.

callback_outbound_requests: A counter representing the number of callback outbound requests made by the history service.

callback_outbound_latency: A histogram representing the latency histogram of outbound callback requests made by the history service.

Nexus Machinery on the Frontend Service​

nexus_requests​

The number of Nexus requests received by the service.

Type: Counter

nexus_latency​

Latency of Nexus requests.

Type: Histogram

nexus_request_preprocess_errors​

The number of Nexus requests for which pre-processing failed.

Type: Counter

nexus_completion_requests​

The number of Nexus completion (callback) requests received by the service.

Type: Counter

nexus_completion_latency​

Latency histogram of Nexus completion (callback) requests.

Type: Histogram

nexus_completion_request_preprocess_errors​

The number of Nexus completion requests for which pre-processing failed.

Type: Counter