Skip to main content
Field-level reference for every telemetry signal Dify Enterprise emits through Data Push. For configuring and operating the push service itself, see Data Push.

Getting Started

You do not need to read every table. Jump directly to the signal layer that matches your observability goal.

Scenario-to-Signal Map

Telemetry Data Flow and Routing

All telemetry signals exported by Dify comply with the OpenTelemetry specification and can be split and routed by a standard OTel Collector:

Transmission Rules and Resource Attributes

Resource Attributes

Global environment metadata attached to every exported OTel signal:

Common Identity and Correlation Attributes

Spans carry the dify.-prefixed form; metric labels and the correlation keys on logs (trace_id, span_id, tenant_id, user_id) use the bare form. Rows listing two names cover both cases. The three apprunner.* attributes apply to workflow Spans (dify.workflow.run), node Spans (dify.node.execution), and their companion logs in deployed environments. They aren’t used as metric labels.

OTel-to-Prometheus Naming Conventions

OTel signal names are automatically converted to underscore format when ingested by Prometheus. Counters gain a _total suffix; Histograms produce _bucket, _sum, and _count series, plus a unit suffix (_seconds for metrics measured in s), as the example below shows. The table below shows common examples; all other metrics follow the same rules.

Null Values and Content-Gated Fields

  • Null value behavior: attributes with no value are omitted from Trace Spans; in structured log JSON output they are explicitly rendered as null.
  • Content-gating: attributes that may hold sensitive text (Input/Output/Query/Prompt) are gated by ENTERPRISE_INCLUDE_CONTENT. When content push is disabled (the default), the value is replaced by a reference string (e.g. ref:{id_type}={uuid}) rather than null.
  • High-cardinality warning: tenant_id, app_id, and trace_id are high-cardinality dimensions. Do not store unbounded text fields as Prometheus labels.

Traces and Spans

Traces present the complete time sequence and nesting topology of a single call. The OTel trace_id is derived deterministically from the business run ID, so every signal of one run shares it; query Jaeger or Tempo by this trace_id. The dify.trace_id attribute keeps the original business UUID (for example the workflow run ID) for joining with business data.

dify.workflow.run

dify.node.execution

dify.node.execution.draft

Carries the same attributes as dify.node.execution, but is produced only during a single-node test or a draft preview run in the Studio canvas. Use this signal to separate production telemetry from test runs.

Metrics

Counters

All counters are cumulative and are never sampled: the trace sampling rate (ENTERPRISE_OTEL_SAMPLING_RATE, see the environment variable reference) does not affect them.

Token Counters

Labels: tenant_id, app_id, operation_type, model_provider, model_name, node_type (for node_execution only)
The workflow-level dify.tokens.total already includes tokens from all node_execution operations. Filter by operation_type to avoid double-counting.
Token hierarchy and query patterns Token metrics are emitted at multiple levels. Understanding the hierarchy prevents double-counting:
Key rule: workflow tokens already include all node_execution tokens. Never add both together.
  • Available labels on token metrics: tenant_id, app_id, operation_type, model_provider, model_name, node_type.
  • Application name is carried on the trace side only, as dify.app.name on the span companion logs (see Span Companion Logs), never as a metric label. Use app_id in metric queries.
  • Cost exists only on the node companion logs (dify.node.total_price, see Span Companion Logs); there is no cost metric.

Request Counter

The type label identifies the operation (for example dify_requests_total{type="workflow"}); additional labels vary by type:

Error Counter

The error counter uses the same type label; additional labels vary by type:

Other Counters

Histograms

Unlike Counters, which can only produce averages (which extreme values can distort), Histograms support calculations for the P50, P95, and P99 percentiles that more accurately describe the actual distribution.Each Histogram in Prometheus expands into three series: _bucket, _sum, and _count. P95/P99 may return NaN in low-traffic time windows; increase the query window if this occurs. When latency is concentrated within 5 seconds, such as for TTFT or tool calls, percentile precision is limited. For higher-precision evaluation, extend the query window or contact the Dify team for bucket configuration details.

Structured Logs

Structured logs provide detailed business data. Logs fall into two categories: Span companion logs and standalone business logs.
ENTERPRISE_OTEL_SAMPLING_RATE only affects the number of Spans sent to Trace backends such as Jaeger and Tempo. Structured logs (span_detail and metric_only) are always written to Pod standard output at 100% and are not controlled by the sampling rate.

Span Companion Logs

Signal type: span_detail. These logs are tightly coupled to a Trace Span (sharing trace_id and span_id) and contain content-gated detailed context.

dify.workflow.run companion log

Common attributes: all Span attributes (see Traces section) plus: Event attributes:
  • dify.event.name: dify.workflow.run
  • dify.event.signal: span_detail
  • trace_id, span_id, tenant_id, user_id

dify.node.execution and dify.node.execution.draft companion logs

Common attributes: all Span attributes (see Traces section) plus: Event attributes:
  • dify.event.name: dify.node.execution or dify.node.execution.draft
  • dify.event.signal: span_detail
  • trace_id, span_id, tenant_id, user_id

Standalone Logs

Logs without a structured Span. Their signal type is metric_only: the event feeds counters and carries its detail in this log, with no companion span.

dify.message.run

dify.tool.execution

dify.moderation.check

dify.suggested_question.generation

dify.dataset.retrieval

dify.generate_name.execution

dify.prompt_generation.execution

dify.app.created

dify.app.updated

dify.app.deleted

dify.feedback.created

dify.telemetry.rehydration_failed

A diagnostic event for telemetry pipeline health, emitted when a buffered telemetry payload cannot be reconstructed (rehydrated) for processing. A sustained rate indicates telemetry data loss.

Content-Gated Attributes

The following fields may contain user inputs, model outputs, document content, tool parameters, or feedback text. Raw values are transmitted only when content push is enabled on the deployment: ENTERPRISE_INCLUDE_CONTENT (default false; see the environment variable reference). When disabled, these attributes are replaced with ref:{id_type}={uuid}. The id_type names the correlating entity (for example workflow_run_id, node_execution_id, or message_id); look up that record in the Dify database to retrieve the original content. dify.node.execution.draft carries the same gated fields as dify.node.execution.
In production, it is recommended to disable content push by default. Enable it in a controlled environment only after confirming data classification, access controls, redaction requirements, retention periods, and audit obligations.

Appendix

Operation Types

workflow, node_execution, message, rule_generate, code_generate, structured_output, instruction_modify These values belong to the operation_type label on token metrics and to dify.prompt_generation.operation_type. The request and error counters use the separate type label; its values are listed under Counters.

Node Types

start, end, answer, llm, knowledge-retrieval, knowledge-index, if-else, code, template-transform, question-classifier, http-request, tool, datasource, variable-aggregator, loop, iteration, parameter-extractor, assigner, document-extractor, list-operator, agent, trigger-webhook, trigger-schedule, trigger-plugin, human-input

Workflow Statuses

running, succeeded, failed, stopped, partial-succeeded, paused

Payload Types

workflow, node, message, tool, moderation, suggested_question, dataset_retrieval, generate_name, prompt_generation, app, feedback

Invoke From Values

  • Workflow spans (dify.invoke_from on dify.workflow.run): api, webapp, debug
  • Message logs (dify.invoke_from on dify.message.run): service-api, web-app, debugger, explore

Null Value Behavior

  • Spans: attributes with null values are omitted.
  • Logs: attributes with null values are rendered as null in the JSON output.
  • Content-gated fields: replaced by a reference string rather than set to null.