Getting Started
You do not need to read every table. Jump directly to the signal layer that matches your observability goal.Scenario-to-Signal Map
Telemetry Data Flow and Routing
All telemetry signals exported by Dify comply with the OpenTelemetry specification and can be split and routed by a standard OTel Collector:Transmission Rules and Resource Attributes
Resource Attributes
Global environment metadata attached to every exported OTel signal:Common Identity and Correlation Attributes
Spans carry thedify.-prefixed form; metric labels and the correlation keys on logs (trace_id, span_id, tenant_id, user_id) use the bare form. Rows listing two names cover both cases.
The three
apprunner.* attributes apply to workflow Spans (dify.workflow.run), node Spans (dify.node.execution), and their companion logs in deployed environments. They aren’t used as metric labels.
OTel-to-Prometheus Naming Conventions
OTel signal names are automatically converted to underscore format when ingested by Prometheus. Counters gain a_total suffix; Histograms produce _bucket, _sum, and _count series, plus a unit suffix (_seconds for metrics measured in s), as the example below shows.
The table below shows common examples; all other metrics follow the same rules.
Null Values and Content-Gated Fields
- Null value behavior: attributes with no value are omitted from Trace Spans; in structured log JSON output they are explicitly rendered as
null. - Content-gating: attributes that may hold sensitive text (Input/Output/Query/Prompt) are gated by
ENTERPRISE_INCLUDE_CONTENT. When content push is disabled (the default), the value is replaced by a reference string (e.g.ref:{id_type}={uuid}) rather thannull. - High-cardinality warning:
tenant_id,app_id, andtrace_idare high-cardinality dimensions. Do not store unbounded text fields as Prometheus labels.
Traces and Spans
Traces present the complete time sequence and nesting topology of a single call. The OTeltrace_id is derived deterministically from the business run ID, so every signal of one run shares it; query Jaeger or Tempo by this trace_id.
The dify.trace_id attribute keeps the original business UUID (for example the workflow run ID) for joining with business data.
dify.workflow.run
dify.node.execution
dify.node.execution.draft
Carries the same attributes asdify.node.execution, but is produced only during a single-node test or a draft preview run in the Studio canvas. Use this signal to separate production telemetry from test runs.
Metrics
Counters
All counters are cumulative and are never sampled: the trace sampling rate (ENTERPRISE_OTEL_SAMPLING_RATE, see the environment variable reference) does not affect them.
Token Counters
Labels:
tenant_id, app_id, operation_type, model_provider, model_name, node_type (for node_execution only)
Token hierarchy and query patterns
Token metrics are emitted at multiple levels. Understanding the hierarchy prevents double-counting:
workflow tokens already include all node_execution tokens. Never add both together.
- Available labels on token metrics:
tenant_id,app_id,operation_type,model_provider,model_name,node_type. - Application name is carried on the trace side only, as
dify.app.nameon the span companion logs (see Span Companion Logs), never as a metric label. Useapp_idin metric queries. - Cost exists only on the node companion logs (
dify.node.total_price, see Span Companion Logs); there is no cost metric.
Request Counter
The
type label identifies the operation (for example dify_requests_total{type="workflow"}); additional labels vary by type:
Error Counter
The error counter uses the same
type label; additional labels vary by type:
Other Counters
Histograms
Unlike Counters, which can only produce averages (which extreme values can distort), Histograms support calculations for the P50, P95, and P99 percentiles that more accurately describe the actual distribution.Each Histogram in Prometheus expands into three series:
_bucket, _sum, and _count. P95/P99 may return NaN in low-traffic time windows; increase the query window if this occurs. When latency is concentrated within 5 seconds, such as for TTFT or tool calls, percentile precision is limited. For higher-precision evaluation, extend the query window or contact the Dify team for bucket configuration details.Structured Logs
Structured logs provide detailed business data. Logs fall into two categories: Span companion logs and standalone business logs.ENTERPRISE_OTEL_SAMPLING_RATE only affects the number of Spans sent to Trace backends such as Jaeger and Tempo. Structured logs (span_detail and metric_only) are always written to Pod standard output at 100% and are not controlled by the sampling rate.Span Companion Logs
Signal type:span_detail. These logs are tightly coupled to a Trace Span (sharing trace_id and span_id) and contain content-gated detailed context.
dify.workflow.run companion log
Common attributes: all Span attributes (see Traces section) plus: Event attributes:dify.event.name:dify.workflow.rundify.event.signal:span_detailtrace_id,span_id,tenant_id,user_id
dify.node.execution and dify.node.execution.draft companion logs
Common attributes: all Span attributes (see Traces section) plus: Event attributes:dify.event.name:dify.node.executionordify.node.execution.draftdify.event.signal:span_detailtrace_id,span_id,tenant_id,user_id
Standalone Logs
Logs without a structured Span. Their signal type ismetric_only: the event feeds counters and carries its detail in this log, with no companion span.
dify.message.run
dify.tool.execution
dify.moderation.check
dify.suggested_question.generation
dify.dataset.retrieval
dify.generate_name.execution
dify.prompt_generation.execution
dify.app.created
dify.app.updated
dify.app.deleted
dify.feedback.created
dify.telemetry.rehydration_failed
A diagnostic event for telemetry pipeline health, emitted when a buffered telemetry payload cannot be reconstructed (rehydrated) for processing. A sustained rate indicates telemetry data loss.Content-Gated Attributes
The following fields may contain user inputs, model outputs, document content, tool parameters, or feedback text. Raw values are transmitted only when content push is enabled on the deployment:ENTERPRISE_INCLUDE_CONTENT (default false; see the environment variable reference). When disabled, these attributes are replaced with ref:{id_type}={uuid}.
The id_type names the correlating entity (for example workflow_run_id, node_execution_id, or message_id); look up that record in the Dify database to retrieve the original content.
dify.node.execution.draft carries the same gated fields as dify.node.execution.
Appendix
Operation Types
workflow, node_execution, message, rule_generate, code_generate, structured_output, instruction_modify
These values belong to the operation_type label on token metrics and to dify.prompt_generation.operation_type. The request and error counters use the separate type label; its values are listed under Counters.
Node Types
start, end, answer, llm, knowledge-retrieval, knowledge-index, if-else, code, template-transform, question-classifier, http-request, tool, datasource, variable-aggregator, loop, iteration, parameter-extractor, assigner, document-extractor, list-operator, agent, trigger-webhook, trigger-schedule, trigger-plugin, human-input
Workflow Statuses
running, succeeded, failed, stopped, partial-succeeded, paused
Payload Types
workflow, node, message, tool, moderation, suggested_question, dataset_retrieval, generate_name, prompt_generation, app, feedback
Invoke From Values
- Workflow spans (
dify.invoke_fromondify.workflow.run):api,webapp,debug - Message logs (
dify.invoke_fromondify.message.run):service-api,web-app,debugger,explore
Null Value Behavior
- Spans: attributes with null values are omitted.
- Logs: attributes with null values are rendered as
nullin the JSON output. - Content-gated fields: replaced by a reference string rather than set to
null.