> ## Documentation Index
> Fetch the complete documentation index at: https://enterprise-docs.dify.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Push

Data Push configuration is a core function of the enterprise data governance module, built on the OpenTelemetry (OTel) open-source standard, aiming to solve the problem of enterprise data silos. Through a standardized data pipeline, system administrators can push granular platform business data (including application performance metrics, Token consumption, user behavior, etc.) to the enterprise's own BI system or data warehouse in real-time, achieving unified data asset accumulation, refined operational analysis, and global business insights.

## 1. Feature Introduction

In enterprise usage scenarios, Dify platform's business data is scattered across various workspaces and applications, making it difficult to unify and aggregate into the enterprise's own data system, leading to the following pain points:

* **Data Silos**: Unable to perform global analysis on application operation, Token consumption, and user behavior across workspaces.
* **High Integration Cost**: Lack of a standardized data output mechanism requires enterprises to develop customized data collection solutions.
* **Privacy Risks**: Lack of controllability over sensitive content during data push, easily leading to compliance issues.

### 1.1 Feature Value

* **Standardized Compatibility**: Based on the OpenTelemetry protocol, it integrates with existing enterprise monitoring systems, data warehouses, or BI platforms.
* **Full-Link Data Coverage**: Pushes granular business data, supporting multi-dimensional analysis of users, applications, workspaces, etc.
* **High Reliability Guarantee**: Uses a buffer mechanism to handle network fluctuations, prioritizing the protection of main business operations.
* **Privacy Control**: Provides switches to control whether to push sensitive content like Input/Output, meeting compliance requirements.
* **Real-time Push**: Real-time data transmission supports timely operational review and anomaly response by enterprises.

### 1.2 Applicable Scenarios

* **Global Operational Analysis**: Synchronize Dify platform application operation data to the internal BI system for cross-departmental assessment.
* **Refined Cost Accounting**: Calculate Token consumption by workspace/application/user dimensions for cost allocation and optimization.
* **Performance Monitoring & Troubleshooting**: Monitor metrics like request latency and failure rates to quickly locate issues.
* **Compliant Data Management**: Strictly control sensitive data output, pushing only non-sensitive statistical dimension data.

### 1.3 Core Concepts

**Data Transmission & Standards**

* **Data Push**: A service that transmits running logs, monitoring metrics, and other data generated by the Dify platform to external systems in real-time via the OTel standard protocol.

* **OpenTelemetry (OTel)**: An open-source standard used to generate, collect, manage, and export telemetry data (metrics, logs, traces).

* **Unified Endpoint**: Push both Metrics and Traces data to the same endpoint.

* **Separated Endpoints**: Push Metrics and Traces data to the same or different endpoints separately.

**Operation & Control Mechanisms**

* **Buffer**: A temporary storage queue (FIFO) between the push service and the receiver, used to handle network jitter or slow processing by the receiver; when the buffer is full, new data is discarded to protect the main business.
* **Privacy Control Switch**: Controls whether to push specific content of Input/Output; when closed, only metadata like Message ID is pushed, and enterprises need to check the content themselves.

### 1.4 Data Dictionary

For detailed data field definitions, metric descriptions, and log formats of the Data Push feature, please refer to the [Data Push Data Dictionary](/en/3.11.x/administer/data-push-data-dictionary).

## 2. Operation Guide

### 2.0 Quick Start

The typical configuration flow for Data Push includes: **Enter Configuration Page > Fill in OTel Parameters > Test Connectivity > Save Configuration > Start Push Service**.

1. **Enter Configuration Page**
   Log in to the Enterprise Admin Panel, go to \[Data Governance] > \[Data Push Configuration].
2. **Fill in OTel Parameters**
   If the service is not connected, click \[Parameter Configuration] in the top right corner to fill in OpenTelemetry Collector information.
3. **Test & Save**
   Click \[Test Connectivity] to verify network and configuration validity. After confirmation, save the configuration and confirm to restart the service.
4. **Start Service**
   Return to the main page and click \[Connect and Push Data] to start the service.

### 2.1 View Service Status

After entering the \[Data Push Configuration] page, the real-time running status of the service is displayed by default.

**Status 1: Service Not Running**

* Page prompt: "Not connected, please configure parameters and start connection".
* Click the \[Connect and Push Data] button on the right to start the connection, or click \[Parameter Configuration] in the top right to enter the configuration flow.

**Status 2: Service Running (Connected)**
The page displays the following key information:

* **Status Indicator**: Green light + "Service Running".
* **Connection Info**: Start time of the current connection (YYYY-MM-DD HH:mm:ss).
* **Data Statistics**: Pushed data volume (real-time update, unit: MB/GB), unpushed backlog data volume (data temporarily stored in buffer).
* **Service Logs**: Displays exception logs, network exception logs, etc., fed back by OpenTelemetry.

**Status 3: Service Running Exception (Connected)**

* **Status Indicator**: Orange light + "Service Running Exception".

> **Exception Handling Mechanism Explanation:**
>
> * Push uses an asynchronous mechanism and **does not affect the main business link**.
> * When push fails or the connection is interrupted, data is automatically stored in the buffer and push resumes after the connection is restored.
> * When the buffer is full, the newest data is prioritized for discard to ensure main business user interactions are not blocked.
> * *Note: The page will prompt for scenarios that may lead to data loss (e.g., push service exception, downstream consumption slow leading to buffer overflow, system restart).*

### 2.2 Parameter Configuration

Click \[Parameter Configuration] to pop up the configuration form and complete the following settings:

**1. Required Configuration Items**

* **Endpoint URL**: OpenTelemetry Collector endpoint address (e.g., `http://otel-collector:4318`), ensuring the Dify server can access it.
* **Transport Protocol**:
  * **http/protobuf** (Recommended default)
  * **grpc** (Higher performance, suitable for stable internal enterprise networks)
  * **http/json** (For debugging only, lower performance)
* **Compression**:
  * **gzip** (Recommended default)
  * **none** (For debugging only)

**2. Optional Configuration Items**

* **Timeout**: Push timeout, default is 5 seconds.
* **Headers**: Custom request headers for authentication, etc. (e.g., `Authorization: Bearer <token>`), supports multi-line input.
* **Enable TLS Secure Connection**: After enabling, you need to upload certificate files (Certificate File (CA), Client Key File, Client Certificate File). **Recommended for production environments**.

**3. Configuration Operations**

* **Test Connectivity**: Connection validation, prompting success/failure reasons (network unreachable, invalid certificate, etc.).
* **Save Configuration**: A confirmation box "Save application configuration and restart service?" pops up. Confirm to save and return to the running status page.

### 2.3 Privacy Control Settings

In the Privacy Control area of the \[Data Push Configuration] page, you can set:

* **Push Input/Output Content (Switch)**:
  * **ON**: Push request Input content and model Output content.
  * **OFF**: Only push metadata such as Message ID, User ID, App ID, etc., excluding specific content.
* After modification, click \[Save Settings] to take effect.

### 2.4 Start and Stop Push Service

* **Start Service**: In the disconnected state, click \[Connect and Push Data], and the status changes to "Service Running".
* **Stop Service**: In the running state, click \[Stop Push], confirm to stop, and the status changes to "Not Connected".

## 3. Troubleshooting

* **Unable to connect to OpenTelemetry Collector**: Check if the Endpoint URL is correct and the network is connected; if TLS is enabled, check the validity and format of the CA/Client Certificate/Private Key; use \[Test Connectivity] to view specific errors and correct the configuration.
* **Data Push Interruption**: Check running logs to troubleshoot network fluctuations or Collector exceptions; check if the Collector is running normally and if it has reached its reception limit.
* **Data Backlog Continues to Increase**: Usually caused by Collector consumption speed being insufficient or network latency. It is recommended to optimize Collector resource configuration or expand instances.
* **Configuration Save Failed**: Check if required items are filled in completely; if TLS is enabled, confirm relevant certificate files are uploaded correctly.
* **Data Loss**: May occur during service exceptions, buffer overflows, or system restarts (this is a degradation strategy to protect the main business). If critical data cannot be lost, it is recommended to improve Collector processing capability to avoid long-term backlog.

## 4. FAQ

### Will data push affect the performance of the Dify platform's main business?

No. Data push uses an asynchronous mechanism and is isolated from the main business link; when the buffer is full, new data will be discarded, and it will not block user interaction or application operation.

### What content does data push include?

Covers granular information (examples below):

* **Metadata**: Message ID, User ID, App ID, Workspace ID, Timestamp.
* **Operation Metrics**: Token consumption (Input/Output), Request Latency (RT), Running Status (Success/Failure).
* **User Behavior**: Like/Dislike feedback.
  *(Note: Whether to push Input/Output specific content can be decided via the privacy control switch)*.

### Why does data backlog occur?

Usually, the Collector processing speed is slower than the data generation speed, or due to network latency. Backlogged data enters the buffer and continues to be pushed after the connection is restored; if the buffer is full, new data will be discarded.

### What is the recommended configuration for production environments?

It is recommended to use `http/protobuf` or `grpc` for the transport protocol and `gzip` for compression. It is recommended to enable TLS secure connection and configure authentication Headers. If there are compliance requirements, it is recommended to turn off Input/Output content push.

## 5. Best Practices

* **Security & Compliance First**: It is strongly recommended to enable TLS and configure authentication Headers in production environments; in scenarios with strict data privacy requirements, be sure to turn off Input/Output content push.
* **Real-time Monitoring & Alerting**: Regularly check the running status and logs in the management background; it is recommended to configure monitoring alerts on the enterprise Collector side (such as connection interruption, data backlog exceeding threshold, etc.) for timely intervention.
* **Performance Tuning**: In large-scale high-concurrency scenarios, prioritize using the `grpc` protocol; at the same time, ensure sufficient downstream Collector resources to avoid buffer overflow and data loss on the platform side due to insufficient consumption.