> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# FlowX.AI data architecture: which service uses which store

> See which data stores FlowX.AI uses, what each one holds, and which services read and write each PostgreSQL database, MongoDB database, and bucket.

## Overview

FlowX.AI uses multiple specialized data stores, each chosen for specific workload characteristics: PostgreSQL for process definitions and instance state, MongoDB for runtime configuration and the FlowX Database, Redis for caching and sessions, Kafka for event streaming, S3-compatible object storage for files and documents, Elasticsearch for audit logs and search indexes, and Qdrant for vector embeddings and semantic retrieval. They fall into two categories:

* **Embedded data components**: shipped with the platform, managed by FlowX.AI
* **Platform data services**: third-party dependencies deployed by your team

```mermaid theme={"dark"}
flowchart LR
    FX["FlowX.AI Platform"]

    FX -->|"process definitions, instance state,\nplatform config, metadata"| PG[("PostgreSQL")]
    FX -->|"runtime config, workflow state,\nFlowX Database"| MONGO[("MongoDB")]
    FX -->|"cache, sessions,\ndistributed locks"| REDIS[("Redis")]
    FX -->|"event streaming,\nasync messaging"| KAFKA[("Kafka")]
    FX -->|"files, documents,\nbinary assets"| S3[("S3-compatible\nobject storage")]
    FX -->|"audit logs,\nsearch indexes"| ES[("Elasticsearch")]
    FX -->|"vector embeddings,\nsemantic retrieval"| QD[("Qdrant")]

    style PG fill:#336791,color:#fff
    style MONGO fill:#4DB33D,color:#000
    style REDIS fill:#DC382D,color:#fff
    style KAFKA fill:#231F20,color:#fff
    style S3 fill:#FF9900,color:#000
    style ES fill:#FED10A,color:#000
    style QD fill:#DC244C,color:#fff
```

***

## Embedded data components

> [!NOTE]
>
> Embedded data components are delivered as part of the FlowX.AI platform (Docker images + Helm charts). They run on Kubernetes only, and their versioning is managed by FlowX.AI. These components are included in the standard deployment and cannot be replaced by alternative implementations.

> [card] [Qdrant](https://qdrant.tech/documentation/)
>
> **Type:** Vector database
>
> Qdrant is used by the AI Platform RAG path (`knowledgebase-rag`, `embedder`, `knowledgebase-indexer-v2`) and by AI agents for semantic retrieval. It stores vector embeddings and serves similarity search over indexed knowledge.
>
> It holds separate collections for **runtime knowledge bases** (queried by business agents at run time), **design-time knowledge bases** (used by the config-time agents in FlowX Designer), and the **AI assistant** corpus. Qdrant runs embedded as a single node by default; cluster mode is recommended for production.

***

## Platform data services

> [!NOTE]
>
> The following data services are required by the FlowX.AI platform but are third-party dependencies. You choose the deployment model (managed service or self-hosted, inside or outside the Kubernetes cluster) according to your enterprise standards. FlowX.AI connects to these services via configuration.
>
> For supported versions and compatibility details, see the [Third-party components](https://docs.flowx.ai/6.x/docs/platform-deep-dive/third-party-components.md#compatibility-matrix) compatibility matrix.

### Relational storage with PostgreSQL or Oracle

PostgreSQL is the primary relational database and system-of-record for most core services. Oracle Database is also supported as an alternative. They store:

* Process definitions and metadata
* Runtime instance state
* Platform configuration
* Administrative data

Most FlowX.AI microservices use PostgreSQL or Oracle as their authoritative storage layer.

### Document storage with MongoDB

MongoDB provides document-oriented storage for unstructured and semi-structured content. It stores:

* Runtime configuration
* Runtime workflow state
* Flexible data structures used by apps and integrations

MongoDB is also the storage layer for [FlowX Database](https://docs.flowx.ai/6.x/docs/platform-deep-dive/integrations/flowx-database.md), a cross-instance, long-term storage feature that lets apps persist and query data beyond a single process instance.

### Caching with Redis

Redis is an in-memory cache used for performance optimization across the platform. It stores:

* Cached process definitions
* Transient session data
* Distributed locks across microservices

> [!NOTE]
>
> Redis is used strictly for short-lived, performance-related data and is **not** a system-of-record. Long-term persistence is handled by PostgreSQL and MongoDB.

> [!NOTE]
>
> Message-start deduplication and pending external (received) messages are persisted in PostgreSQL, which is their source of truth. Message-start deduplication relies on a PostgreSQL unique constraint. Redis is read only as a fallback for pending messages written before the upgrade, so in-flight message processing survives a Redis restart or data loss.

### Event streaming with Kafka

Kafka is the event streaming backbone that enables asynchronous, event-driven communication between microservices. It handles:

* Internal event propagation between platform services
* External integration messaging
* Decoupled processing workflows

> [!NOTE]
>
> Kafka is used for transient message propagation. Authoritative business data is persisted in PostgreSQL and MongoDB, not in Kafka.

### S3-compatible object storage

S3-compatible object storage provides persistent file storage. It stores:

* File attachments
* Document outputs (generated PDFs, converted files)
* Large binary assets
* Kafka messages above the payload size limit
* Runtime content export and import files (see [S3-compatible object storage buckets](#s3-compatible-object-storage-buckets))

FlowX.AI accesses object storage via an S3-compatible interface.

> [!TIP]
>
> For providers without native S3 support (such as Azure Blob Storage), an S3 proxy is required to expose an S3-compatible interface. If you use a native S3 service (like AWS S3), no proxy is needed.

> [!WARNING]
>
> On Azure Blob Storage, multipart uploads (files over 5 MB) through an S3 proxy can require Azure roles that include blob index tag permissions, which `Storage Blob Data Contributor` does not cover. See [Azure Blob Storage through an S3 proxy](https://docs.flowx.ai/6.x/setup-guides/documents-plugin-setup.md#azure-blob-storage-through-an-s3-proxy) for the required roles.

### Search and indexing with Elasticsearch

Elasticsearch is used primarily for audit logging and full-text search across workflow and runtime data. It stores:

* Searchable representations of audit events
* Platform activity logs
* Indexed workflow data for fast retrieval

Elasticsearch is optimized for fast search and traceability rather than primary data storage.

***

## AI Platform data stores

The AI Platform does not introduce a separate data stack. Its services reuse the same store families as the rest of the platform (PostgreSQL, MongoDB, Kafka, and object storage) and add **Qdrant** as the one AI-specific store for vector embeddings.

| Data store | How the AI Platform uses it |
| - | - |
| **Qdrant** | Vector embeddings and similarity search for knowledge-base RAG and AI agents. Separate collections for runtime knowledge bases, design-time knowledge bases, and the AI assistant. |
| **PostgreSQL** | Service state for AI components: embedding-job tracking and document-enrichment artifacts. Standalone services (document parser, speech-to-text) keep durable job state here. |
| **Kafka** | Asynchronous AI job topics (document parsing, speech-to-text, web crawling, evaluations, data privacy) and the internal knowledge-base pipeline that chains indexing, embedding, and enrichment. |
| **Object storage (S3)** | Knowledge-base enrichment payloads, email attachment scan results, files that AI nodes read and write, dynamic classifier model artifacts, Knowledge Base vectors during runtime content export and import, and Kafka messages above the payload size limit. Document services (document parser, document converter, speech-to-text, data privacy) also read the files a request points to, usually stored in Document Plugin buckets. See [AI Platform buckets](#ai-platform-buckets). |
| **MongoDB** | AI Gateway conversation and message history (`ai-gateway` database). Knowledge-base **metadata** (store and entry definitions), owned by Integration Designer. The vectors themselves live in Qdrant, so a knowledge base spans MongoDB (metadata) and Qdrant (embeddings). |

> [!NOTE]
>
> All vector storage and similarity search across the AI Platform use **Qdrant**. PostgreSQL holds only service/job state for the AI components, not embeddings.

For exact databases, environment variables, Kafka topics, and provisioning details, see the [AI Platform setup](https://docs.flowx.ai/6.x/setup-guides/ai-platform-setup.md) guide.

***

## Quick reference

| Data Store | Role | What it stores | Persistence |
| - | - | - | - |
| **PostgreSQL / Oracle** | System of record | Process definitions, config, metadata, instance state | Permanent |
| **MongoDB** | Document store | Runtime state, flexible data, FlowX Database | Permanent |
| **Redis** | Cache | Process definitions cache, sessions, locks | Transient |
| **Kafka** | Event streaming | Inter-service messages, integration events | Transient |
| **Object Storage** | File store | Attachments, documents, binary assets | Permanent |
| **Elasticsearch** | Search engine | Audit logs, indexed workflow data | Permanent |
| **Qdrant** | Vector database (AI) | Embeddings for RAG and semantic retrieval | Permanent |

***

## Service-level database mapping

The tables and diagrams below show which FlowX.AI services connect to which databases and what operations they perform. This is useful for infrastructure sizing, backup planning, and troubleshooting.

### PostgreSQL databases

Each core service owns its own PostgreSQL database and manages its schema through Liquibase migrations at startup.

```mermaid theme={"dark"}
flowchart LR
    subgraph Services
        ADM["Admin"]
        ADV["Advancing Controller"]
        APPM["Application Manager"]
        AUTH["Authorization System"]
        DOC["Document Plugin"]
        EMAIL["Email Gateway"]
        INTD["Integration Designer"]
        LIC["License"]
        ORG["Organization Manager"]
        ENG["Process Engine"]
        TASK["Task Management"]
        DSYNC["Data Sync"]
        WHG["Webhook Gateway"]
    end

    subgraph PostgreSQL
        ADVDB[("advancing")]
        APPMDB[("app_manager")]
        AUTHDB[("authorization_system")]
        DOCDB[("document_plugin")]
        EMAILDB[("email_gateway")]
        LICDB[("license")]
        ORGDB[("org_manager")]
        ENGDB[("process_engine")]
        WHGDB[("webhook_gateway")]
    end

    ADM -->|"reads, writes"| ENGDB
    ADV -->|"creates, reads, writes"| ADVDB
    APPM -->|"creates, reads, writes"| APPMDB
    AUTH -->|"creates, reads, writes"| AUTHDB
    DOC -->|"creates, reads, writes"| DOCDB
    EMAIL -->|"creates, reads, writes"| EMAILDB
    INTD -->|"reads, writes"| ADVDB
    LIC -->|"creates, reads, writes"| LICDB
    ORG -->|"creates, reads, writes"| ORGDB
    ENG -->|"creates, reads, writes"| ENGDB
    ENG -->|"reads, writes"| ADVDB
    TASK -->|"reads, writes"| ENGDB
    WHG -->|"creates, reads, writes"| WHGDB
    DSYNC -.->|"manages migrations"| APPMDB
    DSYNC -.->|"manages migrations"| AUTHDB
    DSYNC -.->|"manages migrations"| ENGDB
```

| Service | Database | Operations |
| - | - | - |
| Admin | `process_engine` | Reads, writes |
| Advancing Controller | `advancing` | Creates, reads, writes |
| Application Manager | `app_manager` | Creates, reads, writes |
| Authorization System | `authorization_system` | Creates, reads, writes |
| Document Plugin | `document_plugin` | Creates, reads, writes |
| Email Gateway | `email_gateway` | Creates, reads, writes |
| Integration Designer | `advancing` | Reads, writes |
| License | `license` | Creates, reads, writes |
| Organization Manager | `org_manager` | Creates, reads, writes |
| Process Engine | `process_engine` | Creates, reads, writes |
| Process Engine | `advancing` | Reads, writes |
| Task Management | `process_engine` | Reads, writes |
| Webhook Gateway | `webhook_gateway` | Creates, reads, writes |

> [!NOTE]
>
> Admin and Task Management share the `process_engine` database with Process Engine. Process Engine and Integration Designer share the `advancing` database with Advancing Controller. Data Sync acts as a centralized migration coordinator for `app_manager`, `authorization_system`, and `process_engine`.

### MongoDB databases

Each service manages its own MongoDB database. The `app-runtime` database is a shared read-only dependency for multiple services.

```mermaid theme={"dark"}
flowchart LR
    subgraph Services
        ADM["Admin"]
        APPM["Application Manager"]
        RTM["Runtime Manager"]
        CMS["CMS Core"]
        DOC["Document Plugin"]
        INTD["Integration Designer"]
        NOTIF["Notification Plugin"]
        SCHED["Scheduler"]
        TASK["Task Management"]
        ENG["Process Engine"]
        EMAIL["Email Gateway"]
        FILEG["File Gateway"]
        AIGW["AI Gateway"]
        NOSQL["NoSQL DB Runner"]
        DUSR["Designer User"]
    end

    subgraph MongoDB
        DM[("data-model")]
        AR[("app-runtime")]
        CMSDB[("cms-core")]
        DOCDB[("document-plugin")]
        NOTIFDB[("notification-plugin")]
        SCHEDDB[("scheduler-core")]
        TASKDB[("task-management-plugin")]
        INTDDB[("integration-designer")]
        NOSQLDB[("nosql-db-runner")]
        FILEGDB[("file-gateway")]
        AIGWDB[("ai-gateway")]
        DUSRDB[("designer-user")]
    end

    ADM -->|"creates, reads, writes"| DM
    APPM -->|"creates, reads, writes"| AR
    RTM -->|"reads, writes"| AR
    CMS -->|"creates, reads, writes"| CMSDB
    CMS -->|"reads"| AR
    DOC -->|"creates, reads, writes"| DOCDB
    DOC -->|"reads"| AR
    INTD -->|"creates, reads, writes"| INTDDB
    INTD -->|"reads"| AR
    NOTIF -->|"creates, reads, writes"| NOTIFDB
    NOTIF -->|"reads"| AR
    SCHED -->|"creates, reads, writes"| SCHEDDB
    TASK -->|"creates, reads, writes"| TASKDB
    TASK -->|"reads"| AR
    ENG -->|"reads"| AR
    EMAIL -->|"reads"| AR
    NOSQL -->|"creates, reads, writes"| NOSQLDB
    FILEG -->|"creates, reads, writes"| FILEGDB
    AIGW -->|"creates, reads, writes"| AIGWDB
    DUSR -->|"creates, reads, writes"| DUSRDB
```

| Service | Database | Operations |
| - | - | - |
| Admin | `data-model` | Creates, reads, writes |
| Application Manager | `app-runtime` | Creates, reads, writes |
| Runtime Manager | `app-runtime` | Reads, writes |
| CMS Core | `cms-core` | Creates, reads, writes |
| CMS Core | `app-runtime` | Reads |
| Document Plugin | `document-plugin` | Creates, reads, writes |
| Document Plugin | `app-runtime` | Reads |
| Integration Designer | `integration-designer` | Creates, reads, writes |
| Integration Designer | `app-runtime` | Reads |
| Notification Plugin | `notification-plugin` | Creates, reads, writes |
| Notification Plugin | `app-runtime` | Reads |
| Scheduler | `scheduler-core` | Creates, reads, writes |
| Task Management | `task-management-plugin` | Creates, reads, writes |
| Task Management | `app-runtime` | Reads |
| Process Engine | `app-runtime` | Reads |
| Email Gateway | `app-runtime` | Reads |
| NoSQL DB Runner | `nosql-db-runner` | Creates, reads, writes |
| File Gateway | `file-gateway` | Creates, reads, writes |
| AI Gateway | `ai-gateway` | Creates, reads, writes |
| Designer User | `designer-user` | Creates, reads, writes |

> [!WARNING]
>
> The `app-runtime` database is created by Application Manager, written by Runtime Manager when it deploys builds, and read by 7 other services. It contains runtime configuration, build data, and deployed app state. Availability of this database is critical for runtime operations.

### Elasticsearch indexes

| Service | Index | Operations |
| - | - | - |
| Audit Core | Audit index | Creates, reads, writes |
| Process Engine | Process instance index | Creates, reads, writes |
| Data Search | Process instance index | Reads |

### Redis

Core services use Redis as a cache via Spring Cache Manager. Events Gateway, Integration Designer, and Process Engine also use Redis for more than caching, as the following table shows.

Services using Redis for caching: Admin, Application Manager, Authorization System, CMS Core, Document Plugin, Email Gateway, Integration Designer, License, Notification Plugin, Organization Manager, Process Engine, Runtime Manager, Task Management.

| Service | Usage |
| - | - |
| Events Gateway | Cache, Redis Streams to route messages to the instance that holds the SSE connection |
| Integration Designer | Cache, pub/sub for workflow cancellation |
| Process Engine | Cache, pub/sub for runtime configuration parameter changes |

### S3-compatible object storage buckets

> [!NOTE]
>
> Bucket names shown are defaults from the configuration and are configurable via environment variables (for example, `MINIO_BUCKET_PREFIX`). Your deployment may use different names. In some deployments, multiple services may share the same bucket. Ensure cross-bucket read access is configured where needed.
>
> The Document Plugin bucket prefix (`MINIO_BUCKET_PREFIX`) is a **prefix**, not a single bucket name. The service derives per-type buckets from it as `<prefix>-<type>`. With the default partition strategy (`NONE`), it stores process documents in one bucket per process instance, `<prefix>-process-id-<processInstanceId>`, files that are not tied to a process instance in `<prefix>-svc-file`, and test files in `<prefix>-test-documents`. The table shows the default prefix. Date partitioning and runtime buckets change this layout, see [Documents plugin setup](https://docs.flowx.ai/6.x/setup-guides/documents-plugin-setup.md#runtime-buckets-configuration).

| Service | Default Bucket | Env Variable | Purpose |
| - | - | - | - |
| Application Manager | `applications-bucket` | — | App builds and exported/imported resources |
| CMS Core | `media-library-bucket` | — | Public media assets |
| CMS Core | `cms-private-storage-bucket` | — | Private theme and font files |
| Document Plugin | `flowx-dev-bucket` | `MINIO_BUCKET_PREFIX` | Document templates and generated files |
| Document Plugin | `temp-bucket` | `MINIO_TEMP_BUCKET` | Temporary processing files |
| Integration Designer | `workflows-bucket` | — | Workflow file uploads and Kafka messages above the payload size limit, including large workflow responses to Process Engine |
| Application Manager, Integration Designer | `runtime-content-bucket` (shared) | `APPLICATION_FILESTORAGE_S3_RUNTIMECONTENTBUCKET` | Runtime content export and import files (Knowledge Base and FlowX Database content). Application Manager creates the bucket if it is missing. |
| Process Engine | Integration Designer's bucket (`workflows-bucket`) | — | Read-only. Downloads the large Kafka messages that Integration Designer stores there. |
| Notification Plugin | `qdevlocal-preview-paperflow` | `APPLICATION_FILESTORAGE_S3_BUCKETPREFIX` | Configured, but not used to store files. The plugin reads email attachments from Document Plugin buckets by their full path. |

> [!NOTE]
>
> Kafka messages larger than the payload size limit (`FLOWX_KAFKA_PAYLOADSIZELIMIT`, 512000 bytes by default) are not sent inline. The producer stores the message as a zip file under `payload-offload/` in its bucket and sends the path instead. The consumer reads the file from the bucket named in that path, so it needs read access to the producer's bucket. Integration Designer and the AI services use `workflows-bucket` for these files.

> [!NOTE]
>
> Browsers upload runtime content import files to `runtime-content-bucket` and download export files from it directly, through presigned URLs. By default, Application Manager signs these URLs with the internal storage endpoint, which only pods can reach. Set `APPLICATION_FILESTORAGE_S3_PRESIGNSERVERURL` on Application Manager to a storage endpoint that browsers can reach.

#### AI Platform buckets

The `flowx-ai` Helm chart does not set AI bucket names. The defaults below are the names the previous charts set. To set them in the `flowx-ai` chart, see [Migrate S3 buckets](https://docs.flowx.ai/6.x/setup-guides/migrate-to-flowx-ai-helm-chart.md#migrate-s3-buckets).

| Service | Default Bucket | Env Variable | Purpose |
| - | - | - | - |
| Agent Builder | `workflows-bucket` | `APPLICATION_FILE_STORAGE_S3_BUCKET_NAME` | Files that AI nodes read and write. Large Kafka messages go to `workflows-bucket`. |
| Document Parser | `email-ingest` | `APPLICATION_FILE_STORAGE_S3_DOCUMENTS_BUCKET_NAME` | Email attachment scan results. Large Kafka messages go to `workflows-bucket`. |
| Document Converter | — | `APPLICATION_FILE_STORAGE_S3_DOCUMENTS_BUCKET_NAME` | PDF previews and conversion logs for email attachments |
| Speech-to-Text | — | — | Reads audio from the bucket the request names. Large Kafka messages go to `workflows-bucket`. |
| Data Privacy | `data-privacy-bucket` | `APPLICATION_FILE_STORAGE_S3_DOCUMENTS_BUCKET_NAME` | Source files for redaction jobs. Redaction jobs for PDF and image files write the redacted file next to the source file. |
| Dynamic Classifier | — | `APPLICATION_FILE_STORAGE_S3_BUCKET_NAME` | Trained model artifacts, under `dynamic-classifier/artifacts/`. Artifact storage is off when no bucket is set. Large Kafka messages go to `workflows-bucket`. |
| Embedder | — | — | Reads and writes Knowledge Base vectors during runtime content export and import, in the bucket and path that each request names (`runtime-content-bucket`) |
| Knowledge Base Indexer, Knowledge Base Enrichment | `kb-enrichment-bucket` (shared) | `APPLICATION_FILE_STORAGE_S3_BUCKET_NAME` | Enrichment payloads the indexer hands to the enrichment service, under `kb-enrichment/`. The indexer creates the bucket if it is missing. |

Document Parser, Document Converter, Speech-to-Text, and Data Privacy also read the file a request points to, usually stored in a Document Plugin bucket, with their own credentials. Grant each of them read access to those buckets. Data Privacy writes the redacted file next to the source file, and Document Converter does the same with converted files when called through its API, so both also need write access there.

### Qdrant collections

The AI Platform stores vector embeddings in separate Qdrant collections. The `embedder` service writes vectors; retrieval and design-time services read them.

| Collection | Written by | Read by | Content |
| - | - | - | - |
| Runtime knowledge bases | Embedder | Knowledgebase RAG, Planner | Embeddings for knowledge bases queried by business agents at run time |
| Design-time knowledge bases | Embedder | KB Enrichment, Planner, Knowledgebase RAG | Layout-aware embeddings for documents used by the config-time agents in FlowX Designer |
| AI assistant | Snapshot on startup | AI Assistant | Pre-indexed documentation corpus for the AI assistant |

> [!NOTE]
>
> Collection names are versioned and encode the embedding models in use, so exact names vary by deployment and release. Treat the collection roles above as the stable reference.

***

## Related resources

> [card] [Third-party components](https://docs.flowx.ai/6.x/docs/platform-deep-dive/third-party-components.md)
>
> Supported versions and compatibility matrix

> [card] [FlowX Database](https://docs.flowx.ai/6.x/docs/platform-deep-dive/integrations/flowx-database.md)
>
> Cross-instance, long-term data storage using MongoDB

> [card] [FlowX.AI architecture](https://docs.flowx.ai/6.x/docs/platform-overview/flowx-architecture.md)
>
> Overall platform architecture and microservices overview

> [card] [Redis configuration](https://docs.flowx.ai/6.x/setup-guides/redis-configuration.md)
>
> Redis deployment modes and configuration

> [card] [AI Platform setup](https://docs.flowx.ai/6.x/setup-guides/ai-platform-setup.md)
>
> AI Platform data stores, environment variables, and Kafka topics


## Related topics

- [FlowX.AI architecture: services, layers, and data flow](https://docs.flowx.ai/6.x/docs/platform-overview/flowx-architecture.md)
- [Migrate self-hosted FlowX.AI to the flowx-ai Helm chart](https://docs.flowx.ai/6.x/setup-guides/migrate-to-flowx-ai-helm-chart.md)
- [Third-party components](https://docs.flowx.ai/6.x/docs/platform-deep-dive/third-party-components.md)
- [Configure the flowx-ai Helm chart for your infrastructure](https://docs.flowx.ai/6.x/setup-guides/configure-flowx-ai-helm-chart.md)
- [FlowX.AI 5.3.0 Release Notes](https://docs.flowx.ai/release-notes/v5.x/v5.3.0-december-2025/v5.3.0-december-2025.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.