Integration Designer interacts with various APIs, some of which return large responses. To handle such cases efficiently, the FlowX WebClient buffer size must be configured to accommodate larger payloads, especially when working with legacy APIs that do not support pagination.
Environment variable
Description
Default value
FLOWX_WEBCLIENT_BUFFERSIZE
Buffer size (in bytes) for FlowX WebClient
1048576 (1MB)
If you encounter truncated API responses or unexpected errors when fetching large payloads, consider increasing the buffer size to at least 10MB by setting FLOWX_WEBCLIENT_BUFFERSIZE=10485760. This ensures smooth handling of large API responses, particularly for legacy APIs without pagination support.
The Advancing Controller is a support service that optimizes advancing operations by ensuring efficient, balanced workload distribution—especially during scale-up and scale-down events. To enable Integration Designer to interact with the Advancing database, configure the following environment variables:
It can connect to the same database as the FlowX.AI Engine.
Number of worker threads for reading from database (picking operations)
1
ADVANCING_PROCESSINGTHREADS
Number of threads for parallel processing of advancing events
20
ADVANCING_PROCESSINGBUFFERSIZE
Maximum buffer size for processing queue. Controls how many events can be queued
20
ADVANCING_BLOCKPICKINGIFNOWORKERAVAILABLE
Block picking operations when no worker threads are available
true
ADVANCING_PICKINGPAUSEMILLIS
Pause duration between picking batches (ms)
50
ADVANCING_COOLDOWNAFTERSECONDS
Cooldown period after processing a batch (seconds)
120
ADVANCING_SCHEDULER_HEARTBEAT_CRONEXPRESSION
Cron expression for the heartbeat
"*/2 * * * * ?"
How the new advancing controller works:
Picking threads (ADVANCING_PICKINGTHREADS): Controls how many worker threads read events from the database. This handles only the picking/reading operations.
Processing buffer (ADVANCING_PROCESSINGBUFFERSIZE): Acts as a queue between picking and processing. When the buffer is full, no new events are read. When there’s available space (even just 1 position), that amount of events will be read.
Processing threads (ADVANCING_PROCESSINGTHREADS): Controls how many threads process the advancing events in parallel. Events are processed instantly if processing threads are available. If all processing threads are busy, events accumulate in the buffer until it reaches capacity.
The Advancing Controller supports multiple database systems including PostgreSQL and Oracle. Ensure you configure the appropriate JDBC URL and driver class for your chosen database system.
Integration Designer requires two MongoDB databases for managing integration-specific data and runtime data:
Integration Designer Database (integration-designer): Stores data specific to Integration Designer, such as integration configurations, metadata, and other operational data.
Shared Runtime Database (app-runtime): Shared across multiple services, this database manages runtime data essential for integration and data flow execution.
Integration Designer requires a runtime connection to function correctly. Starting the service without a configured and active runtime MongoDB connection is not supported.
There are two types of Config Params that can be read from the environment: variables and secrets. There is one provider for variables and secrets extracted from the environment variables, and two providers for the ones extracted from Kubernetes. By default, the variables and secrets are extracted from environment variables (env provider).
Configuration parameters from environment variables (default)
The env provider used for variables and secrets extracts them from environment variables. For security reasons, the env provider uses an allow list regex which defaults to FLOWX_CONFIGPARAM_.*. This means only environment variables that match this naming pattern can be read at runtime into configuration params (either as variables or secrets). Feel free to edit it to match the environment variables that you use in your deployment.
Environment variable
Description
Default value
FLOWX_CONFIGPARAMS_VARS_PROVIDER
Provider type for variables
env
FLOWX_CONFIGPARAMS_VARS_ALLOWLISTREGEX
Regular expression to match allowed env variables for variables
FLOWX_CONFIGPARAM_.*
FLOWX_CONFIGPARAMS_SECRETS_PROVIDER
Provider type for secrets
env
FLOWX_CONFIGPARAMS_SECRETS_ALLOWLISTREGEX
Regular expression to match allowed env variables for secrets
You can configure multiple secrets and ConfigMaps by incrementing the index number (e.g., FLOWX_CONFIGPARAMS_PROVIDERS_K8SSECRETS_SECRETSLIST_1, FLOWX_CONFIGPARAMS_PROVIDERS_K8SCONFIGMAPS_CONFIGMAPSLIST_1). Values are overridden based on the order in which the maps are defined.The default provider is env, but there is a built-in allowlist with the regex pattern FLOWX_CONFIGPARAM_.*. This means only configuration parameters that match this naming pattern can be read at runtime, whether they are environment variables or secret variables.
The Integration Designer uses a structured topic naming convention that follows a standardized pattern, ensuring consistency across environments and making topics easily identifiable.
Topic naming components
Environment variable
Description
Default value
KAFKA_TOPIC_NAMING_PACKAGE
Base package for topics
ai.flowx.
KAFKA_TOPIC_NAMING_ENVIRONMENT
Environment identifier
KAFKA_TOPIC_NAMING_VERSION
Topic version
.v1
KAFKA_TOPIC_NAMING_SEPARATOR
Topic name separator
.
KAFKA_TOPIC_NAMING_SEPARATOR2
Alternative separator
-
KAFKA_TOPIC_NAMING_ENGINERECEIVEPATTERN
Engine receive pattern
engine.receive.
KAFKA_TOPIC_NAMING_INTEGRATIONRECEIVEPATTERN
Integration receive pattern
integration.receive.
Topics are constructed using the following pattern:
When configuring Kafka topics in the FlowX ecosystem, ensure proper coordination between services:
Topic name matching: Output topics from one service must match the expected input topics of another service.
Pattern consistency: The pattern values must be consistent across services:
Process Engine listens to topics matching: ai.flowx.dev.engine.receive.*
Integration Designer listens to topics matching: ai.flowx.dev.integration.receive.*
Communication flow:
Other services write to topics matching the Engine’s pattern → Process Engine listens
Process Engine writes to topics matching the Integration Designer’s pattern → Integration Designer listens
The exact pattern value isn’t critical, but it must be identical across all connected services. Some deployments require manually creating Kafka topics in advance rather than dynamically. In these cases, all topic names must be explicitly defined and coordinated.
Large message handling for workflow instances topic
The workflow instances topic requires special configuration to handle large messages. By default, Kafka has message size limitations that may prevent Integration Designer from processing large workflow payloads.Recommended max.message.bytes value: 10485760 (10 MB)
The Integration Designer service uses the standard FlowX.AI ingress pattern. For complete setup instructions including the full ingress template, CORS configuration, and troubleshooting, see the Ingress Configuration Guide.Service-specific values for Integration Designer:
Integration Designer requires specific RBAC (Role-Based Access Control) permissions to access Kubernetes ConfigMaps and Secrets, which store necessary configurations and credentials. Set up these permissions by enabling RBAC and defining the required rules:
This configuration grants read access (get, list, watch) to ConfigMaps, Secrets, and Pods, which is essential for retrieving application settings and credentials required by Integration Designer.
