Skip to main content

​
Overview

The Data-Sync Job synchronizes data across multiple databases to maintain consistency and up-to-date information throughout your system. It operates by connecting to various databases, retrieving data, and synchronizing changes across them. The job logs all actions and can be scheduled to run at regular intervals.

​
Quick Start

# 1. Configure your environment variables in a data-sync-job.yaml file
# 2. Apply the configuration
kubectl apply -f data-sync-job.yaml
# 3. Monitor the job status
kubectl get jobs
# 4. Check logs if needed
kubectl logs job/data-sync-job

​
Required environment variables

​
Core configuration

VariableDescriptionExample
FLOWX_SKIPPEDRESOURCESERVICESComma-separated list of services to skip during synchronizationdocument-plugin,notification-plugin
⚠️ Warning: Do not include spaces in the FLOWX_SKIPPEDRESOURCESERVICES value.

​
Database connections

The Data-Sync Job requires connection details for multiple databases. Configure the following sections based on your deployment.

​
MongoDB connections

Each MongoDB-based service requires the following variables:
ComponentRequired Variables
CMSFLOWX_DATASOURCE_CMS_URI, CMS_MONGO_USERNAME, CMS_MONGO_PASSWORD, CMS_MONGO_DATABASE
SchedulerFLOWX_DATASOURCE_SCHEDULER_URI, SCHEDULER_MONGO_USERNAME, SCHEDULER_MONGO_PASSWORD, SCHEDULER_MONGO_DATABASE
Task ManagerFLOWX_DATASOURCE_TASKMANAGER_URI, TASKMANAGER_MONGO_USERNAME, TASKMANAGER_MONGO_PASSWORD, TASKMANAGER_MONGO_DATABASE
Document PluginFLOWX_DATASOURCE_DOCUMENTPLUGIN_URI, DOCUMENTPLUGIN_MONGO_USERNAME, DOCUMENTPLUGIN_MONGO_PASSWORD, DOCUMENTPLUGIN_MONGO_DATABASE
Notification PluginFLOWX_DATASOURCE_NOTIFICATIONPLUGIN_URI, NOTIFICATIONPLUGIN_MONGO_USERNAME, NOTIFICATIONPLUGIN_MONGO_PASSWORD, NOTIFICATIONPLUGIN_MONGO_DATABASE
App RuntimeFLOWX_DATASOURCE_APPRUNTIME_URI, APPRUNTIME_MONGO_USERNAME, APPRUNTIME_MONGO_PASSWORD, APPRUNTIME_MONGO_DATABASE
Integration DesignerFLOWX_DATASOURCE_INTEGRATIONDESIGNER_URI, INTEGRATIONDESIGNER_MONGO_USERNAME, INTEGRATIONDESIGNER_MONGO_PASSWORD, INTEGRATIONDESIGNER_MONGO_DATABASE
AdminFLOWX_DATASOURCE_ADMIN_URI, ADMIN_MONGO_USERNAME, ADMIN_MONGO_PASSWORD, ADMIN_MONGO_DATABASE
MongoDB URI format
mongodb://${USERNAME}:${PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${DATABASE}

​
PostgreSQL connections

ComponentRequired Variables
Process EngineFLOWX_DATASOURCE_ENGINE_URL, FLOWX_DATASOURCE_ENGINE_USERNAME, FLOWX_DATASOURCE_ENGINE_PASSWORD, FLOWX_DATASOURCE_ENGINE_DRIVERCLASSNAME
Application ManagerFLOWX_DATASOURCE_APPMANAGER_URL, FLOWX_DATASOURCE_APPMANAGER_USERNAME, FLOWX_DATASOURCE_APPMANAGER_PASSWORD, FLOWX_DATASOURCE_APPMANAGER_DRIVERCLASSNAME
Authentication SystemFLOWX_DATASOURCE_AUTHSYSTEM_URL, FLOWX_DATASOURCE_AUTHSYSTEM_USERNAME, FLOWX_DATASOURCE_AUTHSYSTEM_PASSWORD
Driver class names
  • PostgreSQL: org.postgresql.Driver
  • Oracle: oracle.jdbc.OracleDriver

​
Additional configuration

VariableDescription
SPRING_JPA_DATABASEDatabase type for Spring JPA (e.g., postgresql, oracle)
SPRING_JPA_PROPERTIES_HIBERNATE_DEFAULTSCHEMADefault schema for Hibernate
LOGGING_CONFIG_FILEPath to logging configuration file

​
Service to database mapping

Each service in your environment corresponds to specific database datasources:
ServiceDatasources
scheduler-corescheduler
cms-corecms
task-management-plugintask-manager
document-plugindocument-plugin
notification-pluginnotification-plugin
runtime-managerapp-runtime, app-manager
integration-designerintegration-designer
adminadmin, engine
process-engineengine
application-managerapp-manager
authorization-systemauth-system

​
Sample configuration

apiVersion: batch/v1
kind: Job
metadata:
  name: data-sync-job
spec:
  template:
    spec:
      containers:
      - name: data-sync
        image: your-registry/data-sync:latest
        env:
        - name: FLOWX_SKIPPEDRESOURCESERVICES
          value: "document-plugin,notification-plugin"
        # MongoDB connections
        - name: FLOWX_DATASOURCE_CMS_URI
          value: "mongodb://${CMS_MONGO_USERNAME}:${CMS_MONGO_PASSWORD}@mongodb-0.mongodb-headless:27017/${CMS_MONGO_DATABASE}"
        # Add all other required environment variables
      restartPolicy: Never
  backoffLimit: 3

​
SpiceDB configuration

Environment VariableDescriptionDefault Value
FLOWX_SPICEDB_HOSTSpiceDB server hostnamespicedb
FLOWX_SPICEDB_PORTSpiceDB server port50051
FLOWX_SPICEDB_TOKENSpiceDB authentication token-

​
Best practices

  1. Store sensitive credentials in Kubernetes Secrets and reference them in your deployment
  2. Include the Data-Sync Job in your CI/CD pipeline for automated deployment
  3. Schedule regular runs using a Kubernetes CronJob for periodic synchronization
  4. Monitor job execution and set up alerts for failures

​
Troubleshooting

Symptoms: The Data-Sync Job fails to start or exits with database connection errors.Solutions:
  1. Verify MongoDB and PostgreSQL connection strings are correctly formatted
  2. Check that database credentials are correct and the user has appropriate permissions
  3. Ensure network connectivity between the job pod and database services
  4. For MongoDB, confirm the replica set is healthy and reachable
  5. For PostgreSQL, verify the JDBC URL format and driver class name
Symptoms: Data synchronization completes partially or fails.Note: The Data-Sync Job does not use Kafka. It is a batch job that connects directly to each service’s database to synchronize data.Solutions:
  1. Check job logs for specific error messages: kubectl logs job/data-sync-job
  2. Verify all database connection strings are correct and reachable from the job pod
  3. Ensure database users have the required read/write permissions
  4. Confirm that services not installed are listed in FLOWX_SKIPPEDRESOURCESERVICES
Symptoms: Data across services appears out of date or mismatched after the job completes.Solutions:
  1. Verify that the correct services are being synced and none are accidentally listed in FLOWX_SKIPPEDRESOURCESERVICES
  2. Check the service-to-database mapping to ensure each service points to the right datasource
  3. Re-run the Data-Sync Job and monitor logs for partial failures
  4. Confirm that no concurrent writes occurred during the sync window
Symptoms: Job fails immediately on startup with configuration errors.Solutions:
  1. Ensure all required environment variables are set for each database connection
  2. Check for typos in environment variable names
  3. If a service is not installed, add it to FLOWX_SKIPPEDRESOURCESERVICES instead of leaving its variables unconfigured

Redis Configuration

Complete Redis setup including Sentinel and Cluster modes

Kafka Authentication

Configure Kafka security and authentication

IAM Configuration

Identity and access management setup
Last modified on March 25, 2026