Data-Sync job setup

Comprehensive guide for configuring and deploying the Data-Sync Job in your Kubernetes environment

Overview

The Data-Sync Job synchronizes data across multiple databases to maintain consistency and up-to-date information throughout your system. It operates by connecting to various databases, retrieving data, and synchronizing changes across them. The job logs all actions and can be scheduled to run at regular intervals.

Quick Start

# 1. Configure your environment variables in a data-sync-job.yaml file
# 2. Apply the configuration
kubectl apply -f data-sync-job.yaml
# 3. Monitor the job status
kubectl get jobs
# 4. Check logs if needed
kubectl logs job/data-sync-job

Required environment variables

Core configuration

Variable	Description	Example
CONFIG_PROFILE	Specifies which configuration profile to use (required)	k8stemplate_v2,kafka-auth

Database connections

The Data-Sync Job requires connection details for multiple databases. Configure the following sections based on your deployment.

MongoDB connections

Each MongoDB-based service requires the following variables:

Component	Required Variables
CMS	FLOWX_DATASOURCE_CMS_URI, CMS_MONGO_USERNAME, CMS_MONGO_PASSWORD, CMS_MONGO_DATABASE
Scheduler	FLOWX_DATASOURCE_SCHEDULER_URI, SCHEDULER_MONGO_USERNAME, SCHEDULER_MONGO_PASSWORD, SCHEDULER_MONGO_DATABASE
Task Manager	FLOWX_DATASOURCE_TASKMANAGER_URI, TASKMANAGER_MONGO_USERNAME, TASKMANAGER_MONGO_PASSWORD, TASKMANAGER_MONGO_DATABASE
Document Plugin	FLOWX_DATASOURCE_DOCUMENTPLUGIN_URI, DOCUMENTPLUGIN_MONGO_USERNAME, DOCUMENTPLUGIN_MONGO_PASSWORD, DOCUMENTPLUGIN_MONGO_DATABASE
Notification Plugin	FLOWX_DATASOURCE_NOTIFICATIONPLUGIN_URI, NOTIFICATIONPLUGIN_MONGO_USERNAME, NOTIFICATIONPLUGIN_MONGO_PASSWORD, NOTIFICATIONPLUGIN_MONGO_DATABASE
App Runtime	FLOWX_DATASOURCE_APPRUNTIME_URI, APPRUNTIME_MONGO_USERNAME, APPRUNTIME_MONGO_PASSWORD, APPRUNTIME_MONGO_DATABASE
Integration Designer	FLOWX_DATASOURCE_INTEGRATIONDESIGNER_URI, INTEGRATIONDESIGNER_MONGO_USERNAME, INTEGRATIONDESIGNER_MONGO_PASSWORD, INTEGRATIONDESIGNER_MONGO_DATABASE
Admin	FLOWX_DATASOURCE_ADMIN_URI, ADMIN_MONGO_USERNAME, ADMIN_MONGO_PASSWORD, ADMIN_MONGO_DATABASE

MongoDB URI format

mongodb://${USERNAME}:${PASSWORD}@mongodb-0.mongodb-headless,mongodb-1.mongodb-headless,mongodb-arbiter-0.mongodb-arbiter-headless:27017/${DATABASE}

PostgreSQL connections

Component	Required Variables
Process Engine	FLOWX_DATASOURCE_ENGINE_URL, FLOWX_DATASOURCE_ENGINE_USERNAME, FLOWX_DATASOURCE_ENGINE_PASSWORD, FLOWX_DATASOURCE_ENGINE_DRIVERCLASSNAME
Application Manager	FLOWX_DATASOURCE_APPMANAGER_URL, FLOWX_DATASOURCE_APPMANAGER_USERNAME, FLOWX_DATASOURCE_APPMANAGER_PASSWORD, FLOWX_DATASOURCE_APPMANAGER_DRIVERCLASSNAME

Driver class names

• PostgreSQL: org.postgresql.Driver
• Oracle: oracle.jdbc.OracleDriver

Additional configuration

Variable	Description
SPRING_JPA_DATABASE	Database type for Spring JPA (e.g., postgresql, oracle)
SPRING_JPA_PROPERTIES_HIBERNATE_DEFAULTSCHEMA	Default schema for Hibernate
LOGGING_CONFIG_FILE	Path to logging configuration file

Service to database mapping

Each service in your environment corresponds to specific database datasources:

Service	Datasources
scheduler-core	scheduler
cms-core	cms
task-management-plugin	task-manager
document-plugin	document-plugin
notification-plugin	notification-plugin
runtime-manager	app-runtime, appmanager
integration-designer	integration-designer
admin	admin, engine
process-engine	engine
application-manager	appmanager

Sample configuration

apiVersion: batch/v1
kind: Job
metadata:
  name: data-sync-job
spec:
  template:
    spec:
      containers:
      - name: data-sync
        image: your-registry/data-sync:latest
        env:
        - name: CONFIG_PROFILE # deprecated since v4.7.6
          value: k8stemplate_v2,kafka-auth
        # MongoDB connections
        - name: FLOWX_DATASOURCE_CMS_URI
          value: "mongodb://${CMS_MONGO_USERNAME}:${CMS_MONGO_PASSWORD}@mongodb-0.mongodb-headless:27017/${CMS_MONGO_DATABASE}"
        # Add all other required environment variables
      restartPolicy: Never
  backoffLimit: 3

Troubleshooting

Common issues

1. Database connection failures:

   • Verify MongoDB and PostgreSQL connection strings
   • Check that database credentials are correct
   • Ensure network connectivity between the job pod and databases

2. Missing Required Variables:

   • Ensure all required environment variables are properly set
   • Check for typos in environment variable names

Logs

Monitor the Data-Sync Job logs to diagnose issues:

kubectl logs job/data-sync-job

Best practices

1. Store sensitive credentials in Kubernetes Secrets and reference them in your deployment
2. Include the Data-Sync Job in your CI/CD pipeline for automated deployment
3. Schedule regular runs using a Kubernetes CronJob for periodic synchronization
4. Monitor job execution and set up alerts for failures

Next steps

After successfully configuring and deploying the Data-Sync Job:

1. Verify data consistency across databases
2. Set up monitoring and alerting for job status
3. Consider automating deployment through your CI/CD pipeline