ConfigMap Best Practices

This document provides a best practices guide for ConfigMaps in the GeniSpace task scheduling system, helping you efficiently manage configurations for tasks, operators, and workflows.

System Architecture Overview

GeniSpace employs a layered configuration management architecture:

┌─────────────────────────────────────────────────────────────┐
│                    Task Execution Layer (TaskWorker)          │
├─────────────────────────────────────────────────────────────┤
│                  Workflow Engine (WorkflowEngine)             │
├─────────────────────────────────────────────────────────────┤
│              Node Execution Layer (Operators/Agents)          │
├─────────────────────────────────────────────────────────────┤
│                Configuration Layer (ConfigMap Service)        │
└─────────────────────────────────────────────────────────────┘

Configuration Flow

1. Task-level Configuration → TaskWorker → WorkflowEngine
2. Workflow Configuration → Node input parsing → Environment variable injection
3. Operator Configuration → Runtime configuration merge → Execution context

Core Configuration Scenarios

1. External Service Configuration

Purpose: Configure connection information for AI services, databases, message queues, and other external services

name: "external-services-config"
description: "External service connection configuration"
variables:
  DATABASE_URL:
    value: "postgresql://user:pass@db:5432/mydb"
    description: "Database connection string"
    isSecret: true
  
  DASHSCOPE_API_KEY:
    value: "sk-xxx"
    description: "Tongyi Qianwen API key"
    isSecret: true
  
  OPENAI_API_KEY:
    value: "sk-xxx"
    description: "OpenAI API key"
    isSecret: true
  
  SENDGRID_API_KEY:
    value: "SG.xxx"
    description: "SendGrid email service API key"
    isSecret: true
  
  REDIS_URL:
    value: "redis://localhost:6379"
    description: "Redis cache connection address"

2. Operator Runtime Configuration

Purpose: Configure API services, database connections, and other runtime environments for operators

name: "operator-runtime-config"
description: "Operator runtime configuration"
variables:
  API_SERVICE_URL:
    value: "http://api-service:3000"
    description: "API service address"
  
  DATABASE_URL:
    value: "postgresql://user:pass@db:5432/mydb"
    description: "Database connection string"
    isSecret: true
  
  DASHSCOPE_API_KEY:
    value: "sk-xxx"
    description: "Tongyi Qianwen API key"
    isSecret: true

3. Workflow Node Configuration

Purpose: Configure parameters for AI nodes and data processing nodes in workflows

name: "workflow-nodes-config"
description: "Workflow node configuration"
variables:
  DEFAULT_MODEL:
    value: "gpt-4"
    description: "Default AI model"
  
  MAX_TOKENS:
    value: "4000"
    description: "Maximum token count"
  
  BATCH_SIZE:
    value: "100"
    description: "Batch processing size"
  
  SMTP_HOST:
    value: "smtp.example.com"
    description: "Email server address"
  
  SMTP_PASSWORD:
    value: "email_password"
    description: "Email account password"
    isSecret: true

4. Containerized Node Configuration

Purpose: Configure runtime environments for containerized operators

name: "container-runtime-config"
description: "Containerized node runtime configuration"
variables:
  # Docker configuration
  DOCKER_SOCKET:
    value: "/var/run/docker.sock"
    description: "Docker socket path"
  
  # Kubernetes configuration
  KUBE_CONFIG_PATH:
    value: "/etc/kubernetes/config"
    description: "Kubernetes config file path"
  
  KUBE_NAMESPACE:
    value: "mydb"
    description: "Kubernetes namespace"
  
  # Railway configuration
  RAILWAY_API_KEY:
    value: "railway_key_xxx"
    description: "Railway API key"
    isSecret: true
  
  RAILWAY_PROJECT_ID:
    value: "project_123"
    description: "Railway project ID"
  
  # Resource limits
  DEFAULT_CPU_LIMIT:
    value: "1000m"
    description: "Default CPU limit"
  
  DEFAULT_MEMORY_LIMIT:
    value: "512Mi"
    description: "Default memory limit"

5. Environment-Specific Configuration

Purpose: Differentiate configurations for development, testing, and production environments

# Production environment configuration
name: "production-env-config"
description: "Production environment configuration"
variables:
  NODE_ENV:
    value: "production"
    description: "Runtime environment"
  
  LOG_LEVEL:
    value: "error"
    description: "Log level"
  
  HEALTH_CHECK_INTERVAL:
    value: "30000"
    description: "Health check interval (milliseconds)"
  
  HEARTBEAT_INTERVAL:
    value: "30000"
    description: "Heartbeat interval (milliseconds)"
  
  # Production database
  PROD_DATABASE_URL:
    value: "postgresql://prod_user:prod_pass@prod-db:5432/mydb_prod"
    description: "Production database connection"
    isSecret: true
  
  # Production Redis
  PROD_REDIS_URL:
    value: "redis://prod-redis:6379"
    description: "Production Redis connection"
    isSecret: true

---
# Development environment configuration
name: "development-env-config"
description: "Development environment configuration"
variables:
  NODE_ENV:
    value: "development"
    description: "Runtime environment"
  
  LOG_LEVEL:
    value: "debug"
    description: "Log level"
  
  LOG_TASK_TO_FILE:
    value: "true"
    description: "Whether to log tasks to file"
  
  # Development database
  DEV_DATABASE_URL:
    value: "postgresql://dev_user:dev_pass@localhost:5432/mydb_dev"
    description: "Development database connection"
  
  # Development Redis
  DEV_REDIS_URL:
    value: "redis://localhost:6379"
    description: "Development Redis connection"

Configuration Usage Patterns

1. Referencing Configuration in Workflow Nodes

Use the {{VARIABLE_NAME}} format in node configurations:

// Node configuration example
{
  "type": "api-call",
  "config": {
    "url": "{{API_SERVICE_URL}}/users",
    "headers": {
      "Authorization": "Bearer {{API_TOKEN}}"
    },
    "timeout": "{{REQUEST_TIMEOUT}}"
  }
}

2. Referencing Environment Variables in Data Flows

Use the env.VARIABLE_NAME format:

// Edge configuration example
{
  "source": "node1:output",
  "target": "node2:input", 
  "dataPath": "env.DATABASE_HOST"
}

3. Accessing Configuration in Operator Code

Access configuration variables via context.env:

// Operator code example
async execute(inputs, context) {
  const apiUrl = context.env.API_SERVICE_URL;
  const apiKey = context.env.API_KEY;
  
  // Use configuration to execute business logic
  const response = await axios.get(`${apiUrl}/data`, {
    headers: { 'Authorization': `Bearer ${apiKey}` }
  });
  
  return response.data;
}

Configuration Priority

GeniSpace follows this configuration priority (from highest to lowest):

1. Task Direct Configuration - Variables directly defined in the Task's envVars array
2. Operator-Bound ConfigMap - The specific ConfigMap bound to an operator
3. Default ConfigMap - The ConfigMap marked as default (global)

Priority Example

# Default ConfigMap (lowest priority)
name: "global-config"
isDefault: true
variables:
  LOG_LEVEL: "info"
  API_URL: "https://api.service.com"

# Operator-bound ConfigMap (medium priority)  
name: "operator-config"
variables:
  LOG_LEVEL: "debug"  # Overrides default configuration
  DB_HOST: "db.operator.internal"

# Task direct configuration (highest priority)
# In the task definition:
envVars: [
  {
    "key": "LOG_LEVEL", 
    "value": "warn"  # Overrides all other configurations
  }
]

Organization Strategies

Group by Function (Recommended)

├── infrastructure-config    # Infrastructure configuration
├── runtime-config          # Runtime configuration  
├── service-config          # Service configuration
└── environment-config      # Environment configuration

Group by Business Domain

├── auth-service-config     # Authentication service configuration
├── payment-service-config  # Payment service configuration
├── notification-config     # Notification service configuration
└── data-processing-config  # Data processing configuration

Naming Conventions

ConfigMap Naming

Format: {function}-{purpose}-config

Examples:

• task-scheduler-config - Task scheduling configuration
• ai-service-config - AI service configuration
• database-connection-config - Database connection configuration

Configuration Variable Naming

Format: {SERVICE}_{CATEGORY}_{NAME}

Examples:

# Database-related
DB_HOST: "localhost"
DB_PORT: "5432" 
DB_PASSWORD: "secret"

# API-related
API_SERVICE_URL: "http://api:3000"
API_TIMEOUT: "30"
API_KEY: "secret123"

# Task-related  
TASK_MAX_RETRIES: "3"
TASK_TIMEOUT: "1800"

Security Best Practices

1. Sensitive Information Management

• Mark passwords, API keys, and similar data as sensitive
• Use isSecret: true to flag sensitive variables
• Do not include sensitive information in descriptions

variables:
  DATABASE_PASSWORD:
    value: "super_secret_password"
    description: "Database connection password"
    isSecret: true
  
  API_KEY:
    value: "sk-xxx"
    description: "API access key"
    isSecret: true

2. Access Control

• Each team can only access its own ConfigMaps
• Use default and bound ConfigMaps appropriately
• Regularly review configuration access permissions

Common Usage Patterns

1. Environment-Specific Configuration

# Production environment
name: "production-config"
variables:
  NODE_ENV: "production"
  LOG_LEVEL: "error"
  DATABASE_URL: "postgresql://prod-db:5432/app"

# Development environment  
name: "development-config"
variables:
  NODE_ENV: "development"
  LOG_LEVEL: "debug"
  DATABASE_URL: "postgresql://localhost:5432/app_dev"

2. Service-Specific Configuration

# Authentication service configuration
name: "auth-service-config"
variables:
  JWT_SECRET: "jwt_secret_key"
  TOKEN_EXPIRATION: "24h"
  OAUTH_CLIENT_ID: "oauth_client_id"

# Notification service configuration
name: "notification-config"  
variables:
  SMTP_HOST: "smtp.example.com"
  SMTP_PORT: "587"
  SMTP_USER: "noreply@example.com"

3. Feature Toggle Configuration

name: "feature-toggles"
variables:
  ENABLE_NEW_DASHBOARD: "true"
  MAINTENANCE_MODE: "false"
  MAX_UPLOAD_SIZE_MB: "100"
  ENABLE_DEBUG_MODE: "false"

Maintenance Recommendations

1. Regular Backups

• Regularly back up important configurations using the export feature
• Back up existing configurations before making major changes
• Maintain a configuration change log

2. Configuration Validation

• Carefully check YAML format before importing
• Verify that sensitive information is properly flagged
• Test the impact scope of configuration changes

3. Documentation Maintenance

• Add clear descriptions to each ConfigMap
• Add meaningful explanations to configuration variables
• Keep configuration documentation in sync with actual usage

Troubleshooting

Common Issues

1. Configuration Not Taking Effect

   • Check that configuration priority is correct
   • Confirm the ConfigMap is properly bound
   • Verify variable names are spelled correctly

2. Sensitive Information Leakage

   • Ensure sensitive variables are flagged with isSecret: true
   • Check logs for accidental output of sensitive information
   • Regularly update sensitive configurations

3. Configuration Conflicts

   • Understand configuration priority rules
   • Check for duplicate variable names
   • Use the search function to quickly locate configurations

By following these best practices, you can establish an efficient and secure configuration management system within GeniSpace.

Related Documentation

• ConfigMap Overview — ConfigMap basic concepts and operations
• YAML Format Guide — YAML import/export format
• Environment Variables — Using configuration variables in workflows