Argo CD Sync Waves and Hooks: How to Control Deployment Order

apiVersion: v1
kind: ConfigMap
metadata:
  name: application-config
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
data:
  APP_ENVIRONMENT: "production"
  LOG_LEVEL: "Information"

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: production
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: orders-api
          image: example-registry/orders-api:2.0.0
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: application-config

apiVersion: batch/v1
kind: Job
metadata:
  name: orders-database-migration
  namespace: production
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/sync-wave: "-1"
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 300
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migration
          image: example-registry/orders-api:2.0.0
          command:
            - dotnet
            - Orders.Api.dll
            - migrate
          envFrom:
            - secretRef:
                name: orders-database-secret

apiVersion: batch/v1
kind: Job
metadata:
  name: deployment-validation
  namespace: production
  annotations:
    argocd.argoproj.io/hook: Sync
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: validation
          image: example-registry/deployment-tools:1.0.0
          command:
            - /bin/sh
            - -c
            - echo "Running deployment validation"

apiVersion: batch/v1
kind: Job
metadata:
  name: orders-api-smoke-test
  namespace: production
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  backoffLimit: 1
  activeDeadlineSeconds: 180
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke-test
          image: curlimages/curl:8.12.1
          command:
            - /bin/sh
            - -c
            - |
              curl --fail --retry 5 --retry-delay 2 \
                http://orders-api.production.svc.cluster.local:8080/health

This example assumes the application exposes a /health endpoint through the specified Kubernetes Service and that the selected image contains the required command-line utilities.

The retry settings allow the test to tolerate brief startup delays. They do not guarantee that the application is functioning correctly beyond the behavior verified by the health endpoint.

The HookSucceeded policy removes the hook resource after successful execution. If the Job fails, it remains available for investigation unless another cleanup policy or process removes it.

Combine Sync Waves and Hooks

Sync waves and hooks solve related but different problems.

• Sync waves define the relative ordering of resources.

• Hooks define when special resources execute during synchronization.

They can be combined to build a more controlled deployment sequence.

Consider an application that requires a database migration, a ConfigMap, a Deployment, and a smoke test.

The intended sequence might be:

Stage	Resource	Configuration
1	Database migration Job	PreSync, wave -2
2	ConfigMap	Wave -1
3	Application Deployment	Wave 0
4	Smoke-test Job	PostSync

The migration must complete before normal resources are synchronized. The ConfigMap is then applied before the Deployment. After synchronization succeeds and required resources are healthy, the smoke test checks the application.

A simplified directory structure might look like this:

argocd/
└── orders-api/
    ├── configmap.yaml
    ├── deployment.yaml
    ├── service.yaml
    ├── migration-job.yaml
    └── smoke-test-job.yaml

Argo CD can manage these manifests as part of one application, provided the repository source and application configuration include the files.

Important Consideration: Hook Ordering

Hook types and sync waves are evaluated together with Argo CD's synchronization rules.

A PreSync hook runs before the normal resource synchronization phase, regardless of the normal resource waves. A PostSync hook runs after successful synchronization and the required health checks.

Within a phase, wave ordering controls the order in which applicable resources are processed. Resources within the same wave should not be assumed to execute sequentially or to be fully ready in a particular order.

If one resource depends on another resource's successful completion, model that dependency explicitly rather than relying on the order of filenames in the Git repository.

Control Hook Cleanup and Repeated Deployments

Hooks can create Kubernetes Jobs that remain in the cluster after synchronization. Without a cleanup strategy, repeated deployments can leave behind completed or failed Jobs.

Argo CD provides hook deletion policies that control when hook resources are removed.

Common policies include:

• BeforeHookCreation: Remove an existing hook resource before creating a new instance.

• HookSucceeded: Remove the hook after it succeeds.

• HookFailed: Remove the hook after it fails.

Choose a policy based on the operational purpose of the hook.

For example, a routine smoke test may use HookSucceeded to avoid accumulating completed Jobs. A diagnostic hook may retain failed Jobs so that developers can inspect logs and investigate failures.

For Jobs, Kubernetes TTL-based cleanup can also be useful, but it should be configured with an understanding of how it interacts with Argo CD's hook lifecycle and debugging requirements.

A hook's deletion policy should not be treated as a substitute for application-level cleanup or reliable execution logic.

Best Practices for Production Deployments

Make Migration Jobs Idempotent

A migration may be retried after a timeout or partial failure. Its behavior should be designed so that rerunning it does not corrupt the database or duplicate changes.

Use migration tracking, transactional operations where supported, and explicit validation of the current schema state.

Prefer Backward-Compatible Database Changes

A database migration can succeed while an older application instance is still running. During a rolling deployment, old and new versions may temporarily operate against the same database.

Use an expand-and-contract approach when possible:

1. Add new schema elements without immediately removing the old ones.

2. Deploy application code that can work with the compatible schema.

3. Migrate data when necessary.

4. Remove obsolete schema elements in a later deployment after confirming that no running application version depends on them.

This reduces the risk of application failures during rolling updates.

Set Timeouts and Retry Limits

Migration and validation Jobs should have appropriate retry limits and deadlines. A Job that waits indefinitely can prevent a synchronization operation from completing.

Choose timeout values based on the expected operation duration and the recovery procedure for failures.

Use Meaningful Health Checks

A successful Kubernetes rollout does not necessarily prove that every business operation is working correctly.

Use readiness probes for application availability and meaningful smoke tests for critical application behavior. Avoid relying on a single endpoint if important dependencies require separate validation.

Keep Secrets Out of Git

Database credentials, API keys, and other sensitive configuration should not be committed directly to Git repositories.

Use Kubernetes Secrets with appropriate access controls or an approved external secrets management solution. Ensure that hook Jobs receive only the credentials they need.

Test Failure Scenarios

Test what happens when a migration fails, a smoke test returns an error, or a resource remains unhealthy.

A safe deployment process should make failures visible, preserve useful diagnostic information, and define how the team will recover.

Common Mistakes and Troubleshooting

Problem	Possible cause	Recommended action
Migration Job does not run before deployment	Incorrect hook annotation or resource configuration	Verify the PreSync annotation and Argo CD application source
Resources appear to deploy in an unexpected order	Resources share a wave or have different health behavior	Review wave values and synchronization events
Synchronization stops after a migration failure	The PreSync hook failed	Inspect the Job status and logs before retrying
Smoke test never runs	Synchronization did not succeed or required resources are unhealthy	Review application health and synchronization status
Completed Jobs accumulate	No cleanup policy is configured	Add an appropriate hook deletion policy or cleanup strategy
Migration succeeds but the application fails	The new schema is incompatible with an active application version	Use backward-compatible migrations and test rolling deployment behavior
Hook fails with an authorization error	The Job's service account or credentials lack required permissions	Review Kubernetes RBAC, cloud IAM permissions, and secret access

Useful commands for investigating a failed Job include:

kubectl get jobs -n production
kubectl describe job orders-database-migration -n production
kubectl logs job/orders-database-migration -n production

These commands help identify the Job's state, recent events, and container output. If the Job has multiple containers or Pods, inspect the relevant container and Pod logs as well.

For Argo CD-specific issues, review the application's synchronization status, resource events, and controller logs when necessary.

Advantages and Limitations

Advantages

• Predictable deployment sequencing: Resources can be grouped into ordered synchronization waves.

• Pre-deployment checks: Hooks can validate prerequisites before normal synchronization begins.

• Post-deployment verification: Smoke tests can run after the application has synchronized successfully.

• Improved failure visibility: Failed hooks can stop a deployment and provide diagnostic information.

• Flexible deployment workflows: Different hook types support different stages of the release process.

Limitations

• Additional configuration: Waves, hooks, and cleanup policies require maintenance.

• Failed hooks can block releases: A migration or validation failure may prevent synchronization from completing.

• No automatic rollback of external changes: A failed hook does not necessarily undo database changes or other side effects.

• Dependency complexity: Incorrect wave assignments or health checks can create confusing synchronization behavior.

• Idempotency requirements: Hooks may run again during later synchronization operations, so their effects must be considered carefully.

Conclusion

Argo CD sync waves and resource hooks provide practical mechanisms for controlling the order of Kubernetes deployments. Sync waves organize resources into ordered groups, while PreSync, Sync, and PostSync hooks allow developers to execute tasks at specific stages of synchronization.

For production applications, a common pattern is to run database migrations before normal synchronization, apply configuration before deploying the application, and execute smoke tests after the new resources are healthy.

These mechanisms work best when combined with idempotent migration logic, backward-compatible database changes, clear health checks, sensible cleanup policies, and tested recovery procedures. They do not automatically make a deployment transactional, but they help make deployment dependencies explicit and failures easier to investigate.