Drift Detection
8 min
Overview
Kubegrade Drift Detection identifies mismatches between your live cluster state and intended state (Git/IaC/policy baselines) so teams can correct drift before it becomes an outage or audit problem.
Drift sources (cluster vs IaC vs Git)
Cluster vs IaC drift
Live cluster resources differ from Terraform/Helm/Kustomize-defined state.
Cluster vs Git drift
Git-tracked manifests/configs no longer match what is running in the cluster.
Policy baseline drift
Cluster/workload configuration diverges from internal standards or approved baselines.
Cross-environment drift
Staging/prod environments diverge unintentionally over time.
Drift policies (what to flag / severity)
What to flag (examples)
- Manual changes outside GitOps
- Resource limits/requests changed in cluster only
- NetworkPolicy differences
- Ingress/service selector mismatches
- Deprecated/unsupported API usage
- Missing labels/annotations required by policy
- Image tag/version deviations
- RBAC drift
Severity model (recommended)
- Critical: security/compliance risk, production impact likely
- High: likely operational risk or audit issue
- Medium: inconsistency or future risk
- Low: informational / cosmetic drift
Drift remediation via PR
Kubegrade should support turning drift findings into PRs that restore desired state.
Remediation modes
- Reconcile cluster to Git/IaC (preferred in GitOps environments)
- Update Git/IaC to reflect approved runtime changes (controlled exceptions)
- Open review-only recommendation without PR
PR contents
- Exact diff to restore alignment
- Drift classification and severity
- Evidence of where mismatch was detected
- Impact notes (if applicable)
Exclusions and suppression rules
Not all drift should trigger alerts.
Common exclusions
- Auto-generated labels/annotations
- Runtime status fields
- Ephemeral resources
- Known provider-managed mutations
- Approved temporary overrides (time-bound)
Suppression best practices
- Scope suppressions narrowly (resource/namespace/path)
- Add expiration date
- Require reason/comment
- Track suppressions in audit logs