GitOps Remediation
11 min
Overview
GitOps Remediation is Kubegrade’s workflow for converting operational findings (drift, incidents, upgrade blockers, optimization recommendations) into reviewable pull requests and applying changes through your existing GitOps system.
Supported repo layouts
Common layouts
- Monorepo (all environments/services)
- Environment-per-folder
- Service-per-folder
- Repo-per-service
- Platform repo + app repos split
- Helm values repos
- Terraform module + environment composition repos
Best practice
Require users to define:
- Repo path mappings per cluster/environment
- File ownership / CODEOWNERS expectations
- Branch strategy
PR generation rules (naming, commit strategy, reviewers)
Branch naming (recommended)
kubegrade/<module>/<cluster>/<issue-or-workflow-id>
Examples:
kubegrade/drift/prod-eu1/restore-nginx-limits
kubegrade/upgrade/staging/1-29-preflight-fixes
Commit strategy
- One commit per logical remediation (preferred)
- Squash option for noisy generated changes
- Signed commits if required by org policy
Reviewers
- Auto-assign based on CODEOWNERS
- Add platform approvers for prod scopes
- Tag relevant service owners for workload-level changes
Policy checks before PR
Before opening a PR, Kubegrade validates:
- Scope and permissions
- Policy compliance
- Environment restrictions
- Change-type restrictions
- Required approvals
- Optional dry-run validation/tests (if integrated)
If checks fail:
- Open recommendation without PR
- Show blocking reasons and remediation steps
Merge-to-apply mechanics
Typical flow
- PR merged in Git provider
- GitOps tool (Argo CD / Flux) detects commit
- GitOps sync applies changes to cluster
- Kubegrade watches rollout status / post-change signals
- Audit trail links PR → sync → outcome
Recommended controls
- Sync windows
- Manual sync for production
- Health checks before marking complete
Rollback PRs
Rollback approaches
- Revert generated PR commit(s)
- Generate rollback PR from known-good state
- Partial rollback for specific resources (advanced / policy-controlled)
Rollback triggers (examples)
- Failed post-change health checks
- Elevated error rates
- Dependency graph degradation
- SLO breach detected via integrations
Auditability: change provenance
Every GitOps remediation should be traceable.
Provenance fields (recommended)
- Workflow source (drift/troubleshoot/upgrade/etc.)
- Cluster/environment/namespace
- Agent/workflow version
- Trigger type (manual/scheduled/alert-driven)
- Approvers and timestamps
- PR link and commit SHA
- GitOps sync result
- Verification result