Troubleshooting
9 min
Overview
Kubegrade Troubleshooting turns Kubernetes troubleshooting from a reactive, manual process into a guided, context-rich workflow with AI, automation and human control.
Guided investigations (what Kubegrade inspects)
Kubegrade agents investigate issues using correlated signals across Kubernetes layers.
Common inspection domains
- Pod and container status
- Events and recent changes
- Resource pressure (CPU/memory)
- Scheduling constraints
- Service/endpoints/ingress routing
- DNS/network indicators
- ConfigMaps/Secrets references and misconfigurations
- Dependency relationships across workloads/services
Common incident playbooks
CrashLoopBackOff
Kubegrade checks:
- Container logs
- Exit codes
- Liveness/readiness probe failures
- Missing env/config/secret references
- Recent config changes
Typical outputs:
- Root-cause candidates
- Impacted dependent services
- Proposed config or probe adjustments via PR
OOMKilled / resource exhaustion
Kubegrade checks:
- Memory limits/requests
- Actual usage patterns
- Recent release changes
- Node pressure and eviction signals
Typical outputs:
- Right-size recommendations
- HPA/resource tuning suggestions
- PR-ready config updates (if enabled)
5xx / ingress/service routing issues
Kubegrade checks:
- Ingress/controller status
- Service selectors/endpoints
- Pod readiness
- Upstream dependency health
- DNS/network path issues
DNS / network connectivity issues
Kubegrade checks:
- CoreDNS status
- Service DNS resolution
- NetworkPolicy constraints
- Endpoint availability
- Namespace/service name mismatches
Scheduling failures
Kubegrade checks:
- Resource requests vs node capacity
- Taints/tolerations
- Node selectors / affinity
- PDB interactions
Evidence collection + timelines
Evidence types
- Events
- Status transitions
- Metrics snapshots
- Dependency graph changes
- Config diffs / Git changes
- Alert triggers (if integrated)
Timelines
A useful incident timeline should show:
- First observed symptom
- Related alerts and state transitions
- Changes preceding the incident
- Actions taken (manual or Kubegrade-assisted)
- Recovery confirmation
Suggested remediations → PR workflow
Kubegrade can convert troubleshooting findings into reviewable changes.
Flow:
- Investigation identifies root-cause candidates
- Kubegrade proposes remediation options
- User selects/edits preferred option
- Kubegrade generates PR against Git/IaC source
- User reviews and merges
- GitOps applies
- Kubegrade verifies recovery signals
Post-incident review / export
After resolution, the Kubegrade Assistant agent can support review and reporting.
Post-incident review contents
- Incident summary
- Affected clusters/namespaces/services
- Timeline of events and changes
- Root cause (confirmed/probable)
- Actions taken
- PR links / approvals
- Prevention follow-ups