- Comment
- Reblog
-
Subscribe
Subscribed
Already have a WordPress.com account? Log in now.
- Privacy
# Inject 5s latency into 50% of scrape requests for 2 minutes curl -X POST http://localhost:9091/inject/latency \ -d '"duration":"2m","percent":50,"delay":"5s"' If you run Prometheus Operator, pair it with Chaos Mesh (CNCF project) and a NetworkChaos experiment:
In short: How to Run Prometheus Chaos Edition (Step-by-Step) prometheus chaos edition
apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: prometheus-slow-scrape spec: action: delay mode: all selector: pods: prometheus-ns: - prometheus-server-0 delay: latency: "3s" correlation: "100" jitter: "1s" duration: "5m" Apply with kubectl apply -f chaos.yaml . Prometheus will now see all outbound scrape requests delayed. One of the most insidious PCE experiments is injecting malformed OpenMetrics data. # Inject 5s latency into 50% of scrape
The result? A telemetry system that survives real network partitions, overloaded exporters, and misconfigured rules. And a team that actually knows how to debug their monitoring stack under pressure. The result
Enter – a little-known, experimental tool designed to do the unthinkable: intentionally break your Prometheus deployment so you can fix it before a real disaster.
Before we dive into code, let’s address the obvious question: Why would I voluntarily break my monitoring?
Elwood 5566