Rules

Application Metrics

3.531s ago

200.4us

Rule State Error Last Evaluation Evaluation Time
alert: HTTP 5xx Errors expr: (sum by(app, status) (increase(http_request_duration_seconds_count{route!="/health",status="5XX"}[30s]))) > 3 for: 1m labels: team: devops annotations: summary: App {{$labels.app}} has {{ $value }} HTTP {{$labels.status}} Errors ok 3.531s ago 191.7us

BemobQueueProdEbIpSync

3.579s ago

1.408ms

Rule State Error Last Evaluation Evaluation Time
alert: BemobQueueProdEbIpSyncStale expr: (time() - kube_cronjob_status_last_successful_time{cronjob=~"bemob-queue-(prod|staging)-eb-ip-sync",namespace="queue"}) > 1200 for: 5m labels: severity: critical team: devops annotations: description: |- CronJob `bemob-queue-prod-eb-ip-sync` (ns=queue) last successful run was {{ $value | humanizeDuration }} ago. While it's stale, K8s Service `bemob-queue-prod-eb` will NOT auto-update if the EB instance IP changes, breaking api/postback/admin → legacy queue connectivity. VALUE = {{ $value }} LABELS = {{ $labels }} summary: bemob-queue-prod-eb-ip-sync hasn't succeeded in 20+ minutes ok 3.579s ago 380.7us
alert: BemobQueueProdEbIpSyncSuspended expr: kube_cronjob_spec_suspend{cronjob=~"bemob-queue-(prod|staging)-eb-ip-sync",namespace="queue"} == 1 for: 10m labels: severity: warning team: devops annotations: description: |- CronJob `bemob-queue-prod-eb-ip-sync` (ns=queue) has been suspended for 10+ minutes. While suspended, the K8s Service `bemob-queue-prod-eb` is frozen on its last-known IP and will not track EB instance changes. LABELS = {{ $labels }} summary: bemob-queue-prod-eb-ip-sync CronJob is suspended ok 3.579s ago 183.1us
alert: BemobQueueProdEbIpSyncJobFailed expr: kube_job_status_failed{job_name=~"bemob-queue-(prod|staging)-eb-ip-sync-.*",namespace="queue"} > 0 for: 5m labels: severity: warning team: devops annotations: description: |- A scheduled run of `bemob-queue-prod-eb-ip-sync` failed. Single failures are usually fine (AWS API blip, transient K8s API issue) — the next scheduled run in <5 min will recover. Investigate if persistent. Logs: kubectl -n queue logs job/{{ $labels.job_name }} LABELS = {{ $labels }} summary: bemob-queue-prod-eb-ip-sync job `{{ $labels.job_name }}` failed ok 3.579s ago 833us

Clickhouse

2.926s ago

694.6us

Rule State Error Last Evaluation Evaluation Time
alert: Clickhouse Uptime expr: min by(instance) (ClickHouseAsyncMetrics_Uptime) <= 300 for: 1m labels: team: devops annotations: summary: Clickhouse {{$labels.job}} has {{ $value }} seconds uptime. ok 2.926s ago 251.2us
alert: Clickhouse Replicas Max Delay expr: ClickHouseAsyncMetrics_ReplicasMaxAbsoluteDelay >= 30 for: 1m labels: team: devops annotations: summary: Clickhouse {{$labels.job}} has {{ $value }} seconds replicas max delay. ok 2.926s ago 94.29us
alert: Clickhouse Replicas Max Queue Size expr: ClickHouseAsyncMetrics_ReplicasSumQueueSize >= 10 for: 1m labels: team: devops annotations: summary: Clickhouse {{$labels.job}} has {{ $value }} replicas max queue size. ok 2.926s ago 90.53us
alert: Clickhouse Replicas Max Inserts In Queue expr: ClickHouseAsyncMetrics_ReplicasMaxInsertsInQueue >= 10 for: 1m labels: team: devops annotations: summary: Clickhouse {{$labels.job}} has {{ $value }} replicas max inserts in queue. ok 2.926s ago 78.14us
alert: Clickhouse Replicas Max Merges In Queue expr: ClickHouseAsyncMetrics_ReplicasMaxMergesInQueue >= 10 for: 1m labels: team: devops annotations: summary: Clickhouse {{$labels.job}} has {{ $value }} replicas max merges in queue. ok 2.926s ago 81.66us
alert: Clickhouse Distributed Files To Insert expr: ClickHouseMetrics_DistributedFilesToInsert >= 10 for: 1m labels: team: devops annotations: summary: Clickhouse {{$labels.job}} has {{ $value }} distributed files to insert. ok 2.926s ago 87.25us

Deployment

4.38s ago

1.724ms

Rule State Error Last Evaluation Evaluation Time
alert: Deployment at 0 Replicas expr: sum by(deployment, namespace) (kube_deployment_status_replicas{pod_template_hash=""}) < 1 for: 1m labels: team: devops annotations: summary: Deployment {{$labels.deployment}} in {{$labels.namespace}} is currently having no pods running ok 4.38s ago 1.233ms
alert: DaemonSet Unavailable expr: sum by(daemonset, namespace) (kube_daemonset_status_number_unavailable) > 0 for: 5m labels: team: devops annotations: summary: DaemonSet {{$labels.daemonset}} in {{$labels.namespace}} is currently unavailable ok 4.379s ago 269.2us
alert: HPA Scaling Limited expr: (sum by(hpa, namespace) (kube_hpa_status_condition{condition="ScalingLimited",status="true"})) == 1 for: 1m labels: team: devops annotations: summary: HPA named {{$labels.hpa}} in {{$labels.namespace}} namespace has reached scaling limited state ok 4.379s ago 105.1us
alert: HPA at MaxCapacity expr: ((sum by(hpa, namespace) (kube_hpa_spec_max_replicas)) - (sum by(hpa, namespace) (kube_hpa_status_current_replicas))) == 0 for: 1m labels: team: devops annotations: summary: HPA named {{$labels.hpa}} in {{$labels.namespace}} namespace is running at Max Capacity ok 4.379s ago 106.3us

Hosts

4.661s ago

164.1ms

Rule State Error Last Evaluation Evaluation Time
alert: HostOutOfMemory expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10 for: 2m labels: severity: warning team: devops annotations: description: |- Node memory is filling up (< 10% left) VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host out of memory (instance {{ $labels.instance }}) ok 4.662s ago 1.798ms
alert: HostMemoryUnderMemoryPressure expr: rate(node_vmstat_pgmajfault[1m]) > 1000 for: 2m labels: severity: warning team: devops annotations: description: |- The node is under heavy memory pressure. High rate of major page faults VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host memory under memory pressure (instance {{ $labels.instance }}) ok 4.66s ago 748us
alert: HostUnusualNetworkThroughputIn expr: sum by(instance) (rate(node_network_receive_bytes_total[2m])) / 1024 / 1024 > 100 for: 5m labels: severity: warning team: devops annotations: description: |- Host network interfaces are probably receiving too much data (> 100 MB/s) VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host unusual network throughput in (instance {{ $labels.instance }}) ok 4.659s ago 4.668ms
alert: HostUnusualNetworkThroughputOut expr: sum by(instance) (rate(node_network_transmit_bytes_total[2m])) / 1024 / 1024 > 100 for: 5m labels: severity: warning team: devops annotations: description: |- Host network interfaces are probably sending too much data (> 100 MB/s) VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host unusual network throughput out (instance {{ $labels.instance }}) ok 4.655s ago 4.596ms
alert: KinesisEventsDirectoryHighUsage expr: (node_filesystem_avail_bytes{mountpoint="/events/production"} * 100) / node_filesystem_size_bytes{mountpoint="/events/production"} < 30 for: 5m labels: severity: warning team: devops annotations: description: |- Events directory /events/production is running low on space (< 30% free). This may indicate kinesis-agent issues. VALUE = {{ $value }}% LABELS = {{ $labels }} summary: Kinesis events directory high disk usage (instance {{ $labels.instance }}) ok 4.65s ago 94.98us
alert: KinesisEventsDirectoryCritical expr: (node_filesystem_avail_bytes{mountpoint="/events/production"} * 100) / node_filesystem_size_bytes{mountpoint="/events/production"} < 15 for: 2m labels: severity: critical team: devops annotations: description: |- Events directory /events/production is critically low on space (< 15% free). Emergency backup to S3 may be triggered. VALUE = {{ $value }}% LABELS = {{ $labels }} summary: Kinesis events directory critical disk usage (instance {{ $labels.instance }}) ok 4.65s ago 61.09us
alert: HostOutOfDiskSpace expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and on(instance, device, mountpoint) node_filesystem_readonly == 0 for: 2m labels: severity: warning team: devops annotations: description: |- Disk is almost full (< 10% left) VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host out of disk space (instance {{ $labels.instance }}) ok 4.651s ago 28.05ms
alert: HostDiskWillFillIn24Hours expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10 and on(instance, device, mountpoint) predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs"}[1h], 24 * 3600) < 0 and on(instance, device, mountpoint) node_filesystem_readonly == 0 for: 2m labels: severity: warning team: devops annotations: description: |- Filesystem is predicted to run out of space within the next 24 hours at current write rate VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host disk will fill in 24 hours (instance {{ $labels.instance }}) ok 4.623s ago 49.5ms
alert: HostOutOfInodes expr: node_filesystem_files_free{mountpoint="/rootfs"} / node_filesystem_files{mountpoint="/rootfs"} * 100 < 10 and on(instance, device, mountpoint) node_filesystem_readonly{mountpoint="/rootfs"} == 0 for: 2m labels: severity: warning team: devops annotations: description: |- Disk is almost running out of available inodes (< 10% left) VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host out of inodes (instance {{ $labels.instance }}) ok 4.573s ago 153.5us
alert: HostInodesWillFillIn24Hours expr: node_filesystem_files_free{mountpoint="/rootfs"} / node_filesystem_files{mountpoint="/rootfs"} * 100 < 10 and predict_linear(node_filesystem_files_free{mountpoint="/rootfs"}[1h], 24 * 3600) < 0 and on(instance, device, mountpoint) node_filesystem_readonly{mountpoint="/rootfs"} == 0 for: 2m labels: severity: warning team: devops annotations: description: |- Filesystem is predicted to run out of inodes within the next 24 hours at current write rate VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host inodes will fill in 24 hours (instance {{ $labels.instance }}) ok 4.573s ago 163us
alert: HostUnusualDiskReadLatency expr: rate(node_disk_read_time_seconds_total[1m]) / rate(node_disk_reads_completed_total[1m]) > 0.1 and rate(node_disk_reads_completed_total[1m]) > 0 for: 2m labels: severity: warning team: devops annotations: description: |- Disk latency is growing (read operations > 100ms) VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host unusual disk read latency (instance {{ $labels.instance }}) ok 4.573s ago 2.538ms
alert: HostUnusualDiskWriteLatency expr: rate(node_disk_write_time_seconds_total[1m]) / rate(node_disk_writes_completed_total[1m]) > 0.1 and rate(node_disk_writes_completed_total[1m]) > 0 for: 2m labels: severity: warning team: devops annotations: description: |- Disk latency is growing (write operations > 100ms) VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host unusual disk write latency (instance {{ $labels.instance }}) ok 4.571s ago 3.664ms
alert: HostHighCpuLoad expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 80 labels: severity: warning team: devops annotations: description: |- CPU load is > 80% VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host high CPU load (instance {{ $labels.instance }}) ok 4.568s ago 23.64ms
alert: HostCpuStealNoisyNeighbor expr: avg by(instance) (rate(node_cpu_seconds_total{mode="steal"}[5m])) * 100 > 20 labels: severity: warning team: devops annotations: description: |- CPU steal is > 20%. A noisy neighbor is killing VM performances or a spot instance may be out of credit. VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host CPU steal noisy neighbor (instance {{ $labels.instance }}) ok 4.544s ago 2.778ms
alert: HostSwapIsFillingUp expr: (1 - (node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes)) * 100 > 80 for: 2m labels: severity: warning team: devops annotations: description: |- Swap is filling up (>80%) VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host swap is filling up (instance {{ $labels.instance }}) ok 4.542s ago 1.362ms
alert: HostPhysicalComponentTooHot expr: node_hwmon_temp_celsius > 75 for: 5m labels: severity: warning team: devops annotations: description: |- Physical hardware component too hot VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host physical component too hot (instance {{ $labels.instance }}) ok 4.54s ago 453.2us
alert: HostNodeOvertemperatureAlarm expr: node_hwmon_temp_crit_alarm_celsius == 1 labels: severity: critical team: devops annotations: description: |- Physical node temperature alarm triggered VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host node overtemperature alarm (instance {{ $labels.instance }}) ok 4.54s ago 44.58us
alert: HostRaidArrayGotInactive expr: node_md_state{state="inactive"} > 0 labels: severity: critical team: devops annotations: description: |- RAID array {{ $labels.device }} is in degraded state due to one or more disks failures. Number of spare drives is insufficient to fix issue automatically. VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host RAID array got inactive (instance {{ $labels.instance }}) ok 4.54s ago 41.07us
alert: HostRaidDiskFailure expr: node_md_disks{state="failed"} > 0 for: 2m labels: severity: warning team: devops annotations: description: |- At least one device in RAID array on {{ $labels.instance }} failed. Array {{ $labels.md_device }} needs attention and possibly a disk swap VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host RAID disk failure (instance {{ $labels.instance }}) ok 4.54s ago 48.85us
alert: HostEdacCorrectableErrorsDetected expr: increase(node_edac_correctable_errors_total[1m]) > 0 labels: severity: info team: devops annotations: description: |- Host {{ $labels.instance }} has had {{ printf "%.0f" $value }} correctable memory errors reported by EDAC in the last 5 minutes. VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host EDAC Correctable Errors detected (instance {{ $labels.instance }}) ok 4.54s ago 51.17us
alert: HostEdacUncorrectableErrorsDetected expr: node_edac_uncorrectable_errors_total > 0 labels: severity: warning team: devops annotations: description: |- Host {{ $labels.instance }} has had {{ printf "%.0f" $value }} uncorrectable memory errors reported by EDAC in the last 5 minutes. VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host EDAC Uncorrectable Errors detected (instance {{ $labels.instance }}) ok 4.541s ago 34.48us
alert: HostNetworkReceiveErrors expr: rate(node_network_receive_errs_total[2m]) / rate(node_network_receive_packets_total[2m]) > 0.01 for: 2m labels: severity: warning team: devops annotations: description: |- Host {{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf "%.0f" $value }} receive errors in the last five minutes. VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host Network Receive Errors (instance {{ $labels.instance }}) ok 4.541s ago 7.835ms
alert: HostNetworkTransmitErrors expr: rate(node_network_transmit_errs_total[2m]) / rate(node_network_transmit_packets_total[2m]) > 0.01 for: 2m labels: severity: warning team: devops annotations: description: |- Host {{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf "%.0f" $value }} transmit errors in the last five minutes. VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host Network Transmit Errors (instance {{ $labels.instance }}) ok 4.533s ago 8.891ms
alert: HostNetworkInterfaceSaturated expr: (rate(node_network_receive_bytes_total{device!~"^tap.*"}[1m]) + rate(node_network_transmit_bytes_total{device!~"^tap.*"}[1m])) / node_network_speed_bytes{device!~"^tap.*"} > 0.8 for: 1m labels: severity: warning team: devops annotations: description: |- The network interface "{{ $labels.interface }}" on "{{ $labels.instance }}" is getting overloaded. VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host Network Interface Saturated (instance {{ $labels.instance }}) ok 4.524s ago 11.78ms
alert: HostConntrackLimit expr: node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 0.8 for: 5m labels: severity: warning team: devops annotations: description: |- The number of conntrack is approching limit VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host conntrack limit (instance {{ $labels.instance }}) ok 4.513s ago 1.358ms
alert: HostClockSkew expr: (node_timex_offset_seconds > 0.05 and deriv(node_timex_offset_seconds[5m]) >= 0) or (node_timex_offset_seconds < -0.05 and deriv(node_timex_offset_seconds[5m]) <= 0) for: 2m labels: severity: warning team: devops annotations: description: |- Clock skew detected. Clock is out of sync. VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host clock skew (instance {{ $labels.instance }}) ok 4.511s ago 7.048ms
alert: HostClockNotSynchronising expr: min_over_time(node_timex_sync_status[1m]) == 0 and node_timex_maxerror_seconds >= 16 for: 2m labels: severity: warning team: devops annotations: description: |- Clock not synchronising. VALUE = {{ $value }} LABELS = {{ $labels }} summary: Host clock not synchronising (instance {{ $labels.instance }}) ok 4.505s ago 2.658ms

MySQL

2.665s ago

1.555ms

Rule State Error Last Evaluation Evaluation Time
alert: MysqlDown expr: mysql_up == 0 labels: severity: critical annotations: description: |- MySQL instance is down on {{ $labels.instance }} VALUE = {{ $value }} LABELS = {{ $labels }} summary: MySQL down (instance {{ $labels.instance }}) ok 2.665s ago 206.2us
alert: MysqlTooManyConnections(>80%) expr: avg by(instance) (rate(mysql_global_status_threads_connected[1m])) / avg by(instance) (mysql_global_variables_max_connections) * 100 > 80 for: 2m labels: severity: warning annotations: description: |- More than 80% of MySQL connections are in use on {{ $labels.instance }} VALUE = {{ $value }} LABELS = {{ $labels }} summary: MySQL too many connections (> 80%) (instance {{ $labels.instance }}) ok 2.665s ago 259.5us
alert: MysqlHighThreadsRunning expr: avg by(instance) (rate(mysql_global_status_threads_running[1m])) / avg by(instance) (mysql_global_variables_max_connections) * 100 > 60 for: 2m labels: severity: warning annotations: description: |- More than 60% of MySQL connections are in running state on {{ $labels.instance }} VALUE = {{ $value }} LABELS = {{ $labels }} summary: MySQL high threads running (instance {{ $labels.instance }}) ok 2.665s ago 195.6us
alert: MysqlSlaveIoThreadNotRunning expr: mysql_slave_status_master_server_id > 0 and on(instance) mysql_slave_status_slave_io_running == 0 labels: severity: critical annotations: description: |- MySQL Slave IO thread not running on {{ $labels.instance }} VALUE = {{ $value }} LABELS = {{ $labels }} summary: MySQL Slave IO thread not running (instance {{ $labels.instance }}) ok 2.665s ago 144.2us
alert: MysqlSlaveSqlThreadNotRunning expr: mysql_slave_status_master_server_id > 0 and on(instance) mysql_slave_status_slave_sql_running == 0 labels: severity: critical annotations: description: |- MySQL Slave SQL thread not running on {{ $labels.instance }} VALUE = {{ $value }} LABELS = {{ $labels }} summary: MySQL Slave SQL thread not running (instance {{ $labels.instance }}) ok 2.665s ago 162.6us
alert: MysqlSlaveReplicationLag expr: mysql_slave_status_master_server_id > 0 and on(instance) (mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay) > 30 for: 1m labels: severity: critical annotations: description: |- MySQL replication lag on {{ $labels.instance }} VALUE = {{ $value }} LABELS = {{ $labels }} summary: MySQL Slave replication lag (instance {{ $labels.instance }}) ok 2.665s ago 196.8us
alert: MysqlSlowQueries expr: increase(mysql_global_status_slow_queries[1m]) > 0 for: 2m labels: severity: warning annotations: description: |- MySQL server mysql has some new slow query. VALUE = {{ $value }} LABELS = {{ $labels }} summary: MySQL slow queries (instance {{ $labels.instance }}) ok 2.665s ago 115.8us
alert: MysqlInnodbLogWaits expr: rate(mysql_global_status_innodb_log_waits[15m]) > 10 labels: severity: warning annotations: description: |- MySQL innodb log writes stalling VALUE = {{ $value }} LABELS = {{ $labels }} summary: MySQL InnoDB log waits (instance {{ $labels.instance }}) ok 2.665s ago 167.7us
alert: MysqlRestarted expr: mysql_global_status_uptime < 60 labels: severity: info annotations: description: |- MySQL has just been restarted, less than one minute ago on {{ $labels.instance }}. VALUE = {{ $value }} LABELS = {{ $labels }} summary: MySQL restarted (instance {{ $labels.instance }}) ok 2.665s ago 92.81us

PangolinHost

706ms ago

793.4us

Rule State Error Last Evaluation Evaluation Time
alert: PangolinHostDown expr: up{job="pangolin-host"} == 0 for: 2m labels: severity: critical team: devops annotations: description: |- Prometheus has not been able to scrape node_exporter on the Pangolin server (us-west-1b, EIP 54.151.74.26) for 2+ minutes. Either the host is down, node_exporter has stopped, or SG sg-0a4bbe4d623465337 was changed and blocks the EKS NAT IPs (52.86.29.40, 54.85.142.68). Pangolin carries the public ingress for 30+ bemobcloud.com hostnames, so if the host is actually down, expect a platform-wide outage. LABELS = {{ $labels }} summary: Pangolin host node_exporter unreachable ok 706ms ago 198.8us
alert: PangolinHostMemoryCritical expr: node_memory_MemAvailable_bytes{job="pangolin-host"} < 256 * 1024 * 1024 for: 1m labels: severity: critical team: devops annotations: description: |- Available memory on the Pangolin server is {{ $value | humanize1024 }}B for 1+ minute. The 2026-05-11 outage cascaded from exactly this condition: docker daemon wedges, sshd cannot fork, traffic stops. ACT IMMEDIATELY — SSH in and restart the heaviest container, or reboot via `aws ec2 reboot-instances --region us-west-1 --instance-ids i-09be446891581e4eb`. LABELS = {{ $labels }} summary: Pangolin host has less than 256 MiB available memory ok 706ms ago 131.1us
alert: PangolinHostMemoryLow expr: node_memory_MemAvailable_bytes{job="pangolin-host"} < 1 * 1024 * 1024 * 1024 for: 5m labels: severity: warning team: devops annotations: description: |- Available memory on the Pangolin server has been below 1 GiB for 5+ minutes (current {{ $value | humanize1024 }}B). Investigate which container is growing. The host has 4 GiB swap — sustained pressure here will start using it and may trip PangolinHostSwapHigh next. LABELS = {{ $labels }} summary: Pangolin host has less than 1 GiB available memory ok 706ms ago 109us
alert: PangolinHostSwapHigh expr: (node_memory_SwapTotal_bytes{job="pangolin-host"} - node_memory_SwapFree_bytes{job="pangolin-host"}) / node_memory_SwapTotal_bytes{job="pangolin-host"} > 0.5 for: 5m labels: severity: warning team: devops annotations: description: |- Swap usage is {{ $value | humanizePercentage }} for 5+ minutes. Swap is the OOM safety net — if it fills, OOM-killer takes over and may kill traefik, docker daemon, or other critical processes. Investigate memory growth or plan an instance-type upgrade. LABELS = {{ $labels }} summary: Pangolin host swap usage above 50% ok 706ms ago 157.2us
alert: PangolinHostCPUHigh expr: 1 - avg by(instance) (rate(node_cpu_seconds_total{job="pangolin-host",mode="idle"}[5m])) > 0.8 for: 10m labels: severity: warning team: devops annotations: description: |- Average CPU utilization across all cores is {{ $value | humanizePercentage }} for 10+ minutes. t3.large baseline is 30% — sustained use above that burns CPU credits and will eventually throttle. The 2026-05-11 outage was preceded by a CPU spike to ~64% sustained as traffic stalled. LABELS = {{ $labels }} summary: Pangolin host sustained CPU above 80% ok 706ms ago 185.1us

Pods

1.691s ago

11.5ms

Rule State Error Last Evaluation Evaluation Time
alert: Container restarted expr: sum by(pod, namespace, container) (increase(kube_pod_container_status_restarts_total[30s])) > 0 labels: team: devops annotations: summary: Container named {{$labels.container}} in {{$labels.pod}} in {{$labels.namespace}} was restarted ok 1.691s ago 11.49ms

Redis

3.219s ago

3.913ms

Rule State Error Last Evaluation Evaluation Time
alert: RedisDown expr: redis_up == 0 labels: severity: critical team: devops annotations: description: |- Redis instance is down VALUE = {{ $value }} LABELS = {{ $labels }} summary: Redis down (instance {{ $labels.instance }}) ok 3.219s ago 548.5us
alert: RedisReplicationBroken expr: delta(redis_connected_slaves[1m]) < 0 labels: severity: critical team: devops annotations: description: |- Redis instance lost a slave VALUE = {{ $value }} LABELS = {{ $labels }} summary: Redis replication broken (instance {{ $labels.instance }}) ok 3.218s ago 527.4us
alert: RedisClusterFlapping expr: changes(redis_connected_slaves[1m]) > 1 for: 2m labels: severity: critical team: devops annotations: description: |- Changes have been detected in Redis replica connection. This can occur when replica nodes lose connection to the master and reconnect (a.k.a flapping). VALUE = {{ $value }} LABELS = {{ $labels }} summary: Redis cluster flapping (instance {{ $labels.instance }}) ok 3.218s ago 422.4us
alert: RedisOutOfSystemMemory expr: redis_memory_used_bytes / redis_total_system_memory_bytes * 100 > 90 for: 2m labels: severity: warning team: devops annotations: description: |- Redis is running out of system memory (> 90%) VALUE = {{ $value }} LABELS = {{ $labels }} summary: Redis out of system memory (instance {{ $labels.instance }}) ok 3.218s ago 554.7us
alert: RedisOutOfConfiguredMaxmemory expr: redis_memory_used_bytes / redis_memory_max_bytes * 100 > 90 for: 2m labels: severity: warning team: devops annotations: description: |- Redis is running out of configured maxmemory (> 90%) VALUE = {{ $value }} LABELS = {{ $labels }} summary: Redis out of configured maxmemory (instance {{ $labels.instance }}) ok 3.217s ago 965.3us
alert: RedisTooManyConnections expr: redis_connected_clients > 2000 for: 2m labels: severity: warning team: devops annotations: description: |- Redis instance has too many connections VALUE = {{ $value }} LABELS = {{ $labels }} summary: Redis too many connections (instance {{ $labels.instance }}) ok 3.216s ago 416.9us
alert: RedisRejectedConnections expr: increase(redis_rejected_connections_total[1m]) > 0 labels: severity: critical team: devops annotations: description: |- Some connections to Redis has been rejected VALUE = {{ $value }} LABELS = {{ $labels }} summary: Redis rejected connections (instance {{ $labels.instance }}) ok 3.216s ago 462.8us

Zookeeper

1.12s ago

423.6us

Rule State Error Last Evaluation Evaluation Time
alert: ZookeeperDown expr: zk_up == 0 labels: severity: critical team: devops annotations: description: |- Zookeeper down on instance {{ $labels.instance }} VALUE = {{ $value }} LABELS = {{ $labels }} summary: Zookeeper Down (instance {{ $labels.instance }}) ok 1.12s ago 171.3us
alert: ZookeeperMissingLeader expr: sum(zk_server_leader) == 0 labels: severity: critical team: devops annotations: description: |- Zookeeper cluster has no node marked as leader VALUE = {{ $value }} LABELS = {{ $labels }} summary: Zookeeper missing leader (instance {{ $labels.instance }}) ok 1.12s ago 96.26us
alert: ZookeeperTooManyLeaders expr: sum(zk_server_leader) > 1 labels: severity: critical team: devops annotations: description: |- Zookeeper cluster has too many nodes marked as leader VALUE = {{ $value }} LABELS = {{ $labels }} summary: Zookeeper Too Many Leaders (instance {{ $labels.instance }}) ok 1.12s ago 74.88us
alert: ZookeeperNotOk expr: zk_ruok == 0 for: 3m labels: severity: warning team: devops annotations: description: |- Zookeeper instance is not ok VALUE = {{ $value }} LABELS = {{ $labels }} summary: Zookeeper Not Ok (instance {{ $labels.instance }}) ok 1.12s ago 72.9us