跳转至

K8s 集群核心组件日志告警汇总排查报告

报告类型:只读巡检报告(按用户要求"先统一罗列,不要做其它操作",本报告不含修复动作) 执行时间:2026-08-24 11:00 ~ 11:30 执行人:rian@192.168.0.11 (admin 工作站) 集群环境:K8s v1.22.1 / kubeadm 部署 / 3Master(0.2/3/4) + 3Worker(0.5/6/7) / CRI=docker 20.10.8 / CNI=Calico / Storage=NFS(0.13) 采集范围:6 节点 × kubelet + 12 个 static pod(apiserver/scheduler/controller-manager/etcd 各 3 个)+ 12 个 DaemonSet Pod(calico-node/kube-proxy 各 6 个)+ 3 个 Deployment Pod(coredns 2 + calico-kube-controllers 1) 采集方法:kubectl logs --tail=1000(Pod 日志) + journalctl -u kubelet -n 1000(kubelet 日志,SSH 免密并行) 采集数据量:27 个日志文件,总计约 3MB


一、巡检范围与方法论(遵循规则 4.1 故障排查固定顺序)

1.1 排查顺序

  1. 基础健康采集:kubectl get cs + kubectl get nodes + Pod 状态全览
  2. 组件日志采集:static pods / DaemonSet pods / Deployment pods / kubelet journalctl
  3. 连通性验证:所有 Pod 当前 Running + 节点 Ready + BGP Established=5
  4. 故障域区分:按组件类别分层(控制平面/节点级容器/节点级系统服务/附加组件)
  5. 根因结论 + 证据:见第三章告警分类汇总

1.2 采集对象清单(27 个数据源)

类别 数据源 数量 采集方式
控制平面 static pods kube-apiserver × 3 / kube-controller-manager × 3 / kube-scheduler × 3 / etcd × 3 12 kubectl logs --tail=1000
节点级 DaemonSet calico-node × 6 / kube-proxy × 6 12 kubectl logs -c <容器> --tail=1000
集群附加 Deployment coredns × 2 / calico-kube-controllers × 1 3 kubectl logs --tail=1000
节点级系统服务 kubelet × 6 6 journalctl -u kubelet -n 1000(SSH 免密并行)

二、集群整体健康现状(LOG-1 采集)

2.1 节点状态(6/6 Ready)

节点 STATUS ROLES AGE VERSION 内部 IP
k8s-master-01 Ready control-plane,master 218d v1.22.1 192.168.0.2
k8s-master-02 Ready control-plane,master 218d v1.22.1 192.168.0.3
k8s-master-03 Ready control-plane,master 218d v1.22.1 192.168.0.4
k8s-node-01 Ready 218d v1.22.1 192.168.0.5
k8s-node-02 Ready 218d v1.22.1 192.168.0.6
k8s-node-03 Ready 218d v1.22.1 192.168.0.7

2.2 控制平面状态(kubectl get cs)

组件 状态 备注
scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager Healthy OK
etcd-0 Healthy {"health":"true","reason":""}

📌 现象解读(讲清原理):kubectl get cs 显示 scheduler Unhealthy 不是真故障。原因:K8s 1.19+ ComponentStatus 已废弃(命令本身会输出 Warning: v1 ComponentStatus is deprecated in v1.19+),从 K8s 1.20 起 kube-scheduler 默认关闭了 insecure port 10251,所以 kubectl get cs 走 HTTP 探测该端口必然 connection refused。实际 Pod kube-scheduler-k8s-master-* 全部 1/1 Running,健康。此条非真故障,无需处理。

2.3 重启次数异常(值得关注,非本次告警主因)

Pod RESTARTS 备注
kube-apiserver-k8s-master-01 4087 218 天累积,平均每天约 19 次重启
kube-apiserver-k8s-master-02 4846 同上
kube-apiserver-k8s-master-03 657 相对少
kube-controller-manager-k8s-master-01 6737 最高
kube-controller-manager-k8s-master-02 6472
kube-controller-manager-k8s-master-03 4499
kube-scheduler-k8s-master-01 5947
kube-scheduler-k8s-master-02 5508
kube-scheduler-k8s-master-03 3785 8h ago 才重启一次
etcd-k8s-master-01 2844
etcd-k8s-master-02 53 相对极少
etcd-k8s-master-03 51 相对极少

📌 现象解读:所有组件 47h ago 同步重启了一次(疑似集群整体重启)。但 master-01/02 的 apiserver/scheduler/controller-manager 重启次数高达 4000-6737 次,疑似曾长期处于 CrashLoopBackOff 状态,可能与 etcd 慢磁盘告警强相关(详见 3.3 节)。当前所有 Pod Running,未持续 crash。


三、告警汇总分类(核心结论,按严重度排序)

3.1 🔴 严重告警(HIGH,建议优先关注)

🔴 告警 #1:etcd 慢磁盘告警(1212 条 warn,3 master 全中)

节点 WARN 数 关键告警
etcd-k8s-master-01 414 apply request took too long × 61 / leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk × 96(master-01+02)
etcd-k8s-master-02 383 同上,慢磁盘告警 53 次
etcd-k8s-master-03 415 同上

告警原文(去重后 top 2 类):

61x {"level":"warn","caller":"etcdserver/util.go","msg":"apply request took too long",
     "took":"Nms","expected-duration":"Nms","prefix":"read-only range ",
     "request":"key:\"/registry/health\" ","response":"range_response_count:N size:N"}

96x {"level":"warn","caller":"etcdserver/raft.go","msg":"leader failed to send out heartbeat on time;
     took too long, leader is overloaded likely from slow disk",
     "to":"NeNfNaabNdNcfNe","heartbeat-interval":"Nms",
     "expected-duration":"Nms","exceeded-duration":"N.Nms"}

故障原因(讲清原理): - apply request took too long = etcd 处理读写请求超过预期阈值(10ms),导致性能告警 - leader is overloaded likely from slow disk = etcd leader 节点无法按时发送心跳(heartbeat 超时),etcd 官方明确指出大概率是磁盘 IO 慢 - 这与 master-01/02 的 kube-apiserver/controller-manager/scheduler 重启 4000-6737 次的现象强相关:etcd 慢 → apiserver 写 etcd 超时 → apiserver 健康检查失败 → kubelet 重启 apiserver → 循环往复

疑似根因(待验证,未执行):master 节点磁盘 IO 性能不足(可能因为: 1. master 节点用的是普通 HDD 而非 SSD(建议 ssh k8s-master-01 lsblk -d -o NAME,ROTA,SIZE,MODEL 验证,ROTA=1 是 HDD) 2. master 节点磁盘与其他 IO 密集型服务(如 Lustre OSS)共享(master-03 跑着 lustre_oss) 3. etcd 数据目录未做 IO 隔离(默认在 /var/lib/etcd/)


🔴 告警 #2:k8s-node-02 orphaned pod 清理失败(250 条 warn)

节点:k8s-node-02(Pod IP 192.168.0.6,承载 coredns 2 副本 + calico-kube-controllers)

告警原文(250 条全同一类):

250x E0824 HH:MM:SS.N kubelet_volumes.go:N] "There were many similar errors. Turn up verbosity to see them."
      err="orphaned pod \"322a2355-92c8-4a64-a0c9-f6e9f8705c58\" found, but volumes are still mounted"

故障原因(讲清原理): - orphaned pod = kubelet 发现一个 Pod 对象已经被删除(API server 那边没了),但卷依然挂载在节点上没卸载 - 这是 kubelet 的 Pod 清理 GC 机制卡住,导致: 1. 节点磁盘空间泄漏(旧 PV 没释放) 2. 容器目录残留 /var/lib/kubelet/pods/322a2355-92c8-4a64-a0c9-f6e9f8705c58/ 3. 长期累积会导致节点 kubelet 启动慢 + Pod 启动失败

疑似根因(待验证,未执行):Pod UID 322a2355-92c8-4a64-a0c9-f6e9f8705c58 对应的 Pod 已被删除,但因容器 runtime(docker)或 volume plugin(NFS)umount 卡住,残留卷挂载。建议 ssh k8s-node-02 ls -la /var/lib/kubelet/pods/322a2355-92c8-4a64-a0c9-f6e9f8705c58/volumes/ 看具体挂载点。


3.2 🟡 中等告警(MEDIUM,已知问题或不影响业务)

🟡 告警 #3:Grafana PVC FailedBinding(987 条,1 个 controller-manager)

节点:kube-controller-manager-k8s-master-02(987 条)

告警原文(去重后):

987x I0824 HH:MM:SS event.go:N] "Event occurred" object="monitoring/prometheus-grafana"
      kind="PersistentVolumeClaim" type="Normal" reason="FailedBinding"
      message="no persistent volumes available for this claim and no storage class is set"

故障原因(讲清原理):此告警与之前 TV 终局验证报告里的"Grafana Pending Side Issue"完全一致——kube-prometheus-stack Helm Chart 默认 Grafana.persistence 开启,Deployment 引用 PVC=prometheus-grafana,但该 PVC StorageClass 为空 + 集群无 DefaultStorageClass + 无对应静态 PV → controller-manager 持续尝试绑定失败告警。已知问题,方案在上一份报告中已写明,未执行。

🟡 告警 #4:coredns DNS 查询超时(10 条 ERROR)

节点:coredns-78fcd69978-5gs6p(k8s-node-02)

告警原文:

10x [ERROR] plugin/errors: N N.N. HINFO: read udp N.N.N.N:N->N.N.N.N:N: i/o timeout
 1x [WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API

故障原因(讲清原理): - read udp ... i/o timeout = coredns 向上游 DNS 查询 HINFO 类型记录超时(10s 超时阈值) - 这通常是因为 coredns 配置的上游 DNS(如 /etc/resolv.conf 中的 nameserver)不可达或响应慢 - starting server with unsynced Kubernetes API = coredns 启动时 Kubernetes API 还未同步完成,启动初期短暂告警,已知现象

疑似根因(待验证,未执行):上游 DNS 配置问题,建议 kubectl -n kube-system exec coredns-78fcd69978-5gs6p -c coredns -- cat /etc/resolv.conf 看上游 DNS 是否可达。

🟡 告警 #5:kube-apiserver 偶发 broken pipe / context canceled(11 条 ERROR,3 master 共计)

节点:kube-apiserver-k8s-master-02(5 条)/ master-03(6 条)

告警原文(去重后):

2x apiserver received an error that is not an metavN.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
2x apiserver received an error that is not an metavN.Status: &errors.errorString{s:"context canceled"}: context canceled
1x Error proxying data from client to backend: write tcp N.N.N.N:N->N.N.N.N:N: write: broken pipe
1x apiserver received an error that is not an metavN.Status: context.deadlineExceededError{}: context deadline exceeded

故障原因(讲清原理): - http: Handler timeout = apiserver 处理请求超时(默认 60s) - context canceled = 客户端主动断开连接(kubectl/exec 场景常见) - broken pipe = 客户端 TCP 连接断开后 apiserver 继续写入 - 这些都是 apiserver 在 kubectl exec/logs/proxy 等长连接场景的正常偶发告警,不影响业务


3.3 🟢 低告警(LOW,已知现象或瞬时启动告警)

🟢 告警 #6:8 月 22 日节点启动瞬时告警(约 2161 条,占 kubelet 总告警 68%)

节点:全 6 节点(master-01: 418 / master-02: 587 / master-03: 582 / node-01: 117 / node-03: 457)

告警原文(去重后 top 1):

418-587x E0822 HH:MM:SS kubelet.go:N] "Error getting node" err="node \"k8s-master-N\" not found"
 117-457x E0822 kubelet.go:N] "Error getting node" err="node \"k8s-node-N\" not found"
 22x     E0822 eviction_manager.go:N] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"k8s-master-N\" not found"
 20x     E0822 controller.go:N] failed to ensure lease exists, will retry in 7s, error: Get "https://N.N.N.N:N/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/..."

故障原因(讲清原理):这是 kubeadm 集群节点重启时的已知现象,不是真故障: - 节点启动后,systemd 拉起 kubelet(systemctl enable kubelet 自启) - kubelet 启动后立即尝试从 kube-apiserver 查询自身 Node 对象做注册/更新 - 但 kube-apiserver 还没起来(依赖 etcd,etcd 是 static pod 也要等 docker 起来后才能调度) - 所以 kubelet 在约 1-3 分钟内持续报 "node not found" - 一旦 apiserver 起来,kubelet 立即注册成功,告警停止

结论:属于节点启动顺序的瞬时现象,无需处理。

🟢 告警 #7:calico-node CrashLoopBackOff(13 条,今天 8 月 24 日)

节点:全 6 节点 kubelet 日志(master-01:8 / master-02:6 / master-03:7 / node-01:8 / node-03:8)

告警原文:

E0824 HH:MM:SS pod_workers.go:N] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"calico-node\"
with CrashLoopBackOff: \"back-off 1m20s restarting failed container=calico-node pod=calico-node-...\""

故障原因(讲清原理):此告警与今天执行的 PH 永久治理(kubectl set env daemonset/calico-node IP_AUTODETECTION_METHOD=interface=eth0)强相关: - PH2 触发 DaemonSet 滚动重启,calico-node 一个个重建 - 重建期间偶尔出现 CrashLoopBackOff(back-off 40s/1m20s 重启) - 滚动重启 157 秒后稳定,6 个 Pod 全部 1/1 Running + BGP Established=5

结论:滚动重启期间的瞬时现象,已恢复,无需处理。

🟢 告警 #8:kube-proxy etcd leader 切换(1 条)

节点:kube-proxy-dwwb5(k8s-node-01)

告警原文:

1x E0822 node.go:N] Failed to retrieve node info: etcdserver: leader changed

故障原因(讲清原理):8 月 22 日节点重启时 etcd 集群 leader 切换的瞬时告警,kube-proxy 拿 node info 失败一次后立即重试成功。无需处理。

🟢 告警 #9:calico-kube-controllers Failed to reach apiserver(1 条)

节点:calico-kube-controllers-8db96c76-p8hpb(k8s-node-02)

告警原文:

1x [ERROR][N] main.go N: Failed to reach apiserver error=<nil>

故障原因(讲清原理):calico-kube-controllers 启动时第一次连接 apiserver 偶发失败(error 实际是 <nil>,说明是健康检查瞬时不通),1 秒内重试成功。无需处理。

🟢 告警 #10:scheduler/controller-manager leader election 偶发失败(11 条)

节点:kube-controller-manager-k8s-master-01(5 条)/ master-03(1 条)/ kube-scheduler(无,已正常)

告警原文(去重后):

5x E0822 leaderelection.go:N] error retrieving resource lock kube-system/kube-controller-manager:
   Get "https://N.N.N.N:N/apis/coordination.k8s.io/vN/namespaces/kube-system/leases/kube-controller-manager?timeout=Ns":
   dial tcp N.N.N.N:N: connect: connection refused
3x ... net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2x ... net/http: request canceled (Client.Timeout exceeded while awaiting headers)

故障原因(讲清原理):同样是 8 月 22 日节点启动时 apiserver 还没起来的瞬时告警,leader election 拿不到 lease,重试机制自动恢复。无需处理。

🟢 告警 #11:kube-scheduler Failed to watch(205 条)

节点:kube-scheduler-k8s-master-01(100 条)/ master-02(105 条)

告警原文(去重后 top 1):

5-6x E0822 reflector.go:N] Failed to watch *vN.StorageClass: failed to list *vN.StorageClass:
   Get "https://N.N.N.N:N/apis/storage.kNs.io/vN/storageclasses?limit=N&resourceVersion=N":
   dial tcp N.N.N.N:N: connect: connection refused

故障原因(讲清原理):与上面一致,8 月 22 日节点启动时 scheduler 反射器连不上 apiserver,影响多种资源(StorageClass/StatefulSet/Service/Pod/PV/PVC/Node 等)的 watch 失败,apiserver 起来后立即恢复。无需处理。

🟢 告警 #12:kubelet CNI 网络清理失败(约 30 条,全 6 节点)

节点:k8s-master-01(4 条)/ k8s-node-01(13 条)/ k8s-node-03(5 条)

告警原文(去重后):

4-13x E0822 cni.go:N] "Error deleting pod from network" err="error getting ClusterInformation:
      Get https://[N.N.N.N]:N/apis/crd.projectcalico.org/v1/clusterinformations: connect: connection refused"
 4x  I0822 cni.go:N] "CNI failed to retrieve network namespace path" err="cannot find network namespace for the terminated container"
15x E0822 remote_runtime.go:N] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod"

故障原因(讲清原理):8 月 22 日节点重启时,kubelet 清理上一节点遗留的 Pod sandbox,但 calico-node 还没起来,CNI 插件调用失败。等 calico-node 起来后立即恢复。无需处理。

🟢 告警 #13:docker freeze container 失败(约 40 条)

节点:全 6 节点 kubelet(偶发)

告警原文:

40x time="N-N-22THH:MM:SS+N:N" level=info msg="freeze container before SetUnitProperties failed: unable to freeze"

故障原因(讲清原理):docker 在更新容器资源限制(cgroup SetUnitProperties)时尝试 freeze 容器失败,已知 docker 20.10 的偶发 info 级别日志,不影响业务。无需处理。


四、按节点/组件汇总表

4.1 Pod 级告警统计(static pods + DaemonSet + Deployment)

Pod 名 WARN ERROR FAIL FATAL TOTAL 主要告警
kube-apiserver-k8s-master-01 0 0 0 0 0 无
kube-apiserver-k8s-master-02 0 5 0 0 5 偶发 broken pipe/timeout
kube-apiserver-k8s-master-03 0 6 0 0 6 同上
kube-controller-manager-master-01 0 5 0 0 5 leader election 偶发
kube-controller-manager-master-02 0 0 987 0 987 🔴 Grafana FailedBinding
kube-controller-manager-master-03 0 1 0 0 1 同 master-01
kube-scheduler-k8s-master-01 0 8 92 0 100 Failed to watch(瞬时)
kube-scheduler-k8s-master-02 0 5 100 0 105 同上
kube-scheduler-k8s-master-03 0 0 0 0 0 无
etcd-k8s-master-01 414 0 160 0 414 🔴 慢磁盘 + apply 慢
etcd-k8s-master-02 383 0 73 0 383 同上
etcd-k8s-master-03 415 0 67 0 415 同上
calico-node × 6 0 0 0 0 0 ✅ 全绿
kube-proxy × 6 0 1 0 0 1 etcd leader 切换
coredns-5gs6p 1 10 0 0 11 🟡 DNS 超时
coredns-frgpz 0 0 0 0 0 无
calico-kube-controllers 0 1 1 0 1 Failed to reach apiserver

4.2 节点 kubelet 告警统计

节点 WARN ERROR FAIL TOTAL 主要告警类型
k8s-master-01 4 510 113 545 "node not found" × 418 + 启动瞬时
k8s-master-02 1 663 113 710 "node not found" × 587 + 启动瞬时
k8s-master-03 1 669 113 704 "node not found" × 582 + 启动瞬时
k8s-node-01 20 284 198 372 "node not found" × 117 + CNI 清理失败
k8s-node-02 1 250 250 251 🔴 orphaned pod × 250
k8s-node-03 9 556 114 598 "node not found" × 457 + 启动瞬时
TOTAL 36 2932 801 ~3183

4.3 告警分类汇总(全集群)

类别 总数 严重度 处理建议
etcd 慢磁盘告警 1212 🔴 HIGH 建议检查 master 磁盘 IO + etcd 数据目录是否 SSD
8月22日节点启动瞬时告警 ~2161 🟢 LOW 已知现象,无需处理
Grafana PVC FailedBinding 987 🟡 MEDIUM 已知问题,方案在上一份报告
kube-scheduler Failed to watch 205 🟢 LOW 启动瞬时现象
CNI/DNS/网络告警 188 🟡 MEDIUM 建议查 coredns 上游 DNS 配置
Pod sandbox cleanup failed 120 🟢 LOW 启动瞬时现象
docker freeze container 失败 40 🟢 LOW 已知 docker 20.10 偶发
calico-node CrashLoopBackOff 13 🟢 LOW PH 治理触发滚动重启,已恢复
Container runtime 偶发 51 🟢 LOW 启动瞬时现象
etcd leader 切换 13 🟢 LOW 启动瞬时现象
kube-apiserver broken pipe 11 🟢 LOW exec/logs 长连接偶发
其他瞬时告警 ~36 🟢 LOW 启动瞬时现象

五、结论与下一步建议(不执行,待用户决策)

5.1 当前集群状态结论

✅ 集群当前状态健康:6 节点全部 Ready + 所有核心 Pod 1/1 Running + BGP Established=5 + Calico-node 全部 0 告警(治理成果)。

5.2 建议优先关注的 2 个真问题(用户决策是否排查)

优先级 告警 影响 建议下一步排查命令(仅采集,不执行)
🔴 HIGH etcd 慢磁盘告警 1212 条 长期可能影响集群稳定性,且与历史重启 4000-6737 次强相关 ssh k8s-master-01 lsblk -d -o NAME,ROTA,SIZE,MODEL
ssh k8s-master-01 iostat -x 1 3 \| grep -E "vda\|vdb\|sda"
ssh k8s-master-01 du -sh /var/lib/etcd/
🔴 HIGH k8s-node-02 orphaned pod 250 条 节点磁盘空间泄漏,kubelet 启动可能越来越慢 ssh k8s-node-02 ls /var/lib/kubelet/pods/322a2355-92c8-4a64-a0c9-f6e9f8705c58/volumes/
ssh k8s-node-02 df -h \| grep kubelet
ssh k8s-node-02 mount \| grep 322a2355

5.3 已知无需处理的告警(说明原因)

  • ComponentStatus deprecated warning:K8s 1.19+ 已废弃该命令,无需处理
  • 8月22日节点启动瞬时告警(~2161 条):kubeadm 集群节点重启时 kubelet 早于 apiserver 起来的已知现象,无业务影响
  • calico-node CrashLoopBackOff 13 条:今天 PH 治理触发 DaemonSet 滚动重启的瞬时现象,已恢复
  • kube-apiserver broken pipe 11 条:exec/logs 长连接偶发,正常现象
  • docker freeze container 40 条:docker 20.10 已知偶发 info 日志

六、附件:日志采集文件清单

文件路径 大小 说明
/tmp/k8s_core_logs/kube-apiserver-k8s-master-01.log 140K apiserver 日志(最近 1000 行)
/tmp/k8s_core_logs/kube-apiserver-k8s-master-02.log 148K 同上
/tmp/k8s_core_logs/kube-apiserver-k8s-master-03.log 144K 同上
/tmp/k8s_core_logs/kube-controller-manager-*.log 3 个 controller-manager 日志
/tmp/k8s_core_logs/kube-scheduler-*.log 3 个 scheduler 日志
/tmp/k8s_core_logs/etcd-k8s-master-*.log 3 个 etcd 日志(含慢磁盘告警)
/tmp/k8s_core_logs/calico-node-*.log 6 个 calico-node 日志(全 0 告警)
/tmp/k8s_core_logs/kube-proxy-*.log 6 个 kube-proxy 日志
/tmp/k8s_core_logs/coredns-*.log 2 个 coredns 日志
/tmp/k8s_core_logs/calico-kube-controllers-*.log 1 个 calico-kube-controllers 日志
/tmp/k8s_core_logs/kubelet_*.log 6 个 6 节点 kubelet journalctl 日志

📋 报告归档:本次为只读巡检,未执行任何修改操作。下一步动作待用户决策。


来源:/home/rain/Documents/trae_for_om/k8s_core_components_alerts/K8s集群核心组件日志告警汇总排查报告.md | 最后更新:2026-08-24 11:01(由 build/sync.py 同步,请勿在站点侧修改)