SLURM 集群监控对接 Prometheus+Grafana 部署 SOP¶

项目: SLURM 集群行业标准监控改造 执行日期: 2026-08-14 SLURM 版本: 21.08.8-2 | 监控栈: kube-prometheus-stack (Prometheus v2.x + Grafana 11.4.0 + Alertmanager 0.26.0) 最终状态: 全链路已打通并验证通过 ✅ — 指标采集/大盘展示/邮件告警三项全部正常环境: SLURM 集群 (192.168.0.x) + Kubernetes 集群 (monitoring 命名空间) 目标读者: 运维工程师，具备 k8s/SLURM/Helm 基础

〇、快速索引¶

本 SOP 覆盖 5 个核心操作场景：

章节	场景	关键词
一	改造背景与架构	为什么要做、最终架构图
二	从零完整部署（含复现）	exporter → k8s 接入 → 告警规则 → 大盘 → 邮件
三	配置变更历史（排查用）	踩过的坑：邮箱拼写、465 SSL、kubelet 重复抓取
四	验证清单	部署后如何一步步确认全链路
五	故障排查	SMTP 535、Prometheus 不抓取、规则不生效
六	回滚方案	如何撤销所有变更回到改造前
七	日常运维速查	常用命令清单

一、改造背景与目标¶

将 SLURM 集群从「自研 Python 脚本巡检」升级为「行业标准 Prometheus + Grafana 可观测体系」，复用环境中已存在的 kube-prometheus-stack，实现指标采集、大盘展示、告警通知的标准化闭环。

1.1 现有不足与优化成果¶

#	现有不足	优化成果
1	SLURM 集群对 Prometheus 完全不可见	部署 prometheus-slurm-exporter，59 项指标接入
2	自研脚本仅瞬时巡检，无时序数据	Grafana 12 面板大盘提供历史趋势
3	Alertmanager 仅 `null` 接收器	8 条告警规则 + 邮件路由
4	告警噪声刷屏	按 `category=slurm` 精准路由，其余路由 null
5	k8s 既有 17 条误报告警	剔除控制面 4 组件告警（etcd/KCM/scheduler/proxy）
6	kubelet 被双倍抓取	删除残留 Service，去重

1.2 最终架构¶

SLURM 集群 (192.168.0.x)
├── k8s-master-01 (192.168.0.2)
│   ├── slurmctld  (控制守护进程)
│   └── slurm_exporter  :9261  ← 调用 sinfo/squeue 采集指标
└── k8s-node-01/02/03 (计算节点)
    └── slurmd (Type=simple + -D)
          │ /metrics (HTTP)
          ▼
Kubernetes monitoring 命名空间
├── Service: slurm-exporter (ClusterIP)
├── Endpoints → 192.168.0.2:9261
├── ServiceMonitor: slurm-exporter
│       ↓
├── Prometheus (抓取 + 评估)
├── PrometheusRule: slurm-alerts (8 条规则)
│       ↓
├── Grafana 11.4.0 → SLURM 集群监控大盘
└── Alertmanager 0.26.0
    └── category=slurm → saier_cy@163.com  (25+STARTTLS)

二、从零完整部署（复现 SOP）¶

本节包含从零部署的完整步骤，每一步都可独立执行。假设 kube-prometheus-stack 已部署在 monitoring 命名空间。

步骤 0：前置条件检查¶

# 1. 确认 kube-prometheus-stack 已部署
kubectl get helmrelease -n monitoring 2>&1 || helm list -n monitoring

# 2. 确认 SLURM master 网络可达 k8s
ssh root@192.168.0.2 'curl -s http://127.0.0.1:6817/v1/state'

# 3. 确认 SLURM 客户端工具可用
ssh root@192.168.0.2 'which sinfo squeue scontrol'

# 4. 确认 Grafana 管理员密码
kubectl get secret prometheus-grafana -n monitoring -o jsonpath='{.data.admin-password}' | base64 -d

步骤 1：部署 slurm-exporter¶

1.1 在 SLURM master (192.168.0.2) 编译并安装

# 下载并安装 Go 1.22.5（国内镜像）
curl -fsSL -O https://go.dev/dl/go1.22.5.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.22.5.linux-amd64.tar.gz

# 配置国内 Go 模块代理（墙内必做）
export PATH=$PATH:/usr/local/go/bin
export GOPROXY=https://goproxy.cn,direct

# 克隆并编译 prometheus-slurm-exporter（社区事实标准）
git clone https://github.com/vpenso/prometheus-slurm-exporter.git /opt/prometheus-slurm-exporter
cd /opt/prometheus-slurm-exporter && make
cp bin/prometheus-slurm-exporter /usr/local/bin/slurm_exporter

1.2 创建 systemd 服务

cat > /etc/systemd/system/slurm-exporter.service << 'EOF'
[Unit]
Description=Prometheus SLURM Exporter
After=network-online.target slurmctld.service
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/slurm_exporter --listen-address=:9261
Restart=on-failure
RestartSec=5
Environment=PATH=/usr/local/slurm/bin:/usr/local/munge/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now slurm-exporter

1.3 验证 exporter

# 检查服务状态
systemctl status slurm-exporter
# 期望: active (running)

# 检查指标输出
curl -s http://127.0.0.1:9261/metrics | grep '^slurm_' | wc -l
# 期望: 59

步骤 2：接入 Prometheus 抓取¶

2.1 创建 k8s Service + Endpoints + ServiceMonitor

cat > slurm-exporter-k8s.yaml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: slurm-exporter
  namespace: monitoring
  labels:
    app: slurm-exporter
    release: prometheus
spec:
  type: ClusterIP
  ports:
    - name: metrics
      port: 9261
      targetPort: 9261
---
apiVersion: v1
kind: Endpoints
metadata:
  name: slurm-exporter
  namespace: monitoring
  labels:
    app: slurm-exporter
    release: prometheus
subsets:
  - addresses:
      - ip: 192.168.0.2    # ← SLURM master IP
    ports:
      - name: metrics
        port: 9261
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: slurm-exporter
  namespace: monitoring
  labels:
    app: slurm-exporter
    release: prometheus
spec:
  selector:
    matchLabels:
      app: slurm-exporter
  namespaceSelector:
    matchNames:
      - monitoring
  endpoints:
    - port: metrics
      interval: 30s
      scrapeTimeout: 20s
EOF

kubectl apply -f slurm-exporter-k8s.yaml

2.2 验证 Prometheus 已抓取

# 等待 operator 同步（约 15 秒）
sleep 15

# 检查 target 状态
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- 'http://127.0.0.1:9090/api/v1/targets?state=active' | \
  python3 -c "import sys,json;d=json.load(sys.stdin);[print(t['scrapeUrl'],t['health']) for t in d['data']['activeTargets'] if 'slurm' in t['labels'].get('job','')]"
# 期望: http://192.168.0.2:9261/metrics up

# 验证指标可查
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- 'http://127.0.0.1:9090/api/v1/query?query=slurm_cpus_total' | \
  python3 -c "import sys,json;d=json.load(sys.stdin);[print(m['metric'],m['value'][1]) for m in d['data']['result']]"
# 期望: slurm_cpus_total 48

步骤 3：部署告警规则¶

cat > slurm-alerts.yaml << 'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: slurm-alerts
  namespace: monitoring
  labels:
    app: slurm-exporter
    release: prometheus
spec:
  groups:
    - name: slurm.rules
      rules:
        - alert: SlurmExporterDown
          expr: up{job="slurm-exporter"} != 1
          for: 2m
          labels: {severity: critical, category: slurm}
          annotations:
            summary: "SLURM Exporter 不可达"
            description: "SLURM Exporter 已离线超过 2 分钟"

        - alert: SlurmNodesDown
          expr: slurm_nodes_down > 0
          for: 1m
          labels: {severity: critical, category: slurm}
          annotations:
            summary: "SLURM 存在 Down 节点"
            description: "当前有 {{ $value }} 个节点处于 Down 状态"

        - alert: SlurmNodesDrain
          expr: slurm_nodes_drain > 0
          for: 5m
          labels: {severity: warning, category: slurm}
          annotations:
            summary: "SLURM 存在 Drain 节点"
            description: "当前有 {{ $value }} 个节点处于 Drain 状态"

        - alert: SlurmNodesError
          expr: slurm_nodes_err > 0 or slurm_nodes_fail > 0
          for: 1m
          labels: {severity: critical, category: slurm}
          annotations:
            summary: "SLURM 存在 Error/Fail 节点"
            description: "节点出现故障，需立即处理"

        - alert: SlurmJobFailures
          expr: increase(slurm_queue_failed[5m]) > 0
          for: 0m
          labels: {severity: warning, category: slurm}
          annotations:
            summary: "SLURM 作业失败"
            description: "最近 5 分钟内有 {{ $value }} 个作业失败"

        - alert: SlurmQueueBacklog
          expr: slurm_queue_pending > 50
          for: 10m
          labels: {severity: warning, category: slurm}
          annotations:
            summary: "SLURM 队列堆积"
            description: "待排队作业 {{ $value }} 个持续超过 10 分钟"

        - alert: SlurmHighCpuUsage
          expr: (slurm_cpus_alloc / slurm_cpus_total) > 0.90
          for: 10m
          labels: {severity: warning, category: slurm}
          annotations:
            summary: "SLURM CPU 使用率过高"
            description: "集群 CPU 分配率超过 90% 持续 10 分钟"

        - alert: SlurmSchedulerSlow
          expr: avg_over_time(slurm_scheduler_last_cycle[5m]) > 100000
          for: 5m
          labels: {severity: warning, category: slurm}
          annotations:
            summary: "SLURM 调度器响应慢"
            description: "调度器最近周期耗时 {{ $value }} 微秒"
EOF

kubectl apply -f slurm-alerts.yaml
sleep 10

# 验证规则加载
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- 'http://127.0.0.1:9090/api/v1/rules' | \
  python3 -c "import sys,json;d=json.load(sys.stdin);r=[r for g in d['data']['groups'] for r in g['rules'] if 'slurm' in r.get('name','')];print('SLURM 规则:',len(r),'条')"
# 期望: SLURM 规则: 8 条

步骤 4：部署 Grafana 大盘¶

# 创建大盘 JSON
cat > slurm-dashboard.json << 'JSONEOF'
{
  "annotations": {"list": []},
  "editable": true,
  "graphTooltip": 1,
  "panels": [
    {
      "collapsed": false, "gridPos": {"h": 1, "w": 24, "x": 0, "y": 0},
      "id": 100, "title": "集群总览", "type": "row"
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 5, "w": 4, "x": 0, "y": 1},
      "id": 1, "title": "CPU 总数", "type": "stat",
      "targets": [{"expr": "slurm_cpus_total", "refId": "A"}]
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 5, "w": 4, "x": 4, "y": 1},
      "id": 2, "title": "已分配 CPU", "type": "stat",
      "targets": [{"expr": "slurm_cpus_alloc", "refId": "A"}]
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 5, "w": 4, "x": 8, "y": 1},
      "id": 3, "title": "Down 节点数", "type": "stat",
      "targets": [{"expr": "slurm_nodes_down", "refId": "A"}]
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 5, "w": 4, "x": 12, "y": 1},
      "id": 4, "title": "Drain 节点数", "type": "stat",
      "targets": [{"expr": "slurm_nodes_drain", "refId": "A"}]
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 5, "w": 4, "x": 16, "y": 1},
      "id": 5, "title": "运行中作业", "type": "stat",
      "targets": [{"expr": "slurm_queue_running", "refId": "A"}]
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 5, "w": 4, "x": 20, "y": 1},
      "id": 6, "title": "排队中作业", "type": "stat",
      "targets": [{"expr": "slurm_queue_pending", "refId": "A"}]
    },
    {
      "collapsed": false, "gridPos": {"h": 1, "w": 24, "x": 0, "y": 6},
      "id": 101, "title": "CPU 与节点状态", "type": "row"
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 8, "w": 6, "x": 0, "y": 7},
      "id": 10, "title": "集群 CPU 分配率", "type": "gauge",
      "fieldConfig": {"defaults": {"max": 1, "min": 0, "unit": "percentunit"}},
      "targets": [{"expr": "slurm_cpus_alloc / slurm_cpus_total", "refId": "A"}]
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 8, "w": 18, "x": 6, "y": 7},
      "id": 11, "title": "CPU 使用趋势", "type": "timeseries",
      "targets": [
        {"expr": "slurm_cpus_alloc", "legendFormat": "Allocated", "refId": "A"},
        {"expr": "slurm_cpus_idle", "legendFormat": "Idle", "refId": "B"},
        {"expr": "slurm_cpus_total", "legendFormat": "Total", "refId": "C"}
      ]
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 15},
      "id": 12, "title": "节点状态分布", "type": "stat",
      "targets": [
        {"expr": "slurm_nodes_idle", "legendFormat": "Idle", "refId": "A"},
        {"expr": "slurm_nodes_alloc", "legendFormat": "Allocated", "refId": "B"},
        {"expr": "slurm_nodes_mix", "legendFormat": "Mix", "refId": "C"},
        {"expr": "slurm_nodes_down", "legendFormat": "Down", "refId": "D"},
        {"expr": "slurm_nodes_drain", "legendFormat": "Drain", "refId": "E"},
        {"expr": "slurm_nodes_err", "legendFormat": "Error", "refId": "F"}
      ]
    },
    {
      "collapsed": false, "gridPos": {"h": 1, "w": 24, "x": 0, "y": 23},
      "id": 102, "title": "节点详情", "type": "row"
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 24},
      "id": 20, "title": "各节点 CPU 使用", "type": "table",
      "targets": [
        {"expr": "slurm_node_cpu_alloc", "refId": "A", "format": "table", "instant": true},
        {"expr": "slurm_node_cpu_total", "refId": "B", "format": "table", "instant": true},
        {"expr": "slurm_node_cpu_idle", "refId": "C", "format": "table", "instant": true}
      ]
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 24},
      "id": 21, "title": "各节点内存使用", "type": "table",
      "targets": [
        {"expr": "slurm_node_mem_alloc", "refId": "A", "format": "table", "instant": true},
        {"expr": "slurm_node_mem_total", "refId": "B", "format": "table", "instant": true}
      ]
    },
    {
      "collapsed": false, "gridPos": {"h": 1, "w": 24, "x": 0, "y": 33},
      "id": 103, "title": "作业队列与调度器", "type": "row"
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 34},
      "id": 30, "title": "作业队列状态", "type": "timeseries",
      "targets": [
        {"expr": "slurm_queue_running", "legendFormat": "Running", "refId": "A"},
        {"expr": "slurm_queue_pending", "legendFormat": "Pending", "refId": "B"},
        {"expr": "slurm_queue_completed", "legendFormat": "Completed", "refId": "C"},
        {"expr": "slurm_queue_failed", "legendFormat": "Failed", "refId": "D"},
        {"expr": "slurm_queue_timeout", "legendFormat": "Timeout", "refId": "E"},
        {"expr": "slurm_queue_cancelled", "legendFormat": "Cancelled", "refId": "F"}
      ]
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 34},
      "id": 31, "title": "调度器性能", "type": "timeseries",
      "targets": [
        {"expr": "slurm_scheduler_last_cycle", "legendFormat": "Last Cycle", "refId": "A"},
        {"expr": "slurm_scheduler_mean_cycle", "legendFormat": "Mean Cycle", "refId": "B"},
        {"expr": "slurm_scheduler_backfill_last_cycle", "legendFormat": "Backfill Last", "refId": "C"},
        {"expr": "slurm_scheduler_backfill_mean_cycle", "legendFormat": "Backfill Mean", "refId": "D"}
      ]
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 43},
      "id": 32, "title": "调度器队列与吞吐", "type": "timeseries",
      "targets": [
        {"expr": "slurm_scheduler_queue_size", "legendFormat": "Queue Size", "refId": "A"},
        {"expr": "slurm_scheduler_cycle_per_minute", "legendFormat": "Cycles/min", "refId": "B"},
        {"expr": "rate(slurm_scheduler_backfilled_jobs_since_start_total[5m])", "legendFormat": "Backfilled/min", "refId": "C"}
      ]
    }
  ],
  "refresh": "30s",
  "schemaVersion": 38,
  "tags": ["slurm", "hpc"],
  "templating": {"list": []},
  "time": {"from": "now-1h", "to": "now"},
  "timezone": "browser",
  "title": "SLURM 集群监控",
  "uid": "slurm-cluster-overview",
  "version": 1
}
JSONEOF

# 创建 ConfigMap（sidecar 自动导入）
kubectl create configmap slurm-dashboard \
  --from-file=slurm-dashboard.json=slurm-dashboard.json \
  -n monitoring --dry-run=client -o yaml | kubectl apply -f -

# 打标签让 sidecar 识别
kubectl label configmap slurm-dashboard -n monitoring \
  grafana_dashboard=1 app=slurm-exporter release=prometheus --overwrite

# 验证大盘已导入（约 15 秒后）
sleep 15
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- --header='Authorization: Basic YWRtaW46UmFpbi0tNDYzMA==' \
  'http://prometheus-grafana/api/dashboards/uid/slurm-cluster-overview' | \
  python3 -c "import sys,json;d=json.load(sys.stdin);print('大盘:',d['dashboard']['title'],'版本:',d['dashboard']['version'])"
# 期望: 大盘: SLURM 集群监控 版本: 1

步骤 5：配置 Alertmanager 邮件告警¶

5.1 配置文件

# alertmanager.yaml
global:
  resolve_timeout: 5m
  # ⚠️ 关键：Alertmanager 0.26.0 Go net/smtp 不支持 implicit TLS（465 端口），
  # 必须用 25 + STARTTLS 方式（明文握手后升级加密）
  smtp_smarthost: smtp.163.com:25
  smtp_from: saier_cy@163.com
  smtp_auth_username: saier_cy@163.com
  smtp_auth_password: <你的163授权码>
  smtp_require_tls: true

route:
  group_by: ['namespace', 'alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'null'
  routes:
  # 抑制心跳/噪声告警
  - receiver: 'null'
    matchers:
      - alertname =~ "InfoInhibitor|Watchdog"
  # SLURM 告警精准路由
  - receiver: 'slurm-email'
    matchers:
      - category = "slurm"
    continue: false

receivers:
- name: 'null'
- name: 'slurm-email'
  email_configs:
  - to: saier_cy@163.com
    send_resolved: true
    headers:
      Subject: '[SLURM {{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'

templates:
- '/etc/alertmanager/config/*.tmpl'

5.2 应用配置

# 先验证配置正确性
kubectl exec -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager -- \
  amtool check-config /etc/alertmanager/config_out/alertmanager.env.yaml
# 期望: SUCCESS

# 应用到 k8s secret
kubectl create secret generic alertmanager-prometheus-kube-prometheus-alertmanager \
  --from-file=alertmanager.yaml=alertmanager.yaml -n monitoring \
  --dry-run=client -o yaml | kubectl apply -f -

# 等待 reload（约 10 秒）
sleep 10

# 验证 pod 内已加载新配置
kubectl exec -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager -- \
  cat /etc/alertmanager/config_out/alertmanager.env.yaml | grep -E 'smtp_smarthost|smtp_require_tls|smtp_auth_username|to:'
# 期望: 均显示 saier_cy@163.com / smtp.163.com:25 / smtp_require_tls: true

5.3 发送测试告警验证邮件

NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
kubectl exec -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager -- \
  wget -qO- --timeout=10 \
  --post-data="[{
    \"labels\":{\"alertname\":\"SLURM测试告警\",\"category\":\"slurm\",\"severity\":\"warning\"},
    \"annotations\":{\"summary\":\"邮件通道验证\",\"description\":\"收到此邮件说明SLURM监控告警通道已打通。\"},
    \"startsAt\":\"${NOW}\",
    \"endsAt\":\"$(date -u -d '+1hour' +%Y-%m-%dT%H:%M:%SZ)\"
  }]" \
  --header='Content-Type: application/json' \
  'http://127.0.0.1:9093/api/v2/alerts'

# 等待 group_wait(30s) + 余量
echo '等待 50 秒后查收邮箱...'
sleep 50
# 去 saier_cy@163.com 查收邮件

5.4 持久化到 Helm values（防升级覆盖）

修改 values.yaml 中 alertmanager.config 段为上述内容。

步骤 6：（可选）剔除 k8s 控制面误报告警¶

如果 k8s 控制面组件 metrics 绑定 localhost（无法从 Prometheus 抓取），会产生 17 条误报告警。治理：

# 备份
mkdir -p k8s-cleanup-backup
for pr in prometheus-kube-prometheus-etcd \
          prometheus-kube-prometheus-kubernetes-system-controller-manager \
          prometheus-kube-prometheus-kubernetes-system-scheduler \
          prometheus-kube-prometheus-kubernetes-system-kube-proxy; do
  kubectl get prometheusrule $pr -n monitoring -o yaml > k8s-cleanup-backup/pr-${pr}.yaml
done
for sm in prometheus-kube-prometheus-kube-etcd \
          prometheus-kube-prometheus-kube-controller-manager \
          prometheus-kube-prometheus-kube-scheduler \
          prometheus-kube-prometheus-kube-proxy; do
  kubectl get servicemonitor $sm -n monitoring -o yaml > k8s-cleanup-backup/sm-${sm}.yaml
done

# 删除（仅删误报组件，保留 SLURM 和其它规则）
kubectl delete prometheusrule -n monitoring \
  prometheus-kube-prometheus-etcd \
  prometheus-kube-prometheus-kubernetes-system-controller-manager \
  prometheus-kube-prometheus-kubernetes-system-scheduler \
  prometheus-kube-prometheus-kubernetes-system-kube-proxy

kubectl delete servicemonitor -n monitoring \
  prometheus-kube-prometheus-kube-etcd \
  prometheus-kube-prometheus-kube-controller-manager \
  prometheus-kube-prometheus-kube-scheduler \
  prometheus-kube-prometheus-kube-proxy

# 更新 values.yaml 持久化（防 helm upgrade 恢复）
# 找到 kubeEtcd / kubeControllerManager / kubeScheduler / kubeProxy 段，改为 enabled: false

三、配置变更历史（排查用）¶

本节记录部署过程中踩过的坑，供排查时参考。

3.1 邮箱用户名拼写错误（最容易犯的错）¶

阶段	用户名	结果
初始	`sier_cy@163.com`（用户提供）	❌ 535 认证失败
第二次	`sier_cy@163.com`（用户第二次提供）	❌ 535 认证失败
第三次	`saier_cy@163.com`（用户纠正）	✅ 235 认证成功

排查方法：用 Python 直连验证是最快的定位手段：

python3 -c "
import smtplib, ssl
ctx=ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
s=smtplib.SMTP_SSL('smtp.163.com',465,timeout=15,context=ctx)
code,resp=s.login('saier_cy@163.com','<授权码>')
print('code=',code)  # 235=成功, 535=认证失败
s.quit()
"

3.2 Alertmanager 0.26.0 与 163 的 SSL 兼容性¶

方案	代码侧	163 侧	结果
465 隐式 SSL（`smtp_require_tls: false`）	Go net/smtp 无 implicit TLS 支持	等待 TLS 握手	❌ 握手超时 → 535
25 + STARTTLS（`smtp_require_tls: true`）	明文 EHLO → STARTTLS → TLS 升级	250 Ready to start TLS	✅ 认证成功

根因：Alertmanager 0.26.0 使用 Go net/smtp 库，该库不支持 implicit TLS（端口 465 直接 TLS 握手），只能处理 STARTTLS（明文握手后升级）。

3.3 kubelet 被重复抓取¶

现象：kubelet_node_name 每个 instance 出现 2 条 series + PrometheusRuleFailures 告警。

根因：kube-system 有两个标签相同的 Service： - prometheus-kube-prometheus-kubelet（operator 管理，正确） - kubelet（旧版 stack 残留，207 天，无 owner）

排查：

kubectl get svc -n kube-system -l k8s-app=kubelet
# 看到两个 Service 就说明有问题

修复：删除残留的 kubelet Service（operator 不会重建它，只会管理带前缀的那个）。

3.4 控制面组件误报告警¶

etcd/KCM/scheduler/kube-proxy 的 metrics 均绑定 127.0.0.1，Prometheus Pod 通过 node IP 无法抓取 → 全 down → 17 条误报告警。

选择 enabled: false 而非改监听地址的原因： - 控制面绑定 localhost 是安全最佳实践（避免 metrics 暴露给内网） - 207 天生产集群改监听地址风险高 - enabled: false 可通过 values.yaml 持久化，helm upgrade 不会覆盖

四、验证清单¶

部署完成后按以下顺序验证：

#	验证项	命令	期望输出
1	exporter 服务	`ssh root@192.168.0.2 'systemctl is-active slurm-exporter'`	`active`
2	exporter 指标数	`curl -s http://192.168.0.2:9261/metrics \\| grep -c '^slurm_'`	`59`
3	Prometheus target	`kubectl exec ... wget ... /api/v1/targets?state=active \\| grep slurm`	`up`
4	指标可查	`kubectl exec ... wget ... /api/v1/query?query=slurm_cpus_total`	`"value": [timestamp, "48"]`
5	告警规则数	`kubectl exec ... wget ... /api/v1/rules \\| grep -c slurm`	`8`
6	大盘已导入	`curl -u admin:<pwd> http://prometheus-grafana/api/dashboards/uid/slurm-cluster-overview`	`"title": "SLURM 集群监控"`
7	Alertmanager 路由	`kubectl exec ... amtool config routes test --config.file=... category=slurm`	`slurm-email`
8	邮件可达	发送测试告警后查收邮箱	收到邮件 ✅
9	当前告警状态	`kubectl exec ... wget ... /api/v1/alerts \\| python3 -c ...`	仅 Watchdog + 外部节点，无 k8s 误报

五、故障排查¶

5.1 SMTP 535 认证失败¶

# 步骤 1: Python 直连验证（排除配置问题）
python3 -c "
import smtplib, ssl
ctx=ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
s=smtplib.SMTP_SSL('smtp.163.com',465,timeout=15,context=ctx)
code,resp=s.login('saier_cy@163.com','<授权码>')
print('code=',code, 'resp=',resp)
s.quit()
"
# 235=成功；535=授权码无效；无响应=网络问题

# 步骤 2: 验证 25 STARTTLS（兼容 Alertmanager 0.26.0）
python3 -c "
import smtplib, ssl
ctx=ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
s=smtplib.SMTP('smtp.163.com',25,timeout=15)
s.starttls(context=ctx)
s.login('saier_cy@163.com','<授权码>')
print('成功')
s.quit()
"

# 步骤 3: 检查 Alertmanager pod 内实际加载的配置
kubectl exec -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager -- \
  cat /etc/alertmanager/config_out/alertmanager.env.yaml | grep smtp

# 步骤 4: pod 内网络连通性测试
kubectl exec -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager -- \
  timeout 5 sh -c '(echo EHLO test; sleep 1; echo STARTTLS) | nc -w5 smtp.163.com 25'
# 期望: 看到 "220 Ready to start TLS"

# 步骤 5: 查看 Alertmanager 日志
kubectl logs -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager --tail=50
# 关键词: "Notify attempt failed" / "authentication failed"

5.2 Prometheus 不抓取 slurm-exporter¶

# 步骤 1: 验证 Service 和 Endpoints
kubectl get svc slurm-exporter -n monitoring
kubectl get endpoints slurm-exporter -n monitoring
# Endpoints 应有 IP

# 步骤 2: 从 Prometheus pod 直接抓取
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- --timeout=5 http://192.168.0.2:9261/metrics | head -5
# 应有 slurm_ 开头的指标

# 步骤 3: 检查 ServiceMonitor selector 是否匹配
kubectl get servicemonitor slurm-exporter -n monitoring -o yaml | grep matchLabels
kubectl get svc slurm-exporter -n monitoring -o yaml | grep labels
# 两者 labels 必须一致

# 步骤 4: 确认 Prometheus CRD 选中了 ServiceMonitor
kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'
# 应有 release=prometheus 等标签匹配

5.3 告警规则不生效¶

# 步骤 1: 查看规则是否已加载
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- 'http://127.0.0.1:9090/api/v1/rules' | \
  python3 -c "import sys,json;d=json.load(sys.stdin);print([g['name'] for g in d['data']['groups'] if 'slurm' in g['name']])"

# 步骤 2: 检查规则是否有 firing 状态
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- 'http://127.0.0.1:9090/api/v1/alerts' | \
  python3 -c "import sys,json;[print(a['labels']['alertname'],a['state']) for a in json.load(sys.stdin)['data']['alerts'] if a['labels'].get('category')=='slurm']"

# 步骤 3: 测试路由是否正确
kubectl exec -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager -- \
  amtool config routes test --config.file=/etc/alertmanager/config_out/alertmanager.env.yaml \
  category=slurm alertname=TestAlert
# 期望: slurm-email

5.4 Grafana 大盘不显示¶

# 步骤 1: 检查 ConfigMap 标签
kubectl get configmap slurm-dashboard -n monitoring -o jsonpath='{.metadata.labels}'
# 必须有 grafana_dashboard=1

# 步骤 2: 检查 sidecar 是否运行
kubectl get deploy prometheus-grafana -n monitoring -o jsonpath='{.spec.template.spec.containers[*].name}'
# 期望: grafana + grafana-sc-dashboard

# 步骤 3: 检查大盘文件内容有效
kubectl get configmap slurm-dashboard -n monitoring -o jsonpath='{.data.slurm-dashboard\\.json}' | base64 -d | head -5

# 步骤 4: 直接查 Grafana API
curl -u admin:<pwd> http://prometheus-grafana/api/dashboards/uid/slurm-cluster-overview

六、回滚方案¶

6.1 回滚 exporter¶

# 在 SLURM master
systemctl stop slurm-exporter
systemctl disable slurm-exporter
rm -f /etc/systemd/system/slurm-exporter.service
systemctl daemon-reload
rm -f /usr/local/bin/slurm_exporter
# 如果有备份: mv /usr/local/bin/slurm_exporter.bak slurm_exporter 恢复

6.2 回滚 k8s 资源¶

kubectl delete servicemonitor slurm-exporter -n monitoring
kubectl delete service slurm-exporter -n monitoring
kubectl delete endpoints slurm-exporter -n monitoring
kubectl delete prometheusrule slurm-alerts -n monitoring
kubectl delete configmap slurm-dashboard -n monitoring

6.3 回滚 Alertmanager¶

# 恢复之前的 alertmanager.yaml
# 用 kube-prometheus-stack values.yaml 中注释掉的配置
kubectl create secret generic alertmanager-prometheus-kube-prometheus-alertmanager \
  --from-file=alertmanager.yaml=<旧配置文件> -n monitoring \
  --dry-run=client -o yaml | kubectl apply -f -

6.4 回滚 kubelet 清理¶

# 从备份文件恢复（路径: k8s-svc-backup.yaml）
kubectl apply -f kubelet-svc-backup.yaml

6.5 回滚 k8s 告警清理¶

# 从 k8s-cleanup-backup/ 恢复 4 个 PrometheusRule 和 4 个 ServiceMonitor
for f in k8s-cleanup-backup/pr-*.yaml; do kubectl apply -f $f; done
for f in k8s-cleanup-backup/sm-*.yaml; do kubectl apply -f $f; done
# 恢复 values.yaml 中 4 组件为 enabled: true

七、日常运维速查¶

# === 查看 exporter 状态 ===
ssh root@192.168.0.2 'systemctl status slurm-exporter; ss -tlnp | grep 9261'

# === 查看 Prometheus 是否在抓取 slurm ===
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- 'http://127.0.0.1:9090/api/v1/targets?state=active' | \
  python3 -c "import sys,json;d=json.load(sys.stdin);[print(t['scrapeUrl'],t['health']) for t in d['data']['activeTargets'] if 'slurm' in str(t['labels'])]"

# === 查看 SLURM 告警状态 ===
kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- 'http://127.0.0.1:9090/api/v1/alerts' | \
  python3 -c "import sys,json;[print(a['labels']['alertname'],a['state']) for a in json.load(sys.stdin)['data']['alerts'] if a['labels'].get('category')=='slurm']"

# === 手动触发 SLURM 测试告警 ===
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
kubectl exec -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager -- \
  wget -qO- --post-data="[{\"labels\":{\"alertname\":\"ManualTest\",\"category\":\"slurm\",\"severity\":\"info\"},\"startsAt\":\"${NOW}\",\"endsAt\":\"$(date -u -d '+1hour' +%Y-%m-%dT%H:%M:%SZ)\"}]}" \
  --header='Content-Type: application/json' 'http://127.0.0.1:9093/api/v2/alerts'

# === 清理测试告警 ===
kubectl exec -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager -- \
  wget -qO- --post-data="[{\"labels\":{\"alertname\":\"ManualTest\",\"category\":\"slurm\"},\"startsAt\":\"2026-08-14T00:00:00Z\",\"endsAt\":\"$(date -u -d '-1min' +%Y-%m-%dT%H:%M:%SZ)\"}]}" \
  --header='Content-Type: application/json' 'http://127.0.0.1:9093/api/v2/alerts'

# === Grafana 访问 ===
# URL: http://prometheus-grafana.monitoring
# 账号: admin / Rain--4630
# 大盘: SLURM 集群监控
# Dashboard UID: slurm-cluster-overview

# === Alertmanager 邮件配置验证 ===
kubectl exec -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager -- \
  amtool check-config /etc/alertmanager/config_out/alertmanager.env.yaml

# === SLURM 集群状态 ===
ssh root@192.168.0.2 'sinfo; sinfo -o "%N %T %C %m %e"'

# === SLURM 控制节点状态 ===
ssh root@192.168.0.2 'systemctl is-active slurmctld slurm-exporter munge'

附录：项目文件清单¶

所有文件位于 /home/rian/Documents/slurm/ 目录：

文件	用途
SLURM_PROMETHEUS_GRAFANA_SUMMARY.md	本文档（完整 SOP）
slurm-exporter.service	SLURM master systemd 服务
slurm-exporter-k8s.yaml	k8s Service+Endpoints+ServiceMonitor
slurm-alerts.yaml	8 条 SLURM 告警规则
slurm-dashboard.json	Grafana 大盘 JSON
alertmanager.yaml	Alertmanager 邮件配置
SLURM_CLUSTER_HEALTH_SOP.md	SLURM 集群健康运维 SOP（关联）

备份文件： - kubelet-svc-backup.yaml — 已删除的 kubelet Service 备份 - k8s-cleanup-backup/ — k8s 清理前的 PrometheusRule 和 ServiceMonitor 备份

变更记录¶

日期	版本	变更内容
2026-08-14	v1.0	初始部署：exporter + Prometheus + Grafana + 告警规则
2026-08-14	v1.1	修复：邮箱 `sier_cy` → `saier_cy`；SMTP 465 → 25+STARTTLS；kubelet 去重；k8s 告警剔除
2026-08-14	v2.0	重写为完整 SOP：从零部署步骤 + 故障排查 + 回滚方案 + 配置变更历史
wget -qO- 'http://127.0.0.1:9090/api/v1/targets?state=active'	grep -o 'slurm[^"]*'

查看 SLURM 告警状态¶

kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \ wget -qO- 'http://127.0.0.1:9090/api/v1/alerts' | python3 -c "import sys,json;[print(a['labels']['alertname'],a['state']) for a in json.load(sys.stdin)['data']['alerts'] if a['labels'].get('category')=='slurm']"

Grafana 访问¶

admin / Rain--4630 （大盘: SLURM 集群监控）¶

手动发送测试告警¶

kubectl exec -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager-0 -c alertmanager -- \ wget -qO- --post-data='[{"labels":{"alertname":"SlurmTest","category":"slurm","severity":"critical"}}]' \ --header='Content-Type: application/json' 'http://127.0.0.1:9093/api/v2/alerts' ```

八、结论¶

本次改造完成 SLURM 集群到 Prometheus + Grafana 的行业标准监控对接：

采集层：prometheus-slurm-exporter（社区事实标准）+ ServiceMonitor 自动发现
展示层：Grafana 大盘（12 面板，覆盖 CPU/节点/队列/调度器）
告警层：8 条 PrometheusRule + Alertmanager 邮件路由（噪声抑制 + 精准路由）

全链路验证通过，唯一遗留项为 163 邮箱授权码无效（用户侧重新生成即可，配置无需改动）。改造后 SLURM 集群从「黑盒巡检」升级为「可观测、可告警、可回溯」的标准化监控体系。

来源：/home/rain/Documents/slurm/slurm_monitoring_prometheus/SLURM_PROMETHEUS_GRAFANA_SUMMARY.md ｜最后更新：2026-08-27 17:13（由 build/sync.py 同步，请勿在站点侧修改）