通过 prometheus 监控 ceph

https://prometheus.io/

1、监控 ceph node 节点

部署 prometheus：

[root@mgr01 ~]# mkdir /apps
[root@mgr01 ~]# cd /apps
[root@mgr01 apps]# tar xvf prometheus-2.41.0.linux-amd64.tar.gz
[root@mgr01 apps]# ln -sv /apps/prometheus-2.41.0.linux-amd64 /apps/prometheus
‘/apps/prometheus’ -> ‘/apps/prometheus-2.41.0.linux-amd64’

[root@mgr01 apps]# vim /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/apps/prometheus/
ExecStart=/apps/prometheus/prometheus --config.file=/apps/prometheus/prometheus.yml

[Install]
WantedBy=multi-user.target

[root@mgr01 apps]# systemctl daemon-reload
[root@mgr01 apps]# systemctl restart prometheus.service
[root@mgr01 apps]# systemctl enable prometheus.service

访问 prometheus：

部署 node_exporter：

各 node 节点安装 node_exporter

[root@node01 ~]# mkdir /apps
[root@node01 ~]# cd /apps
[root@node01 apps]# rz -E
rz waiting to receive.
[root@node01 apps]# tar -xvf node_exporter-1.5.0.linux-amd64.tar.gz
node_exporter-1.5.0.linux-amd64/
node_exporter-1.5.0.linux-amd64/LICENSE
node_exporter-1.5.0.linux-amd64/NOTICE
node_exporter-1.5.0.linux-amd64/node_exporter
[root@node01 apps]# ln -sv /apps/node_exporter-1.5.0.linux-amd64 /apps/node_exporter
‘/apps/node_exporter’ -> ‘/apps/node_exporter-1.5.0.linux-amd64’
[root@node01 apps]# vim /etc/systemd/system/node-exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
ExecStart=/apps/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target

[root@node01 apps]# systemctl daemon-reload
[root@node01 apps]# systemctl restart node-exporter.service
[root@node01 apps]# systemctl enable node-exporter.service

验证各 node 节点的 node_exporter 数据：

配置 prometheus server 数据并验证：

[root@mgr01 prometheus]# vim prometheus.yml
scrape_configs:
  - job_name: "ceph-node-data"
    static_configs:
      - targets: ['192.168.1.111:9100','192.168.1.112:9100','192.168.1.113:9100']

2、监控 ceph 服务

Ceph manager 内部的模块中包含了 prometheus 的监控模块，并监听在每个 manager 节点的9283 端口，该端口用于将采集到的信息通过 http 接口向 prometheus 提供数据。

启用 prometheus 监控模块：

[root@deploy ~]# ceph mgr module enable prometheus

# manager 节点验证端口

验证 manager 数据：

配置 prometheus 采集数据：

[root@mgr01 prometheus]# vim prometheus.yml
  - job_name: 'ceph-cluster-data'
    static_configs:
      - targets: ['192.168.1.117:9283']

[root@mgr01 prometheus]# systemctl restart prometheus.service 

验证数据：

3、通过 grafana 显示监控数据

通过 granfana 显示对 ceph 的集群监控数据及 node 数据

安装 grafana：

[root@mgr01 apps]# yum install grafana-9.3.2-1.x86_64.rpm
[root@mgr01 apps]# systemctl enable grafana-server.service
[root@mgr01 apps]# systemctl restart grafana-server.service 

登录 grafana：

默认用户名密码：admin/admin

配置数据源：

在 grafana 添加 prometheus 数据源。

导入模板：

https://grafana.com/grafana/dashboards/5336 #ceph OSD
https://grafana.com/grafana/dashboards/5342 #ceph pools
https://grafana.com/grafana/dashboards/7056 #ceph cluser
https://grafana.com/grafana/dashboards/2842