PMM(Percona Monitoring and Management)でmetricsの取得ができなかった場合
PMMでメトリクスを取得できなくなってしまい、若干ハマったので記録する
グラフで取得対象に表示されていなかった
pmm-serverとの接続状況を確認する
[root@localhost docker]# pmm-admin list pmm-admin 1.0.6 PMM Server | 192.168.12.3 Client Name | localhost.localdomain Client Address | 192.168.12.4 Service manager | unix-systemv #CentOS7 だと linux-systemd -------------- ---------------------- ------------ -------- ----------------------------------------- --------------------- SERVICE TYPE NAME CLIENT PORT RUNNING DATA SOURCE OPTIONS -------------- ---------------------- ------------ -------- ----------------------------------------- --------------------- linux:metrics localhost.localdomain 42000 NO - mysql:queries localhost.localdomain 42001 NO root:***@unix(/var/lib/mysql/mysql.sock) query_source=slowlog mysql:metrics localhost.localdomain 42002 NO root:***@unix(/var/lib/mysql/mysql.sock)
考えられる原因
ここまでに行ったこと
- 設定ファイル "/usr/local/percona/pmm-client/pmm.yml" の編集（IPアドレスの変更）
- クライアントの再設定
[root@localhost docker]# pmm-admin config --server 192.168.12.3 OK, PMM server is alive. PMM Server | 192.168.12.3 Client Name | localhost.localdomain Client Address | 192.168.12.4
試行錯誤一覧
- クライアント設定のリペア ->効果なし
[root@localhost docker]# pmm-admin repair No orphaned services found.
- listen状況の確認
[root@localhost docker]# netstat -tan Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.1:42001 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN tcp 0 0 192.168.12.4:22 192.168.12.2:65278 ESTABLISHED tcp 0 0 :::22 :::* LISTEN tcp 0 0 :::3306 :::* LISTEN
気になるこの42001はQANで使用しているポート
- QANのログを確認
[root@localhost docker]# less pmm-mysql-queries-42001.log ...skipping... # Version: percona-qan-agent 1.0.6 # Basedir: /usr/local/percona/qan-agent # Listen: 127.0.0.1:42001 # PID: 2262 # API: 192.168.12.3/qan-api # UUID: 1e077f75b7c34601526d1383f7e04ac6 2016/11/23 19:04:09.522733 main.go:165: Starting agent... 2016/11/23 19:04:09.543769 main.go:343: Agent is ready 2016/11/23 19:04:09.559502 status.go:51: listen tcp 127.0.0.1:42001: bind: address already in use
すでに使われていると…
- 対象サービスを削除
[root@localhost docker]# pmm-admin rm mysql localhost.localdomain [linux:metrics] OK, removed system localhost.localdomain from monitoring. [mysql:metrics] OK, removed MySQL metrics localhost.localdomain from monitoring. [mysql:queries] Error removing MySQL queries localhost.localdomain: timeout 10s waiting on agent to connect to API. [root@localhost docker]# pmm-admin list pmm-admin 1.0.6 PMM Server | 192.168.12.3 Client Name | localhost.localdomain Client Address | 192.168.12.4 Service manager | unix-systemv -------------- ---------------------- ------------ -------- -------------------------------------------- --------------------- SERVICE TYPE NAME CLIENT PORT RUNNING DATA SOURCE OPTIONS -------------- ---------------------- ------------ -------- -------------------------------------------- --------------------- mysql:queries localhost.localdomain 42001 NO root:***@unix(/mysql/MyHome/tmp/mysql.sock) query_source=slowlog
なぜか、QANだけが残る…
- QANのプロセスだけ削除
[root@localhost docker]# pps -ef | grep percona root 1079 1 0 16:41 ? 00:00:13 /usr/local/percona/qan-agent/bin/percona-qan-agent -listen=127.0.0.1:42001 [root@localhost docker]# p /usr/local/percona/qan-agent/bin/percona-qan-agent --help Usage of /usr/local/percona/qan-agent/bin/percona-qan-agent: -basedir string Agent basedir (default "/usr/local/percona/qan-agent") -listen string Agent interface address (default "127.0.0.1:9000") -pid-file string PID file -ping Ping API -version Print version [root@localhost docker]# pkill -15 1079 [root@localhost docker]# ppmm-admin rm mysql [linux:metrics] OK, no system localhost.localdomain under monitoring. [mysql:metrics] OK, no MySQL metrics localhost.localdomain under monitoring. [mysql:queries] OK, removed MySQL queries localhost.localdomain from monitoring. [root@localhost docker]# ppmm-admin add mysql --password=root [linux:metrics] OK, now monitoring this system. [mysql:metrics] OK, already monitoring MySQL metrics. [mysql:queries] OK, now monitoring MySQL queries from slowlog using DSN root:***@unix(/mysql/MyHome/tmp/mysql.sock) [root@localhost.localdomain log]# pmm-admin list pmm-admin 1.0.6 PMM Server | 192.168.12.3 Client Name | localhost.localdomain Client Address | 192.168.12.4 Service manager | unix-systemv -------------- ---------------------- ------------ -------- -------------------------------------------- --------------------- SERVICE TYPE NAME CLIENT PORT RUNNING DATA SOURCE OPTIONS -------------- ---------------------- ------------ -------- -------------------------------------------- --------------------- linux:metrics localhost.localdomain 42000 YES - mysql:queries localhost.localdomain 42001 YES root:***@unix(/mysql/MyHome/tmp/mysql.sock) query_source=slowlog mysql:metrics localhost.localdomain 42002 YES root:***@unix(/mysql/MyHome/tmp/mysql.sock)
これでメトリクスを取得し、グラフに表示できるようになった