PMMでメトリクスを取得できなくなってしまい、若干ハマったので記録する

グラフで取得対象に表示されていなかった

pmm-serverとの接続状況を確認する

[root@localhost docker]# pmm-admin list
pmm-admin 1.0.6

PMM Server      | 192.168.12.3
Client Name     | localhost.localdomain
Client Address  | 192.168.12.4
Service manager | unix-systemv  #CentOS7 だと linux-systemd

-------------- ---------------------- ------------ -------- ----------------------------------------- ---------------------
SERVICE TYPE   NAME                   CLIENT PORT  RUNNING  DATA SOURCE                               OPTIONS        
-------------- ---------------------- ------------ -------- ----------------------------------------- ---------------------
linux:metrics  localhost.localdomain  42000        NO       -                                                        
mysql:queries  localhost.localdomain  42001        NO       root:***@unix(/var/lib/mysql/mysql.sock)  query_source=slowlog
mysql:metrics  localhost.localdomain  42002        NO       root:***@unix(/var/lib/mysql/mysql.sock)

考えられる原因

vmのIPアドレスが変わっていた
1.05 -> 1.06 へupdateした

ここまでに行ったこと

設定ファイル "/usr/local/percona/pmm-client/pmm.yml" の編集（IPアドレスの変更）

クライアントの再設定

[root@localhost docker]# pmm-admin config --server 192.168.12.3
OK, PMM server is alive.

PMM Server      | 192.168.12.3
Client Name     | localhost.localdomain
Client Address  | 192.168.12.4

試行錯誤一覧

クライアント設定のリペア ->効果なし

      [root@localhost docker]# pmm-admin repair
      No orphaned services found.

listen状況の確認

[root@localhost docker]# netstat -tan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 127.0.0.1:42001             0.0.0.0:*                   LISTEN
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN
tcp        0      0 192.168.12.4:22             192.168.12.2:65278          ESTABLISHED
tcp        0      0 :::22                       :::*                        LISTEN
tcp        0      0 :::3306                     :::*                        LISTEN

気になるこの42001はQANで使用しているポート

QANのログを確認

    [root@localhost docker]# less pmm-mysql-queries-42001.log
    ...skipping...
    # Version: percona-qan-agent 1.0.6
    # Basedir: /usr/local/percona/qan-agent
    # Listen:  127.0.0.1:42001
    # PID:     2262
    # API:     192.168.12.3/qan-api
    # UUID:    1e077f75b7c34601526d1383f7e04ac6
    2016/11/23 19:04:09.522733 main.go:165: Starting agent...
    2016/11/23 19:04:09.543769 main.go:343: Agent is ready
    2016/11/23 19:04:09.559502 status.go:51: listen tcp 127.0.0.1:42001: bind: address already in use

すでに使われていると…

対象サービスを削除

      [root@localhost docker]# pmm-admin rm mysql localhost.localdomain
[linux:metrics] OK, removed system localhost.localdomain from monitoring.
[mysql:metrics] OK, removed MySQL metrics localhost.localdomain from monitoring.
[mysql:queries] Error removing MySQL queries localhost.localdomain: timeout 10s waiting on agent to connect to API.

[root@localhost docker]# pmm-admin list
pmm-admin 1.0.6

PMM Server      | 192.168.12.3
Client Name     | localhost.localdomain
Client Address  | 192.168.12.4
Service manager | unix-systemv

-------------- ---------------------- ------------ -------- -------------------------------------------- ---------------------
SERVICE TYPE   NAME                   CLIENT PORT  RUNNING  DATA SOURCE                                  OPTIONS
-------------- ---------------------- ------------ -------- -------------------------------------------- ---------------------
mysql:queries  localhost.localdomain  42001        NO       root:***@unix(/mysql/MyHome/tmp/mysql.sock)  query_source=slowlog

なぜか、QANだけが残る…

QANのプロセスだけ削除

[root@localhost docker]# pps -ef | grep percona
root       1079      1  0 16:41 ?        00:00:13 /usr/local/percona/qan-agent/bin/percona-qan-agent -listen=127.0.0.1:42001

[root@localhost docker]# p /usr/local/percona/qan-agent/bin/percona-qan-agent --help
Usage of /usr/local/percona/qan-agent/bin/percona-qan-agent:
 -basedir string
       Agent basedir (default "/usr/local/percona/qan-agent")
 -listen string
       Agent interface address (default "127.0.0.1:9000")
 -pid-file string
       PID file
 -ping
       Ping API
 -version
       Print version

       [root@localhost docker]# pkill -15 1079

       [root@localhost docker]# ppmm-admin rm mysql
[linux:metrics] OK, no system localhost.localdomain under monitoring.
[mysql:metrics] OK, no MySQL metrics localhost.localdomain under monitoring.
[mysql:queries] OK, removed MySQL queries localhost.localdomain from monitoring.

[root@localhost docker]# ppmm-admin add mysql --password=root
[linux:metrics] OK, now monitoring this system.
[mysql:metrics] OK, already monitoring MySQL metrics.
[mysql:queries] OK, now monitoring MySQL queries from slowlog using DSN root:***@unix(/mysql/MyHome/tmp/mysql.sock)
[root@localhost.localdomain log]# pmm-admin list
pmm-admin 1.0.6

PMM Server      | 192.168.12.3
Client Name     | localhost.localdomain
Client Address  | 192.168.12.4
Service manager | unix-systemv

-------------- ---------------------- ------------ -------- -------------------------------------------- ---------------------
SERVICE TYPE   NAME                   CLIENT PORT  RUNNING  DATA SOURCE                                  OPTIONS
-------------- ---------------------- ------------ -------- -------------------------------------------- ---------------------
linux:metrics  localhost.localdomain  42000        YES      -
mysql:queries  localhost.localdomain  42001        YES      root:***@unix(/mysql/MyHome/tmp/mysql.sock)  query_source=slowlog
mysql:metrics  localhost.localdomain  42002        YES      root:***@unix(/mysql/MyHome/tmp/mysql.sock)

これでメトリクスを取得し、グラフに表示できるようになった

冒険の書

PMM(Percona Monitoring and Management)でmetricsの取得ができなかった場合

グラフで取得対象に表示されていなかった

考えられる原因

ここまでに行ったこと

試行錯誤一覧