[ceph-users] get error when use prometheus plugin of ceph-mgr

Discussion:

shawn tim

2017-08-30 02:47:38 UTC

Hello,
I just want to try prometheus plugin of ceph-mgr.
Following this doc(http://docs.ceph.com/docs/master/mgr/prometheus/)
<http://docs.ceph.com/docs/master/mgr/prometheus/)ïŒI>. I get output like

[***@ceph01 ~]# curl localhost:9283/metrics/ | head
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0
# HELP mds_mem_dir_ Directories closed
# TYPE mds_mem_dir_ counter
mds_mem_dir_{daemon="mds.ceph01"} 0.0
# HELP throttle_msgr_dispatch_throttler_radosclient_max Max value for
throttle
# TYPE throttle_msgr_dispatch_throttler_radosclient_max gauge
throttle_msgr_dispatch_throttler_radosclient_max{daemon="rgw.ceph01"}
104857600.0
# HELP mds_mem_dir+ Directories opened
# TYPE mds_mem_dir+ counter
mds_mem_dir+{daemon="mds.ceph01"} 12.0
...

It seems everythins was OK. But when prometheus connect it, I got errors:

WARN[0002] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
WARN[0007] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
WARN[0012] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
^CWARN[0014] Received SIGTERM, exiting gracefully...
source="main.go:340"

My prometheus config yml is

- job_name: 'ceph mgr'
scrape_interval: 5s
# metrics_path: /metrics/
# uncomment this still get error (Get http://10.10.0.125:9283/metrics:
net/http: invalid header field value "Bearer ..." for key Authorization )
# bearer_token_file: /etc/ceph/ceph.mgr.ceph01.keyring
# bearer_token: AQD6359ZCMK3DxAAyJYErZayMX/CRZyiyk/UGg==

static_configs:
- targets: ['10.10.0.125:9283']
labels:
group: 'production'

My ceph version is

[***@ceph01 ~]# ceph --version
ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)

My prometheus version is

[***@prom01 prometheus-2.0.0-beta.2.linux-amd64]# ./prometheus --version
prometheus, version 2.0.0-beta.2 (branch: HEAD, revision:
a52f082939a566d5269671e98be06fc6bdf61d09)
build user: ***@41a0740ea598
build date: 20170818-08:16:50
go version: go1.8.3

Is there any steps I missed

Thanks for help !

John Spray

2017-08-30 11:33:46 UTC

Permalink

Post by shawn tim
Hello,
I just want to try prometheus plugin of ceph-mgr.
Following this doc(http://docs.ceph.com/docs/master/mgr/prometheus/). I get
output like
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0
# HELP mds_mem_dir_ Directories closed
# TYPE mds_mem_dir_ counter
mds_mem_dir_{daemon="mds.ceph01"} 0.0
# HELP throttle_msgr_dispatch_throttler_radosclient_max Max value for
throttle
# TYPE throttle_msgr_dispatch_throttler_radosclient_max gauge
throttle_msgr_dispatch_throttler_radosclient_max{daemon="rgw.ceph01"}
104857600.0
# HELP mds_mem_dir+ Directories opened
# TYPE mds_mem_dir+ counter
mds_mem_dir+{daemon="mds.ceph01"} 12.0
.

I notice you have some MDS metrics in there, which cause some invalid
output -- can you try this fix?
https://github.com/ceph/ceph/pull/17318

The "no token found stuff" is not familiar to me: when testing the
prometheus plugin I've always just been using it without any
authentication.

John

Post by shawn tim
WARN[0002] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
WARN[0007] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
WARN[0012] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
^CWARN[0014] Received SIGTERM, exiting gracefully...
source="main.go:340"
My prometheus config yml is
- job_name: 'ceph mgr'
scrape_interval: 5s
# metrics_path: /metrics/
net/http: invalid header field value "Bearer ..." for key Authorization )
# bearer_token_file: /etc/ceph/ceph.mgr.ceph01.keyring
# bearer_token: AQD6359ZCMK3DxAAyJYErZayMX/CRZyiyk/UGg==
- targets: ['10.10.0.125:9283']
group: 'production'
My ceph version is
ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
My prometheus version is
a52f082939a566d5269671e98be06fc6bdf61d09)
build date: 20170818-08:16:50
go version: go1.8.3
Is there any steps I missed
Thanks for help !
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

shawn tim

2017-08-30 16:54:14 UTC

Permalink

Hi, John,
Do you mean this error
text format parsing error in line 8: invalid metric name in comment
I install ceph from rpm, but I will try to patch it.
many thanks

when using prometheus 1.7.1 instead of 2.0.0-beta, the "no token found error"
disappeared.
Maybe prometheus 2.0.0-beta still has bugs about it.

Post by John Spray

Post by shawn tim
Hello,
I just want to try prometheus plugin of ceph-mgr.
Following this doc(http://docs.ceph.com/docs/master/mgr/prometheus/). I

get

Post by shawn tim
output like
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0
# HELP mds_mem_dir_ Directories closed
# TYPE mds_mem_dir_ counter
mds_mem_dir_{daemon="mds.ceph01"} 0.0
# HELP throttle_msgr_dispatch_throttler_radosclient_max Max value for
throttle
# TYPE throttle_msgr_dispatch_throttler_radosclient_max gauge
throttle_msgr_dispatch_throttler_radosclient_max{daemon="rgw.ceph01"}
104857600.0
# HELP mds_mem_dir+ Directories opened
# TYPE mds_mem_dir+ counter
mds_mem_dir+{daemon="mds.ceph01"} 12.0
.

I notice you have some MDS metrics in there, which cause some invalid
output -- can you try this fix?
https://github.com/ceph/ceph/pull/17318
The "no token found stuff" is not familiar to me: when testing the
prometheus plugin I've always just been using it without any
authentication.
John

Post by shawn tim
It seems everythins was OK. But when prometheus connect it, I got
WARN[0002] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
WARN[0007] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
WARN[0012] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
^CWARN[0014] Received SIGTERM, exiting gracefully...
source="main.go:340"
My prometheus config yml is
- job_name: 'ceph mgr'
scrape_interval: 5s
# metrics_path: /metrics/
# uncomment this still get error (Get
net/http: invalid header field value "Bearer ..." for key Authorization )
# bearer_token_file: /etc/ceph/ceph.mgr.ceph01.keyring
# bearer_token: AQD6359ZCMK3DxAAyJYErZayMX/CRZyiyk/UGg==
- targets: ['10.10.0.125:9283']
group: 'production'
My ceph version is
ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous

(rc)

Post by shawn tim
My prometheus version is

--version

Post by shawn tim
a52f082939a566d5269671e98be06fc6bdf61d09)
build date: 20170818-08:16:50
go version: go1.8.3
Is there any steps I missed
Thanks for help !
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

John Spray

2017-08-30 16:55:38 UTC

Permalink

Post by shawn tim
Hi, John,
Do you mean this error
text format parsing error in line 8: invalid metric name in comment

Yes, iirc that's the one you get from the plus/minus signs in MDS metric names.

Post by shawn tim
I install ceph from rpm, but I will try to patch it.
many thanks
when using prometheus 1.7.1 instead of 2.0.0-beta, the "no token found
error" disappeared.
Maybe prometheus 2.0.0-beta still has bugs about it.

Huh, weird, I think I was only ever using 2.0. No idea then!

John

Post by shawn tim

Post by John Spray

I notice you have some MDS metrics in there, which cause some invalid
output -- can you try this fix?
https://github.com/ceph/ceph/pull/17318
The "no token found stuff" is not familiar to me: when testing the
prometheus plugin I've always just been using it without any
authentication.
John

Post by shawn tim
WARN[0002] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
WARN[0007] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
WARN[0012] append failed err="no token
found" source="scrape.go:648" target="{__address__="10.10.0.125:9283",
__metrics_path__="/metric
s", __scheme__="http", group="production", instance="10.10.0.125:9283",
job="ceph"}"
^CWARN[0014] Received SIGTERM, exiting gracefully...
source="main.go:340"
My prometheus config yml is
- job_name: 'ceph mgr'
scrape_interval: 5s
# metrics_path: /metrics/
# uncomment this still get error (Get
net/http: invalid header field value "Bearer ..." for key Authorization )
# bearer_token_file: /etc/ceph/ceph.mgr.ceph01.keyring
# bearer_token: AQD6359ZCMK3DxAAyJYErZayMX/CRZyiyk/UGg==
- targets: ['10.10.0.125:9283']
group: 'production'
My ceph version is
ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
My prometheus version is
a52f082939a566d5269671e98be06fc6bdf61d09)
build date: 20170818-08:16:50
go version: go1.8.3
Is there any steps I missed
Thanks for help !
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com