[ceph-users] rbd IO monitoring

[Michael Green]

Hello collective wisdom,

Ceph neophyte here, running v13.2.2 (mimic).

Question: what tools are available to monitor IO stats on RBD level? That is, IOPS, Throughput, IOs inflight and so on?
I'm testing with FIO and want to verify independently the IO load on each RBD image.

--
Michael Green
Customer Support & Integration
***@e8storage.com

[Wido den Hollander]

Post by Michael Green
Hello collective wisdom,
Ceph neophyte here, running v13.2.2 (mimic).
Question: what tools are available to monitor IO stats on RBD level?
That is, IOPS, Throughput, IOs inflight and so on?
I'm testing with FIO and want to verify independently the IO load on each RBD image.

There is no central point in Ceph where all I/O passes, so those
counters can only be found on the client issuing the I/O.

If you enable the admin socket on a librbd client you can use the 'perf
dump' command so see what it's doing.

This is how you enable the socket:
https://ceph.com/geen-categorie/ceph-validate-that-the-rbd-cache-is-active/

Wido

Post by Michael Green
--
*Michael Green*
Customer Support & Integration
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Jan Fajerski]

Post by Michael Green
Hello collective wisdom,
Ceph neophyte here, running v13.2.2 (mimic).
Question: what tools are available to monitor IO stats on RBD level?
That is, IOPS, Throughput, IOs inflight and so on?

There is some brand new code for rbd io monitoring. This PR
(https://github.com/ceph/ceph/pull/25114) added rbd client side perf counters
and this PR (https://github.com/ceph/ceph/pull/25358) will add those counters as
prometheus metrics. There is also room for an "rbd top" tool, though I haven't
seen any code for this.
I'm sure Mykola (the author of both PRs) could go into more detail if needed. I
expect this functionality to land in nautilus.

Post by Michael Green
I'm testing with FIO and want to verify independently the IO load on each RBD image.
--
Michael Green
Customer Support & Integration
References
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Michael Green]

Interesting, thanks for sharing.

I'm looking at the example output in the PR 25114:

write_bytes
409600/107
409600/107

write_latency
2618503617/107

How should these values be interpreted?
--
Michael Green

Post by Michael Green
Question: what tools are available to monitor IO stats on RBD level?
That is, IOPS, Throughput, IOs inflight and so on?

There is some brand new code for rbd io monitoring. This PR (https://github.com/ceph/ceph/pull/25114) added rbd client side perf counters and this PR (https://github.com/ceph/ceph/pull/25358) will add those counters as prometheus metrics. There is also room for an "rbd top" tool, though I haven't seen any code for this.
I'm sure Mykola (the author of both PRs) could go into more detail if needed. I expect this functionality to land in nautilus.

[Jason Dillaman]

The "osd_perf_query" mgr module is just a demo / testing framework.
However, the output was tweaked prior to merge to provide more
readable values instead of the "{value summation} / {count}" in the
original submission.

Post by Michael Green
Interesting, thanks for sharing.
write_bytes
409600/107
409600/107
write_latency
2618503617/107
How should these values be interpreted?
--
Michael Green

Post by Michael Green
Question: what tools are available to monitor IO stats on RBD level?
That is, IOPS, Throughput, IOs inflight and so on?

There is some brand new code for rbd io monitoring. This PR (https://github.com/ceph/ceph/pull/25114) added rbd client side perf counters and this PR (https://github.com/ceph/ceph/pull/25358) will add those counters as prometheus metrics. There is also room for an "rbd top" tool, though I haven't seen any code for this.
I'm sure Mykola (the author of both PRs) could go into more detail if needed. I expect this functionality to land in nautilus.

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com