[ceph-users] Raw space usage in Ceph with Bluestore

[Glider, Jody]

Hello,

I’m trying to find a way to determine real/physical/raw storage capacity usage when storing a similar set of objects in different pools, for example a 3-way replicated pool vs. a 4+2 erasure coded pool, and in particular how this ratio changes from small (where Bluestore block size matters more) to large object sizes.

I find that ceph df detail and rados df don’t report on really-raw storage, I guess because they’re perceiving ‘raw’ storage from their perspective only. If I write a set of objects to each pool, rados df shows the space used as the summation of the logical size of the objects, while ceph df detail shows the raw used storage as the object size * the redundancy factor (e.g. 3 for 3-way replication and 1.5 for 4+2 erasure code).

Any suggestions?

Jody Glider, Principal Storage Architect
Cloud Architecture and Engineering, SAP Labs LLC
3412 Hillview Ave (PAL 02 23.357), Palo Alto, CA 94304
E ***@sap.com<mailto:***@sap.com>, T [phone], M [phone]

[Paul Emmerich]

You can get all the details from the admin socket of the OSDs:

ceph daemon osd.X perf dump

(must be run on the server the OSD is running on)

Examples of relevant metrics are: bluestore_allocated/stored and the
bluefs block for metadata.
Running perf schema might contain some details on the meaning of the
individual metrics.
--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: [phone]

Post by Glider, Jody
Hello,
I’m trying to find a way to determine real/physical/raw storage capacity usage when storing a similar set of objects in different pools, for example a 3-way replicated pool vs. a 4+2 erasure coded pool, and in particular how this ratio changes from small (where Bluestore block size matters more) to large object sizes.
I find that ceph df detail and rados df don’t report on really-raw storage, I guess because they’re perceiving ‘raw’ storage from their perspective only. If I write a set of objects to each pool, rados df shows the space used as the summation of the logical size of the objects, while ceph df detail shows the raw used storage as the object size * the redundancy factor (e.g. 3 for 3-way replication and 1.5 for 4+2 erasure code).
Any suggestions?
Jody Glider, Principal Storage Architect
Cloud Architecture and Engineering, SAP Labs LLC
3412 Hillview Ave (PAL 02 23.357), Palo Alto, CA 94304
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Igor Fedotov]

Hi Jody,

yes, this is a known issue.

Indeed, currently 'ceph df detail' reports raw space usage in GLOBAL
section and 'logical' in the POOLS one. While logical one has some flaws.

There is a pending PR targeted to Nautilus to fix that:

https://github.com/ceph/ceph/pull/19454

If you want to do an analysis at exactly per-pool level this PR is the
only mean AFAIK.


If per-cluster stats are fine then you can also inspect corresponding
OSD performance counters and sum over all OSDs to get per-cluster info.

This is the most precise but quite inconvenient method for low-level
per-osd space analysis.

 "bluestore": {
...

       "bluestore_allocated": 655360, # space allocated at BlueStore
for the specific OSD
        "bluestore_stored": 34768,  # amount of data stored at
BlueStore for the specific OSD
...

Please note, that aggregate numbers built from these parameters include
all the replication/EC overhead.  And bluestore_stored vs.
bluestore_allocated difference is due to allocation overhead and/or
applied compression.


Thanks,

Igor

Post by Glider, Jody
Hello,
I’m trying to find a way to determine real/physical/raw storage
capacity usage when storing a similar set of objects in different
pools, for example a 3-way replicated pool vs. a 4+2 erasure coded
pool, and in particular how this ratio changes from small (where
Bluestore block size matters more) to large object sizes.
I find that /ceph df detail/ and /rados df/ don’t report on really-raw
storage, I guess because they’re perceiving ‘raw’ storage from their
perspective only. If I write a set of objects to each pool, rados df
shows the space used as the summation of the logical size of the
objects, while ceph df detail shows the raw used storage as the object
size * the redundancy factor (e.g. 3 for 3-way replication and 1.5 for
4+2 erasure code).
Any suggestions?
Jody Glider, Principal Storage Architect
Cloud Architecture and Engineering, SAP Labs LLC
3412 Hillview Ave (PAL 02 23.357), Palo Alto, CA 94304
[phone]
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com