[ceph-users] Difference between "ceph osd reweight" and "ceph osd crush reweight"

[Micha Krause]

Hi,

could someone explain to me what the difference is between

ceph osd reweight

and

ceph osd crush reweight


Micha Krause

[Gregory Farnum]

Post by Micha Krause
Hi,
could someone explain to me what the difference is between
ceph osd reweight
and
ceph osd crush reweight

"ceph osd crush reweight" sets the CRUSH weight of the OSD. This
weight is an arbitrary value (generally the size of the disk in TB or
something) and controls how much data the system tries to allocate to
the OSD.

"ceph osd reweight" sets an override weight on the OSD. This value is
in the range 0 to 1, and forces CRUSH to re-place (1-weight) of the
data that would otherwise live on this drive. It does *not* change the
weights assigned to the buckets above the OSD, and is a corrective
measure in case the normal CRUSH distribution isn't working out quite
right. (For instance, if one of your OSDs is at 90% and the others are
at 50%, you could reduce this weight to try and compensate for it.)

It looks like our docs aren't very clear on the difference, when it
even mentions them...and admittedly it's a pretty subtle issue!
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

[Craig Lewis]

Note that 'ceph osd reweight' is not a persistent setting. When an OSD
gets marked out, the osd weight will be set to 0. When it gets marked in
again, the weight will be changed to 1.

Because of this 'ceph osd reweight' is a temporary solution. You should
only use it to keep your cluster running while you're ordering more
hardware.

Post by Gregory Farnum

Post by Micha Krause
Hi,
could someone explain to me what the difference is between
ceph osd reweight
and
ceph osd crush reweight

"ceph osd crush reweight" sets the CRUSH weight of the OSD. This
weight is an arbitrary value (generally the size of the disk in TB or
something) and controls how much data the system tries to allocate to
the OSD.
"ceph osd reweight" sets an override weight on the OSD. This value is
in the range 0 to 1, and forces CRUSH to re-place (1-weight) of the
data that would otherwise live on this drive. It does *not* change the
weights assigned to the buckets above the OSD, and is a corrective
measure in case the normal CRUSH distribution isn't working out quite
right. (For instance, if one of your OSDs is at 90% and the others are
at 50%, you could reduce this weight to try and compensate for it.)
It looks like our docs aren't very clear on the difference, when it
even mentions them...and admittedly it's a pretty subtle issue!
-Greg
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140626/ca887357/attachment.htm>

[Micha Krause]

Hi,

Post by Gregory Farnum
"ceph osd crush reweight" sets the CRUSH weight of the OSD. This
weight is an arbitrary value (generally the size of the disk in TB or
something) and controls how much data the system tries to allocate to
the OSD.
"ceph osd reweight" sets an override weight on the OSD. This value is
in the range 0 to 1, and forces CRUSH to re-place (1-weight) of the
data that would otherwise live on this drive. It does *not* change the
weights assigned to the buckets above the OSD, and is a corrective
measure in case the normal CRUSH distribution isn't working out quite
right. (For instance, if one of your OSDs is at 90% and the others are
at 50%, you could reduce this weight to try and compensate for it.)

thanks, so if I have some older osds, and I want them to receive less data/iops
than the other nodes, I would use "ceph osd crush reweight"?

Micha Krause

[Gregory Farnum]

Yep, definitely use "osd crush reweigh" for your permanent data placement.

Post by Micha Krause
Hi,

Post by Gregory Farnum
"ceph osd crush reweight" sets the CRUSH weight of the OSD. This
weight is an arbitrary value (generally the size of the disk in TB or
something) and controls how much data the system tries to allocate to
the OSD.
"ceph osd reweight" sets an override weight on the OSD. This value is
in the range 0 to 1, and forces CRUSH to re-place (1-weight) of the
data that would otherwise live on this drive. It does *not* change the
weights assigned to the buckets above the OSD, and is a corrective
measure in case the normal CRUSH distribution isn't working out quite
right. (For instance, if one of your OSDs is at 90% and the others are
at 50%, you could reduce this weight to try and compensate for it.)

thanks, so if I have some older osds, and I want them to receive less data/iops
than the other nodes, I would use "ceph osd crush reweight"?
Micha Krause

[Kostis Fardelas]

Hi,
from my experience both "ceph osd crush reweight" and "ceph osd
reweight" will lead to CRUSH map changes and PGs remapping. So both
commands eventually redistribute data beween OSDs. Is there any good
reason in terms of ceph performance best practices to choose the one
over the other?

Post by Gregory Farnum

Post by Micha Krause
Hi,
could someone explain to me what the difference is between
ceph osd reweight
and
ceph osd crush reweight

"ceph osd crush reweight" sets the CRUSH weight of the OSD. This
weight is an arbitrary value (generally the size of the disk in TB or
something) and controls how much data the system tries to allocate to
the OSD.
"ceph osd reweight" sets an override weight on the OSD. This value is
in the range 0 to 1, and forces CRUSH to re-place (1-weight) of the
data that would otherwise live on this drive. It does *not* change the
weights assigned to the buckets above the OSD, and is a corrective
measure in case the normal CRUSH distribution isn't working out quite
right. (For instance, if one of your OSDs is at 90% and the others are
at 50%, you could reduce this weight to try and compensate for it.)
It looks like our docs aren't very clear on the difference, when it
even mentions them...and admittedly it's a pretty subtle issue!
-Greg
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Wido den Hollander]

Post by Kostis Fardelas
Hi,
from my experience both "ceph osd crush reweight" and "ceph osd
reweight" will lead to CRUSH map changes and PGs remapping. So both
commands eventually redistribute data beween OSDs. Is there any good
reason in terms of ceph performance best practices to choose the one
over the other?

The latter wil only change the weight of the OSD and not the weight of
the host, so that will cause slightly less movement of data and it will
only migrate within that host.

Wido

Post by Kostis Fardelas

Post by Gregory Farnum

Post by Micha Krause
Hi,
could someone explain to me what the difference is between
ceph osd reweight
and
ceph osd crush reweight

"ceph osd crush reweight" sets the CRUSH weight of the OSD. This
weight is an arbitrary value (generally the size of the disk in TB or
something) and controls how much data the system tries to allocate to
the OSD.
"ceph osd reweight" sets an override weight on the OSD. This value is
in the range 0 to 1, and forces CRUSH to re-place (1-weight) of the
data that would otherwise live on this drive. It does *not* change the
weights assigned to the buckets above the OSD, and is a corrective
measure in case the normal CRUSH distribution isn't working out quite
right. (For instance, if one of your OSDs is at 90% and the others are
at 50%, you could reduce this weight to try and compensate for it.)
It looks like our docs aren't very clear on the difference, when it
even mentions them...and admittedly it's a pretty subtle issue!
-Greg
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com