Discussion:
osd_recovery_max_chunk value
Add Reply
Karun Josy
2018-02-06 07:31:12 UTC
Reply
Permalink
Raw Message
Hello,

We are seeing slow requests while recovery process going on.

I am trying to slow down the recovery process. I set osd_recovery_max_active
and osd_recovery_sleep as below :
----------
ceph tell osd.* injectargs '--osd_recovery_max_active 1'
ceph tell osd.* injectargs '--osd_recovery_sleep .1'
----------

But I am confused with the osd_recovery_max_chunk. Currently, it shows
8388608.

# ceph daemon osd.4 config get osd_recovery_max_chunk
{
"osd_recovery_max_chunk": "8388608"


In ceph documentation, it shows

-----------
osd recovery max chunk
Description: The maximum size of a recovered chunk of data to push.
Type: 64-bit Unsigned Integer
Default: 8 << 20
------------

I am confused. Can anyone let me know what is the value that I have to give
to reduce this parameter ?



Karun Josy
Christian Balzer
2018-02-06 07:45:06 UTC
Reply
Permalink
Raw Message
Post by Karun Josy
Hello,
We are seeing slow requests while recovery process going on.
I am trying to slow down the recovery process. I set osd_recovery_max_active
----------
ceph tell osd.* injectargs '--osd_recovery_max_active 1'
ceph tell osd.* injectargs '--osd_recovery_sleep .1'
----------
What version of Ceph, in some "sleep" values will make things _worse_!
Would be nice if that was documented in like, the documentation...
Post by Karun Josy
But I am confused with the osd_recovery_max_chunk. Currently, it shows
8388608.
# ceph daemon osd.4 config get osd_recovery_max_chunk
{
"osd_recovery_max_chunk": "8388608"
In ceph documentation, it shows
-----------
osd recovery max chunk
Description: The maximum size of a recovered chunk of data to push.
Type: 64-bit Unsigned Integer
Default: 8 << 20
------------
I am confused. Can anyone let me know what is the value that I have to give
to reduce this parameter ?
This is what you get when programmers write docs.

The above is a left-shift operation, see for example:
http://bit-calculator.com/bit-shift-calculator

Now if shrinking that value is beneficial for reducing recovery load,
that's for you to find out.

Christian
Post by Karun Josy
Karun Josy
--
Christian Balzer Network/Systems Engineer
***@gol.com Rakuten Communications
Karun Josy
2018-02-06 07:57:22 UTC
Reply
Permalink
Raw Message
Hi Christian,

Thank you for your help.

Ceph version is 12.2.2. So is this value bad ? Do you have any suggestions ?


So to reduce the max chunk ,I assume I can choose something like
7 << 20 ,ie 7340032 ?

Karun Josy
Post by Karun Josy
Post by Karun Josy
Hello,
We are seeing slow requests while recovery process going on.
I am trying to slow down the recovery process. I set
osd_recovery_max_active
Post by Karun Josy
----------
ceph tell osd.* injectargs '--osd_recovery_max_active 1'
ceph tell osd.* injectargs '--osd_recovery_sleep .1'
----------
What version of Ceph, in some "sleep" values will make things _worse_!
Would be nice if that was documented in like, the documentation...
Post by Karun Josy
But I am confused with the osd_recovery_max_chunk. Currently, it shows
8388608.
# ceph daemon osd.4 config get osd_recovery_max_chunk
{
"osd_recovery_max_chunk": "8388608"
In ceph documentation, it shows
-----------
osd recovery max chunk
Description: The maximum size of a recovered chunk of data to push.
Type: 64-bit Unsigned Integer
Default: 8 << 20
------------
I am confused. Can anyone let me know what is the value that I have to
give
Post by Karun Josy
to reduce this parameter ?
This is what you get when programmers write docs.
http://bit-calculator.com/bit-shift-calculator
Now if shrinking that value is beneficial for reducing recovery load,
that's for you to find out.
Christian
Post by Karun Josy
Karun Josy
--
Christian Balzer Network/Systems Engineer
Christian Balzer
2018-02-06 08:00:19 UTC
Reply
Permalink
Raw Message
Post by Karun Josy
Hi Christian,
Thank you for your help.
Ceph version is 12.2.2. So is this value bad ? Do you have any suggestions ?
So to reduce the max chunk ,I assume I can choose something like
7 << 20 ,ie 7340032 ?
More like 4MB to match things up nicely in the binary world.

Christian
Post by Karun Josy
Karun Josy
Post by Karun Josy
Post by Karun Josy
Hello,
We are seeing slow requests while recovery process going on.
I am trying to slow down the recovery process. I set
osd_recovery_max_active
Post by Karun Josy
----------
ceph tell osd.* injectargs '--osd_recovery_max_active 1'
ceph tell osd.* injectargs '--osd_recovery_sleep .1'
----------
What version of Ceph, in some "sleep" values will make things _worse_!
Would be nice if that was documented in like, the documentation...
Post by Karun Josy
But I am confused with the osd_recovery_max_chunk. Currently, it shows
8388608.
# ceph daemon osd.4 config get osd_recovery_max_chunk
{
"osd_recovery_max_chunk": "8388608"
In ceph documentation, it shows
-----------
osd recovery max chunk
Description: The maximum size of a recovered chunk of data to push.
Type: 64-bit Unsigned Integer
Default: 8 << 20
------------
I am confused. Can anyone let me know what is the value that I have to
give
Post by Karun Josy
to reduce this parameter ?
This is what you get when programmers write docs.
http://bit-calculator.com/bit-shift-calculator
Now if shrinking that value is beneficial for reducing recovery load,
that's for you to find out.
Christian
Post by Karun Josy
Karun Josy
--
Christian Balzer Network/Systems Engineer
--
Christian Balzer Network/Systems Engineer
***@gol.com Rakuten Communications
Christian Balzer
2018-02-06 07:59:09 UTC
Reply
Permalink
Raw Message
Post by Karun Josy
Hi Christian,
Thank you for your help.
Ceph version is 12.2.2. So is this value bad ? Do you have any suggestions ?
That should be fine AFAIK, some (all?) versions for Jewel definitely are
not.
Post by Karun Josy
----
ceph tell osd.* injectargs '--osd_recovery_sleep .1'
-----
Karun Josy
Post by Karun Josy
Post by Karun Josy
Hello,
We are seeing slow requests while recovery process going on.
I am trying to slow down the recovery process. I set
osd_recovery_max_active
Post by Karun Josy
----------
ceph tell osd.* injectargs '--osd_recovery_max_active 1'
ceph tell osd.* injectargs '--osd_recovery_sleep .1'
----------
What version of Ceph, in some "sleep" values will make things _worse_!
Would be nice if that was documented in like, the documentation...
Post by Karun Josy
But I am confused with the osd_recovery_max_chunk. Currently, it shows
8388608.
# ceph daemon osd.4 config get osd_recovery_max_chunk
{
"osd_recovery_max_chunk": "8388608"
In ceph documentation, it shows
-----------
osd recovery max chunk
Description: The maximum size of a recovered chunk of data to push.
Type: 64-bit Unsigned Integer
Default: 8 << 20
------------
I am confused. Can anyone let me know what is the value that I have to
give
Post by Karun Josy
to reduce this parameter ?
This is what you get when programmers write docs.
http://bit-calculator.com/bit-shift-calculator
Now if shrinking that value is beneficial for reducing recovery load,
that's for you to find out.
Christian
Post by Karun Josy
Karun Josy
--
Christian Balzer Network/Systems Engineer
--
Christian Balzer Network/Systems Engineer
***@gol.com Rakuten Communications
Loading...