[ceph-users] Manual deep scrub

[Richard Arends]

Hi,


When i start a deep scrub on a PG by hand 'ceph pg deep-scrub 1.18d5',
sometimes the deep scrub is executed direct after the command is
entered, but often it's not there is a lot of time between starting and
executing. For example:

2017-01-17 05:25:31.786 session 01162017 :: Starting deep-scrub on pg
1.1a39 for pool openstack_volumes
2017-01-17 06:37:48.135325 osd.1120 <ip>:6814/4237 263 : cluster [INF]
1.1a39 deep-scrub starts
2017-01-17 06:58:07.651926 osd.1120 <ip>:6814/4237 264 : cluster [INF]
1.1a39 deep-scrub ok

De first log line is from our 'deep scrub cron script'.

My question is, what defines how and when and how long a deep scrub is
queued and is there a way to enforce the deep scrub 'now'. I am looking
into this, because in the past we could deep scrub a specific set in 24
hours and now that's not possible anymore.



--
Regards,

Richard.

[David Turner]

You want to look into the settings osd_max_scrubs which indicates how many different scrub operations an OSD can be involved in at once (the well chosen default is 1), as well as osd_scrub_max_interval and osd_deep_scrub_interval. One of the differences in your cluster from before to now is time.

It might be that you're PGs are running into their max interval and forcing to scrub now which is interfering with your script. We also manage our deep scrubs with a cron and have set our osd_deep_scrub_scrub interval to longer than it will take our cron to go through all of the PGs so that they will never automatically deep scrub themselves.

For reference please refer to the osd config ceph document for all of the osd scrub settings available to you. http://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/

________________________________

[cid:***@9b095740.4da3a690]<https://storagecraft.com> David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation<https://storagecraft.com>
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2760 | Mobile: [phone]

________________________________

If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited.

________________________________

________________________________
From: ceph-users [ceph-users-***@lists.ceph.com] on behalf of Richard Arends [***@mosibi.nl]
Sent: Tuesday, January 17, 2017 8:03 AM
To: ceph-***@lists.ceph.com
Subject: [ceph-users] Manual deep scrub


Hi,


When i start a deep scrub on a PG by hand 'ceph pg deep-scrub 1.18d5', sometimes the deep scrub is executed direct after the command is entered, but often it's not there is a lot of time between starting and executing. For example:

2017-01-17 05:25:31.786 session 01162017 :: Starting deep-scrub on pg 1.1a39 for pool openstack_volumes
2017-01-17 06:37:48.135325 osd.1120 <ip>:6814/4237 263 : cluster [INF] 1.1a39 deep-scrub starts
2017-01-17 06:58:07.651926 osd.1120 <ip>:6814/4237 264 : cluster [INF] 1.1a39 deep-scrub ok

De first log line is from our 'deep scrub cron script'.

My question is, what defines how and when and how long a deep scrub is queued and is there a way to enforce the deep scrub 'now'. I am looking into this, because in the past we could deep scrub a specific set in 24 hours and now that's not possible anymore.



--
Regards,

Richard.

[Richard Arends]

On 01/17/2017 04:09 PM, David Turner wrote:

Hi,

> You want to look into the settings osd_max_scrubs which indicates how
> many different scrub operations an OSD can be involved in at once

That's still the default, thus 1. A pg that i wanted to deep-scrub this
afternoon, should be done by an OSD that was not doing another
deep-scrub. Also that OSD was not busy with anything else. So to my
knowledge, the deep-scrub should start.

> (the well chosen default is 1), as well as osd_scrub_max_interval and
> osd_deep_scrub_interval. One of the differences in your cluster from
> before to now is time.
>
> It might be that you're PGs are running into their max interval and
> forcing to scrub now which is interfering with your script. We also
> manage our deep scrubs with a cron and have set our
> osd_deep_scrub_scrub interval to longer than it will take our cron to
> go through all of the PGs so that they will never automatically deep
> scrub themselves.

We are doing the same in our cluster, i also set the the
osd_deep_scrub_interval to a high value. The normal scrub intervals
(osd_scrub_min_interval and osd_scrub_max_interval) are set to the
defaults (1 day and 1 week).


--
Regards,

Richard.

[David Turner]

All OSDs with a copy of the PG need to not be involved in any scrub for the scrub to start immediately. It is not just the primary OSD but all secondary OSDs as well for a scrub to be able to run on a PG.

________________________________

[cid:***@52ae1513.42ace176]<https://storagecraft.com> David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation<https://storagecraft.com>
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2760 | Mobile: [phone]

________________________________

If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited.

________________________________

________________________________
From: ceph-users [ceph-users-***@lists.ceph.com] on behalf of Richard Arends [***@mosibi.nl]
Sent: Tuesday, January 17, 2017 8:27 AM
To: ceph-***@lists.ceph.com
Subject: Re: [ceph-users] Manual deep scrub

On 01/17/2017 04:09 PM, David Turner wrote:

Hi,

You want to look into the settings osd_max_scrubs which indicates how many different scrub operations an OSD can be involved in at once

That's still the default, thus 1. A pg that i wanted to deep-scrub this afternoon, should be done by an OSD that was not doing another deep-scrub. Also that OSD was not busy with anything else. So to my knowledge, the deep-scrub should start.

(the well chosen default is 1), as well as osd_scrub_max_interval and osd_deep_scrub_interval. One of the differences in your cluster from before to now is time.

It might be that you're PGs are running into their max interval and forcing to scrub now which is interfering with your script. We also manage our deep scrubs with a cron and have set our osd_deep_scrub_scrub interval to longer than it will take our cron to go through all of the PGs so that they will never automatically deep scrub themselves.

We are doing the same in our cluster, i also set the the osd_deep_scrub_interval to a high value. The normal scrub intervals (osd_scrub_min_interval and osd_scrub_max_interval) are set to the defaults (1 day and 1 week).



--
Regards,

Richard.

[Richard Arends]

On 01/17/2017 05:15 PM, David Turner wrote:

David,

> All OSDs with a copy of the PG need to not be involved in any scrub
> for the scrub to start immediately. It is not just the primary OSD
> but all secondary OSDs as well for a scrub to be able to run on a PG.

I thought of that and checked if the other (2 in this case) where doing
something and on the first sight it did not look like that. Tomorrow i
will do some more tests. Thanks for the pointers !


--
Regards,

Richard.

[David Turner]

Looking through the additional osd config options for scrubbing show a couple options that can prevent a PG from scrubbing immediately.

osd_scrub_during_recovery - default true - If false, no new scrub can be scheduled while their is active recovery.
osd_scrub_load_threshold - default 0.5 - Ceph will not scrub when the system load is higher than this number.

osd_scrub_load_threshold seems like a likely cause of what you are seeing.



________________________________

[cid:***@bfc72d99.44bb939f]<https://storagecraft.com> David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation<https://storagecraft.com>
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2760 | Mobile: [phone]

________________________________

If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited.

________________________________

________________________________
From: ceph-users [ceph-users-***@lists.ceph.com] on behalf of Richard Arends [***@mosibi.nl]
Sent: Tuesday, January 17, 2017 1:07 PM
To: ceph-***@lists.ceph.com
Subject: Re: [ceph-users] Manual deep scrub

On 01/17/2017 05:15 PM, David Turner wrote:

David,

All OSDs with a copy of the PG need to not be involved in any scrub for the scrub to start immediately. It is not just the primary OSD but all secondary OSDs as well for a scrub to be able to run on a PG.

I thought of that and checked if the other (2 in this case) where doing something and on the first sight it did not look like that. Tomorrow i will do some more tests. Thanks for the pointers !


--
Regards,

Richard.

[Richard Arends]

On 01/17/2017 09:21 PM, David Turner wrote:


> Looking through the additional osd config options for scrubbing show a
> couple options that can prevent a PG from scrubbing immediately.
>
> osd_scrub_during_recovery - default true - If false, no new scrub can
> be scheduled while their is active recovery.

I will check this setting tomorrow, but i am pretty sure that's it still
set to the default value. Our cluster was most of the time today in a
healthy state.

> osd_scrub_load_threshold - default 0.5 - Ceph will not scrub when the
> system load is higher than this number.
>
> osd_scrub_load_threshold seems like a likely cause of what you are seeing.

This one was also on my list, so i upped this to 5.0 on all OSDs for a
brief moment and still nothing happened. But i did not check if the
'copy OSD's' where below 5 at that moment.

--
Regards,

Richard.