[ceph-users] Huge latency spikes

Discussion:

Alex Litvak

2018-11-17 19:17:07 UTC

Hello everyone,

I am trying to troubleshoot cluster exhibiting huge spikes of latency.
I cannot quite catch it because it happens during the light activity and
randomly affects one osd node out of 3 in the pool.

This is a file store.
I see some osds exhibit applied latency of 400 ms, 1 minute load
average shuts to 60. Client commit latency with queue shoots to 300ms
and journal latency (return write ack for client) (journal on Intel
DC-S3710 SSD) shoots on 40 ms

op_w_process_latency showed 250 ms and client read-modify-write
operation readable/applied latency jumped to 1.25 s on one of the OSDs

I rescheduled the scrubbing and deep scrubbing and was watching ceph -w
activity so it is definitely not related.

At the same time node shows 98 % cpu idle no significant changes in
memory utilization, no errors on network with bandwidth utilization
between 20 - 50 Mbit on client and back end networks

OSD node has 12 OSDs (2TB rust) 2 partitioned SSD journal disks, 32 GB
RAM, dial 6 core / 12 thread CPUs

This is perhaps the most relevant part of ceph config

debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug journaler = 0/0
debug osd = 0/0
debug optracker = 0/0
debug objclass = 0/0
debug filestore = 0/0
debug journal = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug heartbeatmap = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0

[osd]
journal_dio = true
journal_aio = true
osd_journal = /var/lib/ceph/osd/$cluster-$id-journal/journal
osd_journal_size = 2048 ; journal size, in megabytes
osd crush update on start = false
osd mount options xfs =
"rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M"
osd_op_threads = 5
osd_disk_threads = 4
osd_pool_default_size = 2
osd_pool_default_min_size = 1
osd_pool_default_pg_num = 512
osd_pool_default_pgp_num = 512
osd_crush_chooseleaf_type = 1
; osd pool_default_crush_rule = 1
; new options 04.12.2015
filestore_op_threads = 4
osd_op_num_threads_per_shard = 1
osd_op_num_shards = 25
filestore_fd_cache_size = 64
filestore_fd_cache_shards = 32
filestore_fiemap = false
; Reduce impact of scrub (needs cfq on osds)
osd_disk_thread_ioprio_class = "idle"
osd_disk_thread_ioprio_priority = 7
osd_deep_scrub_interval = 1211600
osd_scrub_begin_hour = 19
osd_scrub_end_hour = 4
osd_scrub_sleep = 0.1
[client]
rbd_cache = true
rbd_cache_size = 67108864
rbd_cache_max_dirty = 50331648
rbd_cache_target_dirty = 33554432
rbd_cache_max_dirty_age = 2
rbd_cache_writethrough_until_flush = true

OSD logs and system log at that time show nothing interesting.

Any clue of what to look for in order to diagnose the load / latency
spikes would be really appreciated.

Thank you

Kees Meijs

2018-11-17 19:40:42 UTC

Permalink

Hi Alex,

What kind of clients do you use? Is it KVM (QEMU) using NBD driver,
kernel, or...?

Regards,
Kees

Post by Alex Litvak
Hello everyone,
I am trying to troubleshoot cluster exhibiting huge spikes of latency.
I cannot quite catch it because it happens during the light activity
and randomly affects one osd node out of 3 in the pool.
This is a file store.
I see some osds exhibit applied latency of 400 ms, 1 minute load
average shuts to 60. Client commit latency with queue shoots to 300ms
and journal latency (return write ack for client) (journal on Intel
DC-S3710 SSD) shoots on 40 ms
op_w_process_latency showed 250 ms and client read-modify-write
operation readable/applied latency jumped to 1.25 s on one of the OSDs
I rescheduled the scrubbing and deep scrubbing and was watching ceph
-w activity so it is definitely not related.
At the same time node shows 98 % cpu idle no significant changes in
memory utilization, no errors on network with bandwidth utilization
between 20 - 50 Mbit on client and back end networks
OSD node has 12 OSDs (2TB rust) 2 partitioned SSD journal disks, 32 GB
RAM, dial 6 core / 12 thread CPUs
This is perhaps the most relevant part of ceph config
debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug journaler = 0/0
debug osd = 0/0
debug optracker = 0/0
debug objclass = 0/0
debug filestore = 0/0
debug journal = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug heartbeatmap = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0
[osd]
        journal_dio = true
        journal_aio = true
        osd_journal = /var/lib/ceph/osd/$cluster-$id-journal/journal
        osd_journal_size = 2048     ; journal size, in megabytes
    osd crush update on start = false
        osd mount options xfs =
"rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M"
        osd_op_threads = 5
        osd_disk_threads = 4
        osd_pool_default_size = 2
        osd_pool_default_min_size = 1
        osd_pool_default_pg_num = 512
        osd_pool_default_pgp_num = 512
        osd_crush_chooseleaf_type = 1
        ; osd pool_default_crush_rule = 1
    ; new options 04.12.2015
    filestore_op_threads = 4
        osd_op_num_threads_per_shard = 1
        osd_op_num_shards = 25
        filestore_fd_cache_size = 64
        filestore_fd_cache_shards = 32
    filestore_fiemap = false
    ; Reduce impact of scrub (needs cfq on osds)
    osd_disk_thread_ioprio_class = "idle"
    osd_disk_thread_ioprio_priority = 7
    osd_deep_scrub_interval = 1211600
        osd_scrub_begin_hour = 19
        osd_scrub_end_hour = 4
        osd_scrub_sleep = 0.1
[client]
    rbd_cache = true
    rbd_cache_size = 67108864
    rbd_cache_max_dirty = 50331648
    rbd_cache_target_dirty = 33554432
    rbd_cache_max_dirty_age = 2
    rbd_cache_writethrough_until_flush = true
OSD logs and system log at that time show nothing interesting.
Any clue of what to look for in order to diagnose the load / latency
spikes would be really appreciated.
Thank you
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Alex Litvak

2018-11-17 20:06:59 UTC

Permalink

I am using libvirt for block device (openstack, proxmox KVM VMs)
Also I am mounting cephfs inside of VMs and on bare metal hosts. In
this case it would be a kernel based client.

From what I can see based on pool stats cephfs pools have higher
utilization comparing to block pools during the spikes, how ever it is
still small.

Post by Kees Meijs
Hi Alex,
What kind of clients do you use? Is it KVM (QEMU) using NBD driver,
kernel, or...?
Regards,
Kees

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Paul Emmerich

2018-11-17 20:14:44 UTC

Permalink

Are you running FileStore? (The config options you are using looks
like a FileStore config)
Try out BlueStore, we've found that it reduces random latency spikes
due to filesystem weirdness a lot.

Paul
--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

Am Sa., 17. Nov. 2018 um 21:07 Uhr schrieb Alex Litvak

Post by Alex Litvak
I am using libvirt for block device (openstack, proxmox KVM VMs)
Also I am mounting cephfs inside of VMs and on bare metal hosts. In
this case it would be a kernel based client.
From what I can see based on pool stats cephfs pools have higher
utilization comparing to block pools during the spikes, how ever it is
still small.

Post by Kees Meijs
Hi Alex,
What kind of clients do you use? Is it KVM (QEMU) using NBD driver,
kernel, or...?
Regards,
Kees

Post by Alex Litvak
Hello everyone,
I am trying to troubleshoot cluster exhibiting huge spikes of latency.
I cannot quite catch it because it happens during the light activity
and randomly affects one osd node out of 3 in the pool.
This is a file store.
I see some osds exhibit applied latency of 400 ms, 1 minute load
average shuts to 60. Client commit latency with queue shoots to 300ms
and journal latency (return write ack for client) (journal on Intel
DC-S3710 SSD) shoots on 40 ms
op_w_process_latency showed 250 ms and client read-modify-write
operation readable/applied latency jumped to 1.25 s on one of the OSDs
I rescheduled the scrubbing and deep scrubbing and was watching ceph
-w activity so it is definitely not related.
At the same time node shows 98 % cpu idle no significant changes in
memory utilization, no errors on network with bandwidth utilization
between 20 - 50 Mbit on client and back end networks
OSD node has 12 OSDs (2TB rust) 2 partitioned SSD journal disks, 32 GB
RAM, dial 6 core / 12 thread CPUs
This is perhaps the most relevant part of ceph config
debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug journaler = 0/0
debug osd = 0/0
debug optracker = 0/0
debug objclass = 0/0
debug filestore = 0/0
debug journal = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug heartbeatmap = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0
[osd]
journal_dio = true
journal_aio = true
osd_journal = /var/lib/ceph/osd/$cluster-$id-journal/journal
osd_journal_size = 2048 ; journal size, in megabytes
osd crush update on start = false
osd mount options xfs =
"rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M"
osd_op_threads = 5
osd_disk_threads = 4
osd_pool_default_size = 2
osd_pool_default_min_size = 1
osd_pool_default_pg_num = 512
osd_pool_default_pgp_num = 512
osd_crush_chooseleaf_type = 1
; osd pool_default_crush_rule = 1
; new options 04.12.2015
filestore_op_threads = 4
osd_op_num_threads_per_shard = 1
osd_op_num_shards = 25
filestore_fd_cache_size = 64
filestore_fd_cache_shards = 32
filestore_fiemap = false
; Reduce impact of scrub (needs cfq on osds)
osd_disk_thread_ioprio_class = "idle"
osd_disk_thread_ioprio_priority = 7
osd_deep_scrub_interval = 1211600
osd_scrub_begin_hour = 19
osd_scrub_end_hour = 4
osd_scrub_sleep = 0.1
[client]
rbd_cache = true
rbd_cache_size = 67108864
rbd_cache_max_dirty = 50331648
rbd_cache_target_dirty = 33554432
rbd_cache_max_dirty_age = 2
rbd_cache_writethrough_until_flush = true
OSD logs and system log at that time show nothing interesting.
Any clue of what to look for in order to diagnose the load / latency
spikes would be really appreciated.
Thank you
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Alex Litvak

2018-11-17 20:30:08 UTC

Permalink

I am evaluating bluestore on the separate cluster. Unfortunately
upgrading this one is out of the question at the moment for multiple
reasons. That is why I am trying to find a possible root cause.

Post by Paul Emmerich
Are you running FileStore? (The config options you are using looks
like a FileStore config)
Try out BlueStore, we've found that it reduces random latency spikes
due to filesystem weirdness a lot.
Paul

John Petrini

2018-11-17 20:37:39 UTC

Permalink

I'd take a look at cstates if it's only happening during periods of
low activity. If your journals are on SSD you should also check their
health. They may have exceeded their write endurance - high apply
latency is a tell tale sign of this and you'd see high iowait on those
disks.

Alex Litvak

2018-11-17 21:07:43 UTC

Permalink

John,

Thank you for suggestions:

I looked into journal SSDs. It is close to 3 years old showing 5.17% of
wear (352941GB Written to disk with 3.6 PB endurance specs over 5 years)

It could be that smart not telling all but that it what I see.

Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age
Always - 29054
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 4
170 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always
- 0
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always
- 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always
- 0
174 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always
- 3
175 Power_Loss_Cap_Test 0x0033 100 100 010 Pre-fail Always
- 5130 (117 3127)
183 SATA_Downshift_Count 0x0032 100 100 000 Old_age Always
- 0
184 End-to-End_Error 0x0033 100 100 090 Pre-fail Always
- 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always
- 0
190 Temperature_Case 0x0022 074 064 000 Old_age Always
- 26 (Min/Max 23/36)
192 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always
- 3
194 Temperature_Internal 0x0022 100 100 000 Old_age Always
- 26
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always
- 0
199 CRC_Error_Count 0x003e 100 100 000 Old_age Always
- 0
225 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always
- 10518704
226 Workld_Media_Wear_Indic 0x0032 100 100 000 Old_age Always
- 5304
227 Workld_Host_Reads_Perc 0x0032 100 100 000 Old_age Always
- 0
228 Workload_Minutes 0x0032 100 100 000 Old_age Always
- 1743266
232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always
- 0
233 Media_Wearout_Indicator 0x0032 095 095 000 Old_age Always
- 0
234 Thermal_Throttle 0x0032 100 100 000 Old_age Always
- 0/0
241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always
- 10518704
242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always
- 6034

SMART Error Log Version: 1
No Errors Logged

How do you look at cstates?

Post by John Petrini
I'd take a look at cstates if it's only happening during periods of
low activity. If your journals are on SSD you should also check their
health. They may have exceeded their write endurance - high apply
latency is a tell tale sign of this and you'd see high iowait on those
disks.

John Petrini

2018-11-17 21:42:37 UTC

Permalink

You can check if cstates are enabled with cat /proc/acpi/processor/info.
Look for power management: yes/no.

If they are enabled then you can check the current cstate of each core. 0
is the CPU's normal operating range, any other state means the processor is
in a power saving mode. cat /proc/acpi/processor/CPU?/power.

cstates are configured in the bios so a reboot is required to change them.
I know with Dell servers you can trigger the change with omconfig and then
issue a reboot for it to take effect. Otherwise you'll need to disable it
directly in the bios.

As for the SSD's I would just run iostat and check the iowait. If you see
small disk writes causing high iowait then your SSD's are probably at the
end of their life. Ceph journaling is good at destroying SSD's.

Alex Litvak

2018-11-17 22:52:02 UTC

Permalink

Plot thickens:

I checked c-states and apparently I am operating in c1 with all CPUS on. Apparently servers were tuned to use latency-performance

tuned-adm active
Current active profile: latency-performance

turbostat shows
Package Core CPU Avg_MHz %Busy Bzy_MHz TSC_MHz SMI CPU%c1 CPU%c3 CPU%c6 CPU%c7 CoreTmp PkgTmp Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 PkgWatt RAMWatt PKG_% RAM_%
- - - 22 0.84 2600 2400 0 99.16 0.00 0.00 0.00 49 58 0.00 0.00 0.00 0.00 69.51 17.29 0.00 0.00
0 0 0 39 1.52 2600 2400 0 98.48 0.00 0.00 0.00 48 58 0.00 0.00 0.00 0.00 36.30 8.73 0.00 0.00
0 0 12 15 0.56 2600 2400 0 99.44
0 1 2 47 1.81 2600 2400 0 98.19 0.00 0.00 0.00 49
0 1 14 17 0.66 2600 2400 0 99.34
0 2 4 31 1.20 2600 2400 0 98.80 0.00 0.00 0.00 47
0 2 16 18 0.71 2600 2400 0 99.29
0 3 6 31 1.21 2600 2400 0 98.79 0.00 0.00 0.00 49
0 3 18 39 1.50 2600 2400 0 98.50
0 4 8 33 1.27 2600 2400 0 98.73 0.00 0.00 0.00 46
0 4 20 17 0.64 2600 2400 0 99.36
0 5 10 32 1.23 2600 2400 0 98.77 0.00 0.00 0.00 48
0 5 22 20 0.76 2600 2400 0 99.24
1 0 1 25 0.95 2600 2400 0 99.05 0.00 0.00 0.00 44 52 0.00 0.00 0.00 0.00 33.21 8.56 0.00 0.00
1 0 13 9 0.34 2600 2400 0 99.66
1 1 3 9 0.35 2600 2400 0 99.65 0.00 0.00 0.00 42
1 1 15 11 0.42 2600 2400 0 99.58
1 2 5 30 1.17 2600 2400 0 98.83 0.00 0.00 0.00 46
1 2 17 7 0.28 2600 2400 0 99.72
1 3 7 10 0.40 2600 2400 0 99.60 0.00 0.00 0.00 44
1 3 19 10 0.37 2600 2400 0 99.63
1 4 9 9 0.36 2600 2400 0 99.64 0.00 0.00 0.00 45
1 4 21 7 0.27 2600 2400 0 99.73
1 5 11 12 0.45 2600 2400 0 99.55 0.00 0.00 0.00 45
1 5 23 46 1.76 2600 2400 0 98.24

iostat for ssd shows

# iostat -xd -p sdb 1 1000

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.05 26.78 0.20 2299.53 171.42 0.02 0.64 0.11 0.64 0.08 0.20

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 16.00 0.00 392.00 49.00 0.00 0.06 0.00 0.06 0.06 0.10

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 74.00 0.00 880.00 23.78 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 56.00 0.00 240.00 8.57 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 44.00 0.00 676.00 30.73 0.00 0.07 0.00 0.07 0.05 0.20

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 10.00 0.00 92.00 18.40 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 6.00 0.00 84.00 28.00 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 1.00 0.00 20.00 40.00 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 25.00 0.00 212.00 16.96 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 14.00 0.00 100.00 14.29 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 5.00 0.00 112.00 44.80 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 13.00 0.00 508.00 78.15 0.00 0.15 0.00 0.15 0.15 0.20

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 49.00 0.00 820.00 33.47 0.01 0.10 0.00 0.10 0.08 0.40

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 7.00 0.00 52.00 14.86 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 18.00 0.00 180.00 20.00 0.00 0.06 0.00 0.06 0.06 0.10

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 34.00 0.00 476.00 28.00 0.00 0.06 0.00 0.06 0.06 0.20

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 1.00 12.00 4.00 156.00 24.62 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 32.00 0.00 940.00 58.75 0.00 0.03 0.00 0.03 0.03 0.10

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 13.00 0.00 456.00 70.15 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 37.00 0.00 536.00 28.97 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 6.00 0.00 60.00 20.00 0.00 0.17 0.00 0.17 0.17 0.10

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 3.00 0.00 48.00 32.00 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 10.00 0.00 1452.00 290.40 0.00 0.30 0.00 0.30 0.20 0.20

You can check if cstates are enabled with cat /proc/acpi/processor/info. Look for power management: yes/no.||
If they are enabled then you can check the current cstate of each core. 0 is the CPU's normal operating range, any other state means the processor is in a power saving mode. cat
/proc/acpi/processor/CPU?/power.
cstates are configured in the bios so a reboot is required to change them. I know with Dell servers you can trigger the change with omconfig and then issue a reboot for it to take effect. Otherwise
you'll need to disable it directly in the bios.
As for the SSD's I would just run iostat and check the iowait. If you see small disk writes causing high iowait then your SSD's are probably at the end of their life. Ceph journaling is good at
destroying SSD's.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

John Petrini

2018-11-17 23:27:55 UTC

Permalink

Alex Litvak

2018-11-18 04:19:55 UTC

Permalink

The iostat isn't very helpful because there are not many writes. I'd recommend disabling cstates entirely, not sure it's your problem but it's good practice and if your cluster goes as idle as your
iostat suggests it could be the culprit.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Alex Litvak

2018-11-18 04:46:17 UTC

Permalink

Here is another snapshot. I wonder if this write io wait is too big
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 23.00 0.00 336.00 29.22 0.34 14.74 0.00 14.74 2.87 6.60
dm-15 0.00 0.00 0.00 16.00 0.00 200.00 25.00 0.01 0.75 0.00 0.75 0.75 1.20
dm-16 0.00 0.00 0.00 17.00 0.00 276.00 32.47 0.25 14.94 0.00 14.94 3.35 5.70
dm-17 0.00 0.00 0.00 17.00 0.00 252.00 29.65 0.32 18.65 0.00 18.65 4.00 6.80
dm-18 0.00 0.00 0.00 15.00 0.00 152.00 20.27 0.25 16.80 0.00 16.80 4.07 6.10
dm-19 0.00 0.00 0.00 13.00 0.00 152.00 23.38 0.21 15.92 0.00 15.92 4.85 6.30
dm-20 0.00 0.00 0.00 20.00 0.00 248.00 24.80 0.27 13.60 0.00 13.60 3.25 6.50
dm-21 0.00 0.00 0.00 17.00 0.00 188.00 22.12 0.27 16.00 0.00 16.00 3.59 6.10
dm-22 0.00 0.00 0.00 20.00 0.00 156.00 15.60 0.11 5.55 0.00 5.55 2.95 5.90
dm-24 0.00 0.00 0.00 8.00 0.00 56.00 14.00 0.12 14.62 0.00 14.62 4.75 3.80
dm-25 0.00 0.00 0.00 19.00 0.00 200.00 21.05 0.21 10.89 0.00 10.89 2.74 5.20

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 11.00 0.00 136.00 24.73 0.11 9.73 0.00 9.73 1.82 2.00
dm-15 0.00 0.00 0.00 12.00 0.00 136.00 22.67 0.04 3.75 0.00 3.75 1.08 1.30
dm-16 0.00 0.00 0.00 9.00 0.00 104.00 23.11 0.09 10.44 0.00 10.44 2.44 2.20
dm-17 0.00 0.00 0.00 5.00 0.00 160.00 64.00 0.02 4.00 0.00 4.00 4.00 2.00
dm-18 0.00 0.00 0.00 5.00 0.00 52.00 20.80 0.03 5.80 0.00 5.80 3.60 1.80
dm-19 0.00 0.00 0.00 10.00 0.00 104.00 20.80 0.08 7.90 0.00 7.90 2.10 2.10
dm-20 0.00 0.00 0.00 9.00 0.00 132.00 29.33 0.10 11.22 0.00 11.22 2.56 2.30
dm-21 0.00 0.00 0.00 6.00 0.00 68.00 22.67 0.07 12.33 0.00 12.33 3.83 2.30
dm-22 0.00 0.00 0.00 3.00 0.00 20.00 13.33 0.01 3.67 0.00 3.67 3.67 1.10
dm-24 0.00 0.00 0.00 4.00 0.00 24.00 12.00 0.07 18.00 0.00 18.00 5.25 2.10
dm-25 0.00 0.00 0.00 6.00 0.00 64.00 21.33 0.06 10.33 0.00 10.33 3.67 2.20

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 5.00 0.00 140.00 56.00 0.08 15.20 0.00 15.20 5.40 2.70
dm-15 0.00 0.00 0.00 6.00 0.00 236.00 78.67 0.18 30.67 0.00 30.67 6.83 4.10
dm-16 0.00 0.00 0.00 8.00 0.00 84.00 21.00 0.06 7.25 0.00 7.25 1.62 1.30
dm-17 0.00 0.00 0.00 3.00 0.00 84.00 56.00 0.00 0.33 0.00 0.33 0.33 0.10
dm-18 0.00 0.00 0.00 2.00 0.00 20.00 20.00 0.02 12.00 0.00 12.00 12.00 2.40
dm-19 0.00 0.00 0.00 12.00 0.00 80.00 13.33 0.05 4.00 0.00 4.00 2.33 2.80
dm-20 0.00 0.00 0.00 16.00 0.00 256.00 32.00 0.00 0.06 0.00 0.06 0.06 0.10
dm-21 0.00 0.00 0.00 8.00 0.00 500.00 125.00 0.00 0.12 0.00 0.12 0.12 0.10
dm-22 0.00 0.00 0.00 1.00 0.00 8.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 2.00 0.00 32.00 32.00 0.08 40.00 0.00 40.00 20.50 4.10

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 10.00 0.00 108.00 21.60 0.11 10.80 0.00 10.80 1.90 1.90
dm-15 0.00 0.00 0.00 5.00 0.00 60.00 24.00 0.03 6.20 0.00 6.20 3.40 1.70
dm-16 0.00 0.00 0.00 6.00 0.00 68.00 22.67 0.00 0.17 0.00 0.17 0.17 0.10
dm-17 0.00 0.00 0.00 7.00 0.00 100.00 28.57 0.09 12.29 0.00 12.29 2.86 2.00
dm-18 0.00 0.00 0.00 5.00 0.00 44.00 17.60 0.07 14.00 0.00 14.00 4.00 2.00
dm-19 0.00 0.00 0.00 2.00 0.00 24.00 24.00 0.03 14.00 0.00 14.00 8.50 1.70
dm-20 0.00 0.00 0.00 7.00 0.00 96.00 27.43 0.07 9.57 0.00 9.57 2.00 1.40
dm-21 0.00 0.00 0.00 7.00 0.00 84.00 24.00 0.12 16.57 0.00 16.57 4.86 3.40
dm-22 0.00 0.00 0.00 5.00 0.00 60.00 24.00 0.05 10.20 0.00 10.20 3.40 1.70
dm-24 0.00 0.00 0.00 2.00 0.00 16.00 16.00 0.03 17.00 0.00 17.00 9.00 1.80
dm-25 0.00 0.00 0.00 4.00 0.00 76.00 38.00 0.05 11.75 0.00 11.75 4.75 1.90

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 9.00 0.00 104.00 23.11 0.14 15.33 0.00 15.33 4.33 3.90
dm-15 0.00 0.00 0.00 11.00 0.00 128.00 23.27 0.01 0.82 0.00 0.82 0.82 0.90
dm-16 0.00 0.00 0.00 10.00 0.00 120.00 24.00 0.09 9.30 0.00 9.30 3.30 3.30
dm-17 0.00 0.00 0.00 5.00 0.00 72.00 28.80 0.04 8.20 0.00 8.20 8.20 4.10
dm-18 0.00 0.00 0.00 7.00 0.00 68.00 19.43 0.08 11.57 0.00 11.57 6.29 4.40
dm-19 0.00 0.00 0.00 7.00 0.00 76.00 21.71 0.10 13.86 0.00 13.86 6.29 4.40
dm-20 0.00 0.00 0.00 15.00 0.00 176.00 23.47 0.12 7.87 0.00 7.87 1.60 2.40
dm-21 0.00 0.00 0.00 9.00 0.00 112.00 24.89 0.17 18.56 0.00 18.56 6.56 5.90
dm-22 0.00 0.00 0.00 4.00 0.00 56.00 28.00 0.04 10.75 0.00 10.75 7.50 3.00
dm-24 0.00 0.00 0.00 5.00 0.00 532.00 212.80 0.02 4.00 0.00 4.00 3.60 1.80
dm-25 0.00 0.00 0.00 6.00 0.00 60.00 20.00 0.11 19.00 0.00 19.00 7.17 4.30

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 9.00 0.00 112.00 24.89 0.30 33.22 0.00 33.22 4.33 3.90
dm-15 0.00 0.00 0.00 8.00 0.00 96.00 24.00 0.08 9.88 0.00 9.88 2.62 2.10
dm-16 0.00 0.00 0.00 5.00 0.00 92.00 36.80 0.01 1.80 0.00 1.80 1.80 0.90
dm-17 0.00 0.00 0.00 4.00 0.00 104.00 52.00 0.04 10.00 0.00 10.00 10.00 4.00
dm-18 0.00 0.00 0.00 9.00 0.00 92.00 20.44 0.11 12.00 0.00 12.00 4.44 4.00
dm-19 0.00 0.00 0.00 7.00 0.00 64.00 18.29 0.10 14.43 0.00 14.43 4.00 2.80
dm-20 0.00 0.00 0.00 7.00 0.00 84.00 24.00 0.10 14.00 0.00 14.00 3.71 2.60
dm-21 0.00 0.00 0.00 2.00 0.00 16.00 16.00 0.08 39.50 0.00 39.50 20.50 4.10
dm-22 0.00 0.00 0.00 5.00 0.00 36.00 14.40 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 5.00 0.00 52.00 20.80 0.09 17.20 0.00 17.20 5.40 2.70
dm-25 0.00 0.00 0.00 6.00 0.00 48.00 16.00 0.10 16.00 0.00 16.00 5.50 3.30

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 5.00 0.00 48.00 19.20 0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 5.00 0.00 44.00 17.60 0.01 3.00 0.00 3.00 3.00 1.50
dm-16 0.00 0.00 0.00 3.00 0.00 32.00 21.33 0.00 0.00 0.00 0.00 0.00 0.00
dm-17 0.00 0.00 0.00 4.00 0.00 40.00 20.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 3.00 0.00 32.00 21.33 0.04 13.00 0.00 13.00 6.67 2.00
dm-19 0.00 0.00 0.00 3.00 0.00 28.00 18.67 0.00 0.00 0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 6.00 0.00 64.00 21.33 0.11 17.67 0.00 17.67 3.17 1.90
dm-21 0.00 0.00 0.00 1.00 0.00 12.00 24.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 1.00 0.00 12.00 24.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 3.00 0.00 20.00 13.33 0.00 0.33 0.00 0.33 0.33 0.10
dm-25 0.00 0.00 0.00 6.00 0.00 52.00 17.33 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 18.00 0.00 236.00 26.22 0.42 23.28 0.00 23.28 3.72 6.70
dm-15 0.00 0.00 0.00 15.00 0.00 240.00 32.00 0.12 8.27 0.00 8.27 2.00 3.00
dm-16 0.00 0.00 0.00 10.00 0.00 120.00 24.00 0.23 22.80 0.00 22.80 6.50 6.50
dm-17 0.00 0.00 0.00 6.00 0.00 68.00 22.67 0.06 9.83 0.00 9.83 5.00 3.00
dm-18 0.00 0.00 0.00 12.00 0.00 152.00 25.33 0.10 8.17 0.00 8.17 3.17 3.80
dm-19 0.00 0.00 0.00 12.00 0.00 136.00 22.67 0.14 11.58 0.00 11.58 3.42 4.10
dm-20 0.00 0.00 0.00 19.00 0.00 296.00 31.16 0.20 10.63 0.00 10.63 1.84 3.50
dm-21 0.00 0.00 0.00 14.00 0.00 132.00 18.86 0.17 12.14 0.00 12.14 3.71 5.20
dm-22 0.00 0.00 0.00 10.00 0.00 84.00 16.80 0.12 12.00 0.00 12.00 6.60 6.60
dm-24 0.00 0.00 0.00 5.00 0.00 40.00 16.00 0.10 19.40 0.00 19.40 9.80 4.90
dm-25 0.00 0.00 0.00 9.00 0.00 152.00 33.78 0.08 9.33 0.00 9.33 3.44 3.10

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 8.00 0.00 112.00 28.00 0.08 9.75 0.00 9.75 3.88 3.10
dm-15 0.00 0.00 0.00 14.00 0.00 156.00 22.29 0.13 9.43 0.00 9.43 2.71 3.80
dm-16 0.00 0.00 0.00 12.00 0.00 132.00 22.00 0.11 8.83 0.00 8.83 2.17 2.60
dm-17 0.00 0.00 0.00 8.00 0.00 116.00 29.00 0.08 10.12 0.00 10.12 4.25 3.40
dm-18 0.00 0.00 0.00 8.00 0.00 100.00 25.00 0.08 9.62 0.00 9.62 5.00 4.00
dm-19 0.00 0.00 0.00 20.00 0.00 212.00 21.20 0.25 12.75 0.00 12.75 2.60 5.20
dm-20 0.00 0.00 0.00 16.00 0.00 168.00 21.00 0.04 2.69 0.00 2.69 0.81 1.30
dm-21 0.00 0.00 0.00 5.00 0.00 48.00 19.20 0.05 10.80 0.00 10.80 7.80 3.90
dm-22 0.00 0.00 0.00 5.00 0.00 52.00 20.80 0.03 6.80 0.00 6.80 3.40 1.70
dm-24 0.00 0.00 0.00 8.00 0.00 60.00 15.00 0.09 11.75 0.00 11.75 2.25 1.80
dm-25 0.00 0.00 0.00 12.00 0.00 160.00 26.67 0.06 5.08 0.00 5.08 1.25 1.50

I stand corrected, I looked at the device iostat, but it was partitioned. Here is a more correct picture of what is going on now.
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await svctm %util
dm-14             0.00     0.00    0.00   19.00     0.00 4116.00   433.26     0.01    0.68    0.00    0.68   0.05   0.10
dm-15             0.00     0.00    0.00   35.00     0.00 8224.00   469.94     0.03    0.86    0.00    0.86   0.06   0.20
dm-16             0.00     0.00    0.00   53.00     0.00 12428.00   468.98     0.11    2.04    0.00    2.04   0.17   0.90
dm-17             0.00     0.00    0.00   43.00     0.00 8344.00   388.09     0.09    2.14    0.00    2.14   0.42   1.80
dm-18             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-19             0.00     0.00    0.00   75.00     0.00 16824.00   448.64     0.08    1.11    0.00    1.11   0.08   0.60
dm-20             0.00     0.00    0.00   70.00     0.00 16452.00   470.06     0.06    0.90    0.00    0.90   0.09   0.60
dm-21             0.00     0.00    0.00   18.00     0.00 4112.00   456.89     0.02    1.00    0.00    1.00   0.11   0.20
dm-22             0.00     0.00    0.00   53.00     0.00 12324.00   465.06     0.06    0.70    0.00    0.70   0.08   0.40
dm-24             0.00     0.00    0.00   18.00     0.00 4272.00   474.67     0.02    1.06    0.00    1.06   0.17   0.30
dm-25             0.00     0.00    0.00   74.00     0.00 16916.00   457.19     0.09    1.26    0.00    1.26   0.18   1.30
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await svctm %util
dm-14             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-15             0.00     0.00    0.00   17.00     0.00 4108.00   483.29     0.02    1.00    0.00    1.00   0.06   0.10
dm-16             0.00     0.00    0.00   34.00     0.00 8208.00   482.82     0.03    1.00    0.00    1.00   0.06   0.20
dm-17             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-18             0.00     0.00    0.00   36.00     0.00 8220.00   456.67     0.05    1.33    0.00    1.33   0.08   0.30
dm-19             0.00     0.00    0.00    1.00     0.00     8.00    16.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-20             0.00     0.00    0.00   36.00     0.00 8288.00   460.44     0.05    1.42    0.00    1.42   0.08   0.30
dm-21             0.00     0.00    0.00   34.00     0.00 8208.00   482.82     0.03    1.00    0.00    1.00   0.06   0.20
dm-22             0.00     0.00    0.00   18.00     0.00 4128.00   458.67     0.04    3.22    0.00    3.22   0.17   0.30
dm-24             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-25             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await svctm %util
dm-14             0.00     0.00    0.00   20.00     0.00 4032.00   403.20     0.00    0.00    0.00    0.00   0.00   0.00
dm-15             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-16             0.00     0.00    0.00    1.00     0.00    20.00    40.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-17             0.00     0.00    0.00    4.00     0.00    28.00    14.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-18             0.00     0.00    0.00    3.00     0.00    36.00    24.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-19             0.00     0.00    0.00    2.00     0.00    20.00    20.00     0.01    2.50    0.00    2.50   2.50   0.50
dm-20             0.00     0.00    0.00    6.00     0.00    96.00    32.00     0.02    3.33    0.00    3.33   2.00   1.20
dm-21             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-22             0.00     0.00    0.00    2.00     0.00    32.00    32.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-24             0.00     0.00    0.00   22.00     0.00 4184.00   380.36     0.10    4.59    0.00    4.59   0.95   2.10
dm-25             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await svctm %util
dm-14             0.00     0.00    0.00    8.00     0.00 1928.00   482.00     0.01    1.00    0.00    1.00   0.12   0.10
dm-15             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-16             0.00     0.00    0.00    3.00     0.00   312.00   208.00     0.00    0.33    0.00    0.33   0.33   0.10
dm-17             0.00     0.00    0.00   18.00     0.00 4264.00   473.78     0.03    1.67    0.00    1.67   0.11   0.20
dm-18             0.00     0.00    0.00   17.00     0.00 4104.00   482.82     0.03    1.82    0.00    1.82   0.12   0.20
dm-19             0.00     0.00    0.00   18.00     0.00 4112.00   456.89     0.02    1.06    0.00    1.06   0.11   0.20
dm-20             0.00     0.00    0.00   32.00     0.00 4308.00   269.25     0.03    0.81    0.00    0.81   0.34   1.10
dm-21             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-22             0.00     0.00    0.00    8.00     0.00   540.00   135.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-24             0.00     0.00    0.00   35.00     0.00 8228.00   470.17     0.03    0.97    0.00    0.97   0.06   0.20
dm-25             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Serkan Çoban

2018-11-18 05:38:32 UTC

Permalink

10ms w_await for SSD is too much. How that SSD is connected to the system? Any raid card installed on this system? What is the raid mode?

On Sun, Nov 18, 2018 at 8:25 AM Alex Litvak

Here is another snapshot. I wonder if this write io wait is too big
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 23.00 0.00 336.00 29.22 0.34 14.74 0.00 14.74 2.87 6.60
dm-15 0.00 0.00 0.00 16.00 0.00 200.00 25.00 0.01 0.75 0.00 0.75 0.75 1.20
dm-16 0.00 0.00 0.00 17.00 0.00 276.00 32.47 0.25 14.94 0.00 14.94 3.35 5.70
dm-17 0.00 0.00 0.00 17.00 0.00 252.00 29.65 0.32 18.65 0.00 18.65 4.00 6.80
dm-18 0.00 0.00 0.00 15.00 0.00 152.00 20.27 0.25 16.80 0.00 16.80 4.07 6.10
dm-19 0.00 0.00 0.00 13.00 0.00 152.00 23.38 0.21 15.92 0.00 15.92 4.85 6.30
dm-20 0.00 0.00 0.00 20.00 0.00 248.00 24.80 0.27 13.60 0.00 13.60 3.25 6.50
dm-21 0.00 0.00 0.00 17.00 0.00 188.00 22.12 0.27 16.00 0.00 16.00 3.59 6.10
dm-22 0.00 0.00 0.00 20.00 0.00 156.00 15.60 0.11 5.55 0.00 5.55 2.95 5.90
dm-24 0.00 0.00 0.00 8.00 0.00 56.00 14.00 0.12 14.62 0.00 14.62 4.75 3.80
dm-25 0.00 0.00 0.00 19.00 0.00 200.00 21.05 0.21 10.89 0.00 10.89 2.74 5.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 11.00 0.00 136.00 24.73 0.11 9.73 0.00 9.73 1.82 2.00
dm-15 0.00 0.00 0.00 12.00 0.00 136.00 22.67 0.04 3.75 0.00 3.75 1.08 1.30
dm-16 0.00 0.00 0.00 9.00 0.00 104.00 23.11 0.09 10.44 0.00 10.44 2.44 2.20
dm-17 0.00 0.00 0.00 5.00 0.00 160.00 64.00 0.02 4.00 0.00 4.00 4.00 2.00
dm-18 0.00 0.00 0.00 5.00 0.00 52.00 20.80 0.03 5.80 0.00 5.80 3.60 1.80
dm-19 0.00 0.00 0.00 10.00 0.00 104.00 20.80 0.08 7.90 0.00 7.90 2.10 2.10
dm-20 0.00 0.00 0.00 9.00 0.00 132.00 29.33 0.10 11.22 0.00 11.22 2.56 2.30
dm-21 0.00 0.00 0.00 6.00 0.00 68.00 22.67 0.07 12.33 0.00 12.33 3.83 2.30
dm-22 0.00 0.00 0.00 3.00 0.00 20.00 13.33 0.01 3.67 0.00 3.67 3.67 1.10
dm-24 0.00 0.00 0.00 4.00 0.00 24.00 12.00 0.07 18.00 0.00 18.00 5.25 2.10
dm-25 0.00 0.00 0.00 6.00 0.00 64.00 21.33 0.06 10.33 0.00 10.33 3.67 2.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 5.00 0.00 140.00 56.00 0.08 15.20 0.00 15.20 5.40 2.70
dm-15 0.00 0.00 0.00 6.00 0.00 236.00 78.67 0.18 30.67 0.00 30.67 6.83 4.10
dm-16 0.00 0.00 0.00 8.00 0.00 84.00 21.00 0.06 7.25 0.00 7.25 1.62 1.30
dm-17 0.00 0.00 0.00 3.00 0.00 84.00 56.00 0.00 0.33 0.00 0.33 0.33 0.10
dm-18 0.00 0.00 0.00 2.00 0.00 20.00 20.00 0.02 12.00 0.00 12.00 12.00 2.40
dm-19 0.00 0.00 0.00 12.00 0.00 80.00 13.33 0.05 4.00 0.00 4.00 2.33 2.80
dm-20 0.00 0.00 0.00 16.00 0.00 256.00 32.00 0.00 0.06 0.00 0.06 0.06 0.10
dm-21 0.00 0.00 0.00 8.00 0.00 500.00 125.00 0.00 0.12 0.00 0.12 0.12 0.10
dm-22 0.00 0.00 0.00 1.00 0.00 8.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 2.00 0.00 32.00 32.00 0.08 40.00 0.00 40.00 20.50 4.10
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 10.00 0.00 108.00 21.60 0.11 10.80 0.00 10.80 1.90 1.90
dm-15 0.00 0.00 0.00 5.00 0.00 60.00 24.00 0.03 6.20 0.00 6.20 3.40 1.70
dm-16 0.00 0.00 0.00 6.00 0.00 68.00 22.67 0.00 0.17 0.00 0.17 0.17 0.10
dm-17 0.00 0.00 0.00 7.00 0.00 100.00 28.57 0.09 12.29 0.00 12.29 2.86 2.00
dm-18 0.00 0.00 0.00 5.00 0.00 44.00 17.60 0.07 14.00 0.00 14.00 4.00 2.00
dm-19 0.00 0.00 0.00 2.00 0.00 24.00 24.00 0.03 14.00 0.00 14.00 8.50 1.70
dm-20 0.00 0.00 0.00 7.00 0.00 96.00 27.43 0.07 9.57 0.00 9.57 2.00 1.40
dm-21 0.00 0.00 0.00 7.00 0.00 84.00 24.00 0.12 16.57 0.00 16.57 4.86 3.40
dm-22 0.00 0.00 0.00 5.00 0.00 60.00 24.00 0.05 10.20 0.00 10.20 3.40 1.70
dm-24 0.00 0.00 0.00 2.00 0.00 16.00 16.00 0.03 17.00 0.00 17.00 9.00 1.80
dm-25 0.00 0.00 0.00 4.00 0.00 76.00 38.00 0.05 11.75 0.00 11.75 4.75 1.90
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 9.00 0.00 104.00 23.11 0.14 15.33 0.00 15.33 4.33 3.90
dm-15 0.00 0.00 0.00 11.00 0.00 128.00 23.27 0.01 0.82 0.00 0.82 0.82 0.90
dm-16 0.00 0.00 0.00 10.00 0.00 120.00 24.00 0.09 9.30 0.00 9.30 3.30 3.30
dm-17 0.00 0.00 0.00 5.00 0.00 72.00 28.80 0.04 8.20 0.00 8.20 8.20 4.10
dm-18 0.00 0.00 0.00 7.00 0.00 68.00 19.43 0.08 11.57 0.00 11.57 6.29 4.40
dm-19 0.00 0.00 0.00 7.00 0.00 76.00 21.71 0.10 13.86 0.00 13.86 6.29 4.40
dm-20 0.00 0.00 0.00 15.00 0.00 176.00 23.47 0.12 7.87 0.00 7.87 1.60 2.40
dm-21 0.00 0.00 0.00 9.00 0.00 112.00 24.89 0.17 18.56 0.00 18.56 6.56 5.90
dm-22 0.00 0.00 0.00 4.00 0.00 56.00 28.00 0.04 10.75 0.00 10.75 7.50 3.00
dm-24 0.00 0.00 0.00 5.00 0.00 532.00 212.80 0.02 4.00 0.00 4.00 3.60 1.80
dm-25 0.00 0.00 0.00 6.00 0.00 60.00 20.00 0.11 19.00 0.00 19.00 7.17 4.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 9.00 0.00 112.00 24.89 0.30 33.22 0.00 33.22 4.33 3.90
dm-15 0.00 0.00 0.00 8.00 0.00 96.00 24.00 0.08 9.88 0.00 9.88 2.62 2.10
dm-16 0.00 0.00 0.00 5.00 0.00 92.00 36.80 0.01 1.80 0.00 1.80 1.80 0.90
dm-17 0.00 0.00 0.00 4.00 0.00 104.00 52.00 0.04 10.00 0.00 10.00 10.00 4.00
dm-18 0.00 0.00 0.00 9.00 0.00 92.00 20.44 0.11 12.00 0.00 12.00 4.44 4.00
dm-19 0.00 0.00 0.00 7.00 0.00 64.00 18.29 0.10 14.43 0.00 14.43 4.00 2.80
dm-20 0.00 0.00 0.00 7.00 0.00 84.00 24.00 0.10 14.00 0.00 14.00 3.71 2.60
dm-21 0.00 0.00 0.00 2.00 0.00 16.00 16.00 0.08 39.50 0.00 39.50 20.50 4.10
dm-22 0.00 0.00 0.00 5.00 0.00 36.00 14.40 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 5.00 0.00 52.00 20.80 0.09 17.20 0.00 17.20 5.40 2.70
dm-25 0.00 0.00 0.00 6.00 0.00 48.00 16.00 0.10 16.00 0.00 16.00 5.50 3.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 5.00 0.00 48.00 19.20 0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 5.00 0.00 44.00 17.60 0.01 3.00 0.00 3.00 3.00 1.50
dm-16 0.00 0.00 0.00 3.00 0.00 32.00 21.33 0.00 0.00 0.00 0.00 0.00 0.00
dm-17 0.00 0.00 0.00 4.00 0.00 40.00 20.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 3.00 0.00 32.00 21.33 0.04 13.00 0.00 13.00 6.67 2.00
dm-19 0.00 0.00 0.00 3.00 0.00 28.00 18.67 0.00 0.00 0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 6.00 0.00 64.00 21.33 0.11 17.67 0.00 17.67 3.17 1.90
dm-21 0.00 0.00 0.00 1.00 0.00 12.00 24.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 1.00 0.00 12.00 24.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 3.00 0.00 20.00 13.33 0.00 0.33 0.00 0.33 0.33 0.10
dm-25 0.00 0.00 0.00 6.00 0.00 52.00 17.33 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 18.00 0.00 236.00 26.22 0.42 23.28 0.00 23.28 3.72 6.70
dm-15 0.00 0.00 0.00 15.00 0.00 240.00 32.00 0.12 8.27 0.00 8.27 2.00 3.00
dm-16 0.00 0.00 0.00 10.00 0.00 120.00 24.00 0.23 22.80 0.00 22.80 6.50 6.50
dm-17 0.00 0.00 0.00 6.00 0.00 68.00 22.67 0.06 9.83 0.00 9.83 5.00 3.00
dm-18 0.00 0.00 0.00 12.00 0.00 152.00 25.33 0.10 8.17 0.00 8.17 3.17 3.80
dm-19 0.00 0.00 0.00 12.00 0.00 136.00 22.67 0.14 11.58 0.00 11.58 3.42 4.10
dm-20 0.00 0.00 0.00 19.00 0.00 296.00 31.16 0.20 10.63 0.00 10.63 1.84 3.50
dm-21 0.00 0.00 0.00 14.00 0.00 132.00 18.86 0.17 12.14 0.00 12.14 3.71 5.20
dm-22 0.00 0.00 0.00 10.00 0.00 84.00 16.80 0.12 12.00 0.00 12.00 6.60 6.60
dm-24 0.00 0.00 0.00 5.00 0.00 40.00 16.00 0.10 19.40 0.00 19.40 9.80 4.90
dm-25 0.00 0.00 0.00 9.00 0.00 152.00 33.78 0.08 9.33 0.00 9.33 3.44 3.10
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 8.00 0.00 112.00 28.00 0.08 9.75 0.00 9.75 3.88 3.10
dm-15 0.00 0.00 0.00 14.00 0.00 156.00 22.29 0.13 9.43 0.00 9.43 2.71 3.80
dm-16 0.00 0.00 0.00 12.00 0.00 132.00 22.00 0.11 8.83 0.00 8.83 2.17 2.60
dm-17 0.00 0.00 0.00 8.00 0.00 116.00 29.00 0.08 10.12 0.00 10.12 4.25 3.40
dm-18 0.00 0.00 0.00 8.00 0.00 100.00 25.00 0.08 9.62 0.00 9.62 5.00 4.00
dm-19 0.00 0.00 0.00 20.00 0.00 212.00 21.20 0.25 12.75 0.00 12.75 2.60 5.20
dm-20 0.00 0.00 0.00 16.00 0.00 168.00 21.00 0.04 2.69 0.00 2.69 0.81 1.30
dm-21 0.00 0.00 0.00 5.00 0.00 48.00 19.20 0.05 10.80 0.00 10.80 7.80 3.90
dm-22 0.00 0.00 0.00 5.00 0.00 52.00 20.80 0.03 6.80 0.00 6.80 3.40 1.70
dm-24 0.00 0.00 0.00 8.00 0.00 60.00 15.00 0.09 11.75 0.00 11.75 2.25 1.80
dm-25 0.00 0.00 0.00 12.00 0.00 160.00 26.67 0.06 5.08 0.00 5.08 1.25 1.50

Post by Alex Litvak
I stand corrected, I looked at the device iostat, but it was partitioned. Here is a more correct picture of what is going on now.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 19.00 0.00 4116.00 433.26 0.01 0.68 0.00 0.68 0.05 0.10
dm-15 0.00 0.00 0.00 35.00 0.00 8224.00 469.94 0.03 0.86 0.00 0.86 0.06 0.20
dm-16 0.00 0.00 0.00 53.00 0.00 12428.00 468.98 0.11 2.04 0.00 2.04 0.17 0.90
dm-17 0.00 0.00 0.00 43.00 0.00 8344.00 388.09 0.09 2.14 0.00 2.14 0.42 1.80
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 75.00 0.00 16824.00 448.64 0.08 1.11 0.00 1.11 0.08 0.60
dm-20 0.00 0.00 0.00 70.00 0.00 16452.00 470.06 0.06 0.90 0.00 0.90 0.09 0.60
dm-21 0.00 0.00 0.00 18.00 0.00 4112.00 456.89 0.02 1.00 0.00 1.00 0.11 0.20
dm-22 0.00 0.00 0.00 53.00 0.00 12324.00 465.06 0.06 0.70 0.00 0.70 0.08 0.40
dm-24 0.00 0.00 0.00 18.00 0.00 4272.00 474.67 0.02 1.06 0.00 1.06 0.17 0.30
dm-25 0.00 0.00 0.00 74.00 0.00 16916.00 457.19 0.09 1.26 0.00 1.26 0.18 1.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 17.00 0.00 4108.00 483.29 0.02 1.00 0.00 1.00 0.06 0.10
dm-16 0.00 0.00 0.00 34.00 0.00 8208.00 482.82 0.03 1.00 0.00 1.00 0.06 0.20
dm-17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 36.00 0.00 8220.00 456.67 0.05 1.33 0.00 1.33 0.08 0.30
dm-19 0.00 0.00 0.00 1.00 0.00 8.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 36.00 0.00 8288.00 460.44 0.05 1.42 0.00 1.42 0.08 0.30
dm-21 0.00 0.00 0.00 34.00 0.00 8208.00 482.82 0.03 1.00 0.00 1.00 0.06 0.20
dm-22 0.00 0.00 0.00 18.00 0.00 4128.00 458.67 0.04 3.22 0.00 3.22 0.17 0.30
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 20.00 0.00 4032.00 403.20 0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-16 0.00 0.00 0.00 1.00 0.00 20.00 40.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-17 0.00 0.00 0.00 4.00 0.00 28.00 14.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 3.00 0.00 36.00 24.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 2.00 0.00 20.00 20.00 0.01 2.50 0.00 2.50 2.50 0.50
dm-20 0.00 0.00 0.00 6.00 0.00 96.00 32.00 0.02 3.33 0.00 3.33 2.00 1.20
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 2.00 0.00 32.00 32.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 22.00 0.00 4184.00 380.36 0.10 4.59 0.00 4.59 0.95 2.10
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
dm-14 0.00 0.00 0.00 8.00 0.00 1928.00 482.00 0.01 1.00 0.00 1.00 0.12 0.10
dm-15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-16 0.00 0.00 0.00 3.00 0.00 312.00 208.00 0.00 0.33 0.00 0.33 0.33 0.10
dm-17 0.00 0.00 0.00 18.00 0.00 4264.00 473.78 0.03 1.67 0.00 1.67 0.11 0.20
dm-18 0.00 0.00 0.00 17.00 0.00 4104.00 482.82 0.03 1.82 0.00 1.82 0.12 0.20
dm-19 0.00 0.00 0.00 18.00 0.00 4112.00 456.89 0.02 1.06 0.00 1.06 0.11 0.20
dm-20 0.00 0.00 0.00 32.00 0.00 4308.00 269.25 0.03 0.81 0.00 0.81 0.34 1.10
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 8.00 0.00 540.00 135.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 35.00 0.00 8228.00 470.17 0.03 0.97 0.00 0.97 0.06 0.20
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Alex Litvak

2018-11-18 06:04:59 UTC

Permalink

Raid card for journal disks is Perc H730 (Megaraid), RAID 1, battery back cache is on

Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU

I have 2 other nodes with older Perc H710 and similar SSDs with slightly higher wear (6.3% vs 5.18%) but from observation they hardly hit 1.5 ms on rear occasion
Cache, RAID, and battery situation is the same.