2016-02-06 10:31:51 UTC
I am experiencing totally weird filesystem corruptions with the
* Ceph infernalis on Debian8
* 10 OSDs (5 hosts) with spinning disks
* 4 OSDs (1 host, with SSDs)
The SSDs are new in my setup and I am trying to setup a Cache tier.
Now, with the spinning disks Ceph is running since about a year without
any major issues. Replacing disks and all that went fine.
Ceph is used by rbd+libvirt+kvm with
rbd_cache = true
rbd_cache_writethrough_until_flush = true
rbd_cache_size = 128M
rbd_cache_max_dirty = 96M
Also, in libvirt, I have
So far so good.
Now, I've added the SSD-Cache tier to the picture with "cache-mode
The SSD-Machine also has "deadline" scheduler enabled.
Suddenly VMs start to corrupt their filesystems (all ext4) with "Journal
Trying to reboot the machines ends in "No bootable drive"
Using parted and testdisk on the image mapped via rbd reveals that the
partition table is gone.
testdisk finds the proper ones, e2fsck repairs the filesystem beyond
This does not happen to all machines, It happens to those that actually
do some or most fo the IO
elasticsearch, MariaDB+Galera, postgres, backup, GIT
So I thought, yesterday one of my ldap-servers died, and that one is not
Could it be that rbd caching + qemu writeback cache + ceph cach tier
writeback are not playing well together?
I've read through some older mails on the list, where people had similar
problems and suspected somehting like that.
What are the proper/right settings for rdb/qemu/libvirt?
libvirt: cachemode=none (writeback?)
rdb: cache_mode = none
SSD-tier: cachemode: writeback
Thanks for any help,