[ceph-users] Kernel version for Debian 9 CephFS/RBD clients

[Nicolas Huillard]

Hi all,

I'm using Luminous 12.2.4 on all servers, with Debian stock kernel.

I use the kernel cephfs/rbd on the client side, and have a choice of :
* stock Debian 9 kernel 4.9 : LTS, Spectre/Meltdown mitigations in
place, field-tested, probably old libceph inside.
* backports kernel 4.14 : probably better Luminous support, no
Spectre/Meltdown mitigations yet, much less tested (I may have
experienced a kernel-related PPPoE problem lately), not long-term.

Which client kernel would you suggest re. Ceph ?
Does the cephfs/rbd clients benefit from a really newer kernel ?
I expect that the Cpeh server-side kernel don't really matter.

TIA,

[c***@jack.fr.eu.org]

The stock kernel from Debian is perfect
Spectre / meltdown mitigations are worthless for a Ceph point of view,
and should be disabled (again, strictly from a Ceph point of view)

If you need the luminous features, using the userspace implementations
is required (librbd via rbd-nbd or qemu, libcephfs via fuse etc)

Post by Nicolas Huillard
Hi all,
I'm using Luminous 12.2.4 on all servers, with Debian stock kernel.
* stock Debian 9 kernel 4.9 : LTS, Spectre/Meltdown mitigations in
place, field-tested, probably old libceph inside.
* backports kernel 4.14 : probably better Luminous support, no
Spectre/Meltdown mitigations yet, much less tested (I may have
experienced a kernel-related PPPoE problem lately), not long-term.
Which client kernel would you suggest re. Ceph ?
Does the cephfs/rbd clients benefit from a really newer kernel ?
I expect that the Cpeh server-side kernel don't really matter.
TIA,

[Ilya Dryomov]

Post by c***@jack.fr.eu.org
The stock kernel from Debian is perfect
Spectre / meltdown mitigations are worthless for a Ceph point of view,
and should be disabled (again, strictly from a Ceph point of view)
If you need the luminous features, using the userspace implementations
is required (librbd via rbd-nbd or qemu, libcephfs via fuse etc)

luminous cluster-wide feature bits are supported since kernel 4.13.

Thanks,

Ilya

[c***@jack.fr.eu.org]

Post by Ilya Dryomov
luminous cluster-wide feature bits are supported since kernel 4.13.

?

# uname -a
Linux abweb1 4.14.0-0.bpo.3-amd64 #1 SMP Debian 4.14.13-1~bpo9+1
(2018-01-14) x86_64 GNU/Linux
# rbd info truc
rbd image 'truc':
size 20480 MB in 5120 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.9eca966b8b4567
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
# rbd map truc
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by
the kernel with "rbd feature disable pool/truc object-map fast-diff
deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
# dmesg | tail -1
[1108045.667333] rbd: image truc: image uses unsupported features: 0x38

[Ilya Dryomov]

Post by c***@jack.fr.eu.org

Post by Ilya Dryomov
luminous cluster-wide feature bits are supported since kernel 4.13.

?
# uname -a
Linux abweb1 4.14.0-0.bpo.3-amd64 #1 SMP Debian 4.14.13-1~bpo9+1
(2018-01-14) x86_64 GNU/Linux
# rbd info truc
size 20480 MB in 5120 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.9eca966b8b4567
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
# rbd map truc
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by
the kernel with "rbd feature disable pool/truc object-map fast-diff
deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
# dmesg | tail -1
[1108045.667333] rbd: image truc: image uses unsupported features: 0x38

Those are rbd image features. Your email also mentioned "libcephfs via
fuse", so I assumed you had meant cluster-wide feature bits.

Thanks,

Ilya

[c***@jack.fr.eu.org]

Ok ^^

For Cephfs, as far as I know, quota support is not supported in kernel space
This is not specific to luminous, tho

Post by Ilya Dryomov

Post by c***@jack.fr.eu.org

Post by Ilya Dryomov
luminous cluster-wide feature bits are supported since kernel 4.13.

?
# uname -a
Linux abweb1 4.14.0-0.bpo.3-amd64 #1 SMP Debian 4.14.13-1~bpo9+1
(2018-01-14) x86_64 GNU/Linux
# rbd info truc
size 20480 MB in 5120 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.9eca966b8b4567
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
# rbd map truc
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by
the kernel with "rbd feature disable pool/truc object-map fast-diff
deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
# dmesg | tail -1
[1108045.667333] rbd: image truc: image uses unsupported features: 0x38

Those are rbd image features. Your email also mentioned "libcephfs via
fuse", so I assumed you had meant cluster-wide feature bits.
Thanks,
Ilya

[Ilya Dryomov]

Post by c***@jack.fr.eu.org
Ok ^^
For Cephfs, as far as I know, quota support is not supported in kernel space
This is not specific to luminous, tho

quota support is coming, hopefully in 4.17.

Thanks,

Ilya

[Nicolas Huillard]

Post by Ilya Dryomov

Post by c***@jack.fr.eu.org
The stock kernel from Debian is perfect
Spectre / meltdown mitigations are worthless for a Ceph point of view,
and should be disabled (again, strictly from a Ceph point of view)

I know that Ceph itself don't need this, but the cpeh client machines,
specially those hosting VMs or mùore diverse code, should have those
mitigations.

Post by Ilya Dryomov

Post by c***@jack.fr.eu.org
If you need the luminous features, using the userspace
implementations
is required (librbd via rbd-nbd or qemu, libcephfs via fuse etc)

I'd rather use the faster kernel cephfs implementation instead of fuse,
specially with the Meltdown PTI mitigation (I guess fuse implies twice
the userland-to-kernel calls which are costly using PTI).
I don't have an idea yet re. RBD...

Post by Ilya Dryomov
luminous cluster-wide feature bits are supported since kernel 4.13.

This means that there are differences between 4.9 and 4.14 re. Ceph
features. I know that quota are not supported yet in any kernel, but I
don't use this...
Are there some performance/stability improvements in the kernel that
would justify using 4.14 instead of 4.9 ? I can't find any list
anywhere...
Since I'm building a new cluster, I'd rather choose the latest software
from the start if it's justified.

[Ilya Dryomov]

Post by Nicolas Huillard

Post by Ilya Dryomov

Post by c***@jack.fr.eu.org
The stock kernel from Debian is perfect
Spectre / meltdown mitigations are worthless for a Ceph point of view,
and should be disabled (again, strictly from a Ceph point of view)

I know that Ceph itself don't need this, but the cpeh client machines,
specially those hosting VMs or mùore diverse code, should have those
mitigations.

Post by Ilya Dryomov

Post by c***@jack.fr.eu.org
If you need the luminous features, using the userspace
implementations
is required (librbd via rbd-nbd or qemu, libcephfs via fuse etc)

I'd rather use the faster kernel cephfs implementation instead of fuse,
specially with the Meltdown PTI mitigation (I guess fuse implies twice
the userland-to-kernel calls which are costly using PTI).
I don't have an idea yet re. RBD...

Post by Ilya Dryomov
luminous cluster-wide feature bits are supported since kernel 4.13.

This means that there are differences between 4.9 and 4.14 re. Ceph
features. I know that quota are not supported yet in any kernel, but I
don't use this...

luminous cluster-wide features include pg-upmap, which in concert with
the new mgr balancer module can provide the perfect distribution of PGs
accross OSDs, and some other OSD performance and memory usage related
improvements.

Post by Nicolas Huillard
Are there some performance/stability improvements in the kernel that
would justify using 4.14 instead of 4.9 ? I can't find any list
anywhere...
Since I'm building a new cluster, I'd rather choose the latest software
from the start if it's justified.

From the point of view of the kernel client, a number of issues get
fixed in every kernel release, but only a handful of most important
patches get backported.

Given that 4.14 is available in your environment, there is really no
reason to use 4.9, whether you are starting from scratch or not.

Thanks,

Ilya