[ceph-users] CephFS kernel client versions - pg-upmap

[j***@krogh.cc]

Hi.

I tried to enable the "new smart balancing" - backend are on RH luminous
clients are Ubuntu 4.15 kernel.

As per: http://docs.ceph.com/docs/mimic/rados/operations/upmap/
$ sudo ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 1 connected
client(s) look like firefly (missing 0xe01000000200000); 1 connected
client(s) look like firefly (missing 0xe01000000000000); 1 connected
client(s) look like hammer (missing 0xe00000000200000); 55 connected
client(s) look like jewel (missing 0x800000000000000); add
--yes-i-really-mean-it to do it anyway

ok, so 4.15 kernel connects as a "hammer" (<1.0) client? Is there a
huge gap in upstreaming kernel clients to kernel.org or what am I
misreading here?

Hammer is 2015'ish - 4.15 is January 2018'ish?

Is kernel client development lacking behind ?

Jesper

[Ilya Dryomov]

On Sat, Nov 3, 2018 at 10:41 AM <***@krogh.cc> wrote:
>
> Hi.
>
> I tried to enable the "new smart balancing" - backend are on RH luminous
> clients are Ubuntu 4.15 kernel.
>
> As per: http://docs.ceph.com/docs/mimic/rados/operations/upmap/
> $ sudo ceph osd set-require-min-compat-client luminous
> Error EPERM: cannot set require_min_compat_client to luminous: 1 connected
> client(s) look like firefly (missing 0xe01000000200000); 1 connected
> client(s) look like firefly (missing 0xe01000000000000); 1 connected
> client(s) look like hammer (missing 0xe00000000200000); 55 connected
> client(s) look like jewel (missing 0x800000000000000); add
> --yes-i-really-mean-it to do it anyway
>
> ok, so 4.15 kernel connects as a "hammer" (<1.0) client? Is there a
> huge gap in upstreaming kernel clients to kernel.org or what am I
> misreading here?
>
> Hammer is 2015'ish - 4.15 is January 2018'ish?
>
> Is kernel client development lacking behind ?

Hi Jesper,

There are four different groups of clients in that output. Which one
of those four is the kernel client? Are you sure it's just the hammer
one?

Thanks,

Ilya

[Stefan Kooman]

Quoting Ilya Dryomov (***@gmail.com):
> On Sat, Nov 3, 2018 at 10:41 AM <***@krogh.cc> wrote:
> >
> > Hi.
> >
> > I tried to enable the "new smart balancing" - backend are on RH luminous
> > clients are Ubuntu 4.15 kernel.
[cut]
> > ok, so 4.15 kernel connects as a "hammer" (<1.0) client? Is there a
> > huge gap in upstreaming kernel clients to kernel.org or what am I
> > misreading here?
> >
> > Hammer is 2015'ish - 4.15 is January 2018'ish?
> >
> > Is kernel client development lacking behind ?
>
> Hi Jesper,
>
> There are four different groups of clients in that output. Which one
> of those four is the kernel client? Are you sure it's just the hammer
> one?

I'm pretty sure it isn't. I'm trying to do the same (force luminous
clients only) but ran into the same issue. Even when running 4.19 kernel
it's interpreted as a jewel client. Here is the list I made so far:

Kernel 4.13 / 4.15:
"features": "0x7010fb86aa42ada",
"release": "jewel"

kernel 4.18 / 4.19
"features": "0x27018fb86aa42ada",
"release": "jewel"

I have tested both Ubuntu as CentOS mainline kernels. I came accross
this issue made by Sage [1], which is resolved, but which looks similiar
to this.

Gr. Stefan

[1]: https://tracker.ceph.com/issues/20475

--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 [phone] / ***@bit.nl

[Ilya Dryomov]

On Thu, Nov 8, 2018 at 2:15 PM Stefan Kooman <***@bit.nl> wrote:
>
> Quoting Ilya Dryomov (***@gmail.com):
> > On Sat, Nov 3, 2018 at 10:41 AM <***@krogh.cc> wrote:
> > >
> > > Hi.
> > >
> > > I tried to enable the "new smart balancing" - backend are on RH luminous
> > > clients are Ubuntu 4.15 kernel.
> [cut]
> > > ok, so 4.15 kernel connects as a "hammer" (<1.0) client? Is there a
> > > huge gap in upstreaming kernel clients to kernel.org or what am I
> > > misreading here?
> > >
> > > Hammer is 2015'ish - 4.15 is January 2018'ish?
> > >
> > > Is kernel client development lacking behind ?
> >
> > Hi Jesper,
> >
> > There are four different groups of clients in that output. Which one
> > of those four is the kernel client? Are you sure it's just the hammer
> > one?
>
> I'm pretty sure it isn't. I'm trying to do the same (force luminous
> clients only) but ran into the same issue. Even when running 4.19 kernel
> it's interpreted as a jewel client. Here is the list I made so far:
>
> Kernel 4.13 / 4.15:
> "features": "0x7010fb86aa42ada",
> "release": "jewel"
>
> kernel 4.18 / 4.19
> "features": "0x27018fb86aa42ada",
> "release": "jewel"
>
> I have tested both Ubuntu as CentOS mainline kernels. I came accross
> this issue made by Sage [1], which is resolved, but which looks similiar
> to this.

I asked about those hammer ones because there were clearly not 4.13+.
For 4.13+ and CentOS 7.5+ you can force require-min-compat-client with
--yes-i-really-mean-it. See

https://www.spinics.net/lists/ceph-users/msg45071.html
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/029105.html

Thanks,

Ilya

[Stefan Kooman]

Quoting Stefan Kooman (***@bit.nl):
> I'm pretty sure it isn't. I'm trying to do the same (force luminous
> clients only) but ran into the same issue. Even when running 4.19 kernel
> it's interpreted as a jewel client. Here is the list I made so far:
>
> Kernel 4.13 / 4.15:
> "features": "0x7010fb86aa42ada",
> "release": "jewel"
>
> kernel 4.18 / 4.19
> "features": "0x27018fb86aa42ada",
> "release": "jewel"

On a test cluster with kernel clients 4.13, 4.15, 4.19 I have set the
"ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it"
while doing active IO ... no issues. Remount also works ... makes me
wonder how strict this "require-min-compat-client" is ...

Gr. Stefan

--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 [phone] / ***@bit.nl

[Ilya Dryomov]

On Thu, Nov 8, 2018 at 5:10 PM Stefan Kooman <***@bit.nl> wrote:
>
> Quoting Stefan Kooman (***@bit.nl):
> > I'm pretty sure it isn't. I'm trying to do the same (force luminous
> > clients only) but ran into the same issue. Even when running 4.19 kernel
> > it's interpreted as a jewel client. Here is the list I made so far:
> >
> > Kernel 4.13 / 4.15:
> > "features": "0x7010fb86aa42ada",
> > "release": "jewel"
> >
> > kernel 4.18 / 4.19
> > "features": "0x27018fb86aa42ada",
> > "release": "jewel"
>
> On a test cluster with kernel clients 4.13, 4.15, 4.19 I have set the
> "ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it"
> while doing active IO ... no issues. Remount also works ... makes me
> wonder how strict this "require-min-compat-client" is ...

It's there to stop you from accidentally enabling new features that
some of your clients are too old for. In its current form it's quite
easy to bypass but I think there are plans to make it stronger in the
future.

You didn't actually enable any new features by bumping it to luminous.
But you shouldn't see any issues even if you go ahead and do that (e.g.
put the balancer in upmap mode) because your clients are new enough.

Thanks,

Ilya

[Linh Vu]

Kernel 4.13+ (i tested up to 4.18) missed some non-essential feature (explained by a Ceph dev on this ML) that was in Luminous, so they show up as Jewel, but otherwise they're fully compatible with upmap. We have a few hundred nodes on the kernel client with CephFS, and we also run balancer with upmap mode on those clusters successfully.

________________________________
From: ceph-users <ceph-users-***@lists.ceph.com> on behalf of Stefan Kooman <***@bit.nl>
Sent: Friday, 9 November 2018 3:10:09 AM
To: Ilya Dryomov
Cc: ceph-users
Subject: Re: [ceph-users] CephFS kernel client versions - pg-upmap

Quoting Stefan Kooman (***@bit.nl):
> I'm pretty sure it isn't. I'm trying to do the same (force luminous
> clients only) but ran into the same issue. Even when running 4.19 kernel
> it's interpreted as a jewel client. Here is the list I made so far:
>
> Kernel 4.13 / 4.15:
> "features": "0x7010fb86aa42ada",
> "release": "jewel"
>
> kernel 4.18 / 4.19
> "features": "0x27018fb86aa42ada",
> "release": "jewel"

On a test cluster with kernel clients 4.13, 4.15, 4.19 I have set the
"ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it"
while doing active IO ... no issues. Remount also works ... makes me
wonder how strict this "require-min-compat-client" is ...

Gr. Stefan

--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 [phone] / ***@bit.nl