[ceph-users] 12.2.10 rbd kernel mount issue after update

[Ashley Merrick]

I have had some ec backed Mimic RBD's mounted via the kernel module on a
Ubuntu 14.04 VM, these have been running no issues after updating the
kernel to 4.12 to support EC features.

Today I run an apt dist-upgrade which upgraded from 12.2.9 to 12.2.10,
since then I have been getting the following line in the syslog and had to
role back to using rbd-nbd for the moment which continues to work fine.

Not sure if there is a change in 12.2.10 that this is expected with a non
mimic kernel client such as Luminous.

Error in VM syslog:

feature set mismatch, my 40107b86a842ada < server's 60107b86aa42ada,
missing 200000000200000
libceph: mon2 176.9.86.219:6789 missing required protocol features

[Ilya Dryomov]

I have had some ec backed Mimic RBD's mounted via the kernel module on a Ubuntu 14.04 VM, these have been running no issues after updating the kernel to 4.12 to support EC features.
Today I run an apt dist-upgrade which upgraded from 12.2.9 to 12.2.10, since then I have been getting the following line in the syslog and had to role back to using rbd-nbd for the moment which continues to work fine.

Hi Ashley,

Are you sure that the release you upgraded from was 12.2.9?

What upgrade procedure did you follow?

Not sure if there is a change in 12.2.10 that this is expected with a non mimic kernel client such as Luminous.
feature set mismatch, my 40107b86a842ada < server's 60107b86aa42ada, missing 200000000200000
libceph: mon2 176.9.86.219:6789 missing required protocol features

These are standard luminous feature bits. It looks like your cluster
didn't require them before the upgrade and now it does. 12.2.10 should
continue to work 4.12 and any older kernel, as long as nothing luminous
only is enabled.

What is the output of "ceph osd get-require-min-compat-client"?

Thanks,

Ilya

[Ashley Merrick]

Hello,

As mentioned earlier the cluster is seperatly running on the latest mimic.

Due to 14.04 only supporting up to Luminous I was running the 12.2.9
version of ceph-common for the rbd binary.

This is what was upgraded when I did the dist-upgrade on the VM mounting
the RBD.

The cluster it self has not changed and has always been running the latest
point release on mimic.

All that changed was the move of ceph-common and dependencies on the
mounting VM.

12.2.9 + 4.12 Kernel was able to mount a Mimic EC backed RBD via KRBD,
since 12.2.10 I now get the error.

So to me looks like there was a client side change from .9 to .10 as no
change cluster side.

,Ashley

Post by Ashley Merrick

Post by Ashley Merrick
I have had some ec backed Mimic RBD's mounted via the kernel module on a

Ubuntu 14.04 VM, these have been running no issues after updating the
kernel to 4.12 to support EC features.

Post by Ashley Merrick
Today I run an apt dist-upgrade which upgraded from 12.2.9 to 12.2.10,

since then I have been getting the following line in the syslog and had to
role back to using rbd-nbd for the moment which continues to work fine.
Hi Ashley,
Are you sure that the release you upgraded from was 12.2.9?
What upgrade procedure did you follow?

Post by Ashley Merrick
Not sure if there is a change in 12.2.10 that this is expected with a

non mimic kernel client such as Luminous.

Post by Ashley Merrick
feature set mismatch, my 40107b86a842ada < server's 60107b86aa42ada,

missing 200000000200000

Post by Ashley Merrick
libceph: mon2 :6789 <http://176.9.86.219:6789> missing required

protocol features
These are standard luminous feature bits. It looks like your cluster
didn't require them before the upgrade and now it does. 12.2.10 should
continue to work 4.12 and any older kernel, as long as nothing luminous
only is enabled.
What is the output of "ceph osd get-require-min-compat-client"?
Thanks,
Ilya

[Ilya Dryomov]

Post by Ashley Merrick
Hello,
As mentioned earlier the cluster is seperatly running on the latest mimic.
Due to 14.04 only supporting up to Luminous I was running the 12.2.9 version of ceph-common for the rbd binary.
This is what was upgraded when I did the dist-upgrade on the VM mounting the RBD.
The cluster it self has not changed and has always been running the latest point release on mimic.
All that changed was the move of ceph-common and dependencies on the mounting VM.
12.2.9 + 4.12 Kernel was able to mount a Mimic EC backed RBD via KRBD, since 12.2.10 I now get the error.
So to me looks like there was a client side change from .9 to .10 as no change cluster side.

The error is coming from the kernel, not from "rbd map". "rbd map"
doesn't really do much beyond gathering options and setting up the
keys. I don't think the client side ceph upgrade is the root cause
here.

You didn't answer my other question: what is the output of "ceph osd
get-require-min-compat-client"?

Thanks,

Ilya

[Ashley Merrick]

That command returns luminous.

Understand where your coming from just weird as running the exact same
kernel before and after the package upgraded. As the only updates available
where the CEPH related ones.

I haven’t tried rolling back to .9 as I moved to rbd-nbd while opening this
thread, but I could try it would be of any use.

,Ashley

Post by Ashley Merrick

Post by Ashley Merrick
Hello,
As mentioned earlier the cluster is seperatly running on the latest

mimic.

Post by Ashley Merrick
Due to 14.04 only supporting up to Luminous I was running the 12.2.9

version of ceph-common for the rbd binary.

Post by Ashley Merrick
This is what was upgraded when I did the dist-upgrade on the VM mounting

the RBD.

Post by Ashley Merrick
The cluster it self has not changed and has always been running the

latest point release on mimic.

Post by Ashley Merrick
All that changed was the move of ceph-common and dependencies on the

mounting VM.

Post by Ashley Merrick
12.2.9 + 4.12 Kernel was able to mount a Mimic EC backed RBD via KRBD,

since 12.2.10 I now get the error.

Post by Ashley Merrick
So to me looks like there was a client side change from .9 to .10 as no

change cluster side.
The error is coming from the kernel, not from "rbd map". "rbd map"
doesn't really do much beyond gathering options and setting up the
keys. I don't think the client side ceph upgrade is the root cause
here.
You didn't answer my other question: what is the output of "ceph osd
get-require-min-compat-client"?
Thanks,
Ilya

[Ilya Dryomov]

Post by Ashley Merrick
That command returns luminous.

This is the issue.

My guess is someone ran "ceph osd set-require-min-compat-client
luminous", making it so that only luminous aware clients are allowed to
connect to the cluster. Kernel 4.12 doesn't support luminous features,
so it isn't allowed to connect. Perhaps you wanted to experiment with
the balancer module?

Thanks,

Ilya

[Ashley Merrick]

That is correct, but that command was run weeks ago.

And the RBD connected fine on 2.9 via the kernel 4.12 so I’m really lost to
why suddenly it’s now blocking a connection it originally allowed through
(even if by mistake)

Which kernel do I need to run to support luminous level?

,Ash

Post by Ilya Dryomov

Post by Ashley Merrick
That command returns luminous.

This is the issue.
My guess is someone ran "ceph osd set-require-min-compat-client
luminous", making it so that only luminous aware clients are allowed to
connect to the cluster. Kernel 4.12 doesn't support luminous features,
so it isn't allowed to connect. Perhaps you wanted to experiment with
the balancer module?
Thanks,
Ilya

[Ilya Dryomov]

Post by Ashley Merrick
That is correct, but that command was run weeks ago.
And the RBD connected fine on 2.9 via the kernel 4.12 so I’m really lost to why suddenly it’s now blocking a connection it originally allowed through (even if by mistake)

When was it last mapped, before that command was ran or after? If
before, and the command was ran with --yes-i-really-mean-it, that would
mostly explain it.

Post by Ashley Merrick
Which kernel do I need to run to support luminous level?

4.13 or newer.

Thanks,

Ilya

[Ashley Merrick]

Been mounted many times after, was never mounted before the VM’s was
created after the change was made.

I will upgrade the kernel and re-rest.

Thanks for your help

Post by Ashley Merrick
That is correct, but that command was run weeks ago.
And the RBD connected fine on 2.9 via the kernel 4.12 so I’m really lost

to why suddenly it’s now blocking a connection it originally allowed
through (even if by mistake)
When was it last mapped, before that command was ran or after? If
before, and the command was ran with --yes-i-really-mean-it, that would
mostly explain it.

Post by Ashley Merrick
Which kernel do I need to run to support luminous level?

4.13 or newer.
Thanks,
Ilya