Discussion:
Broken Buckets after Jewel->Luminous Upgrade
(too old to reply)
Ingo Reimann
2018-01-30 09:32:04 UTC
Permalink
Raw Message
Hi,

We got a nasty issue during our Jewel->Luminous upgrade. The Mon/MGR and
OSD part went well, but the first of our rgws is incompatible to the
others:

The problem:
Some Buckets are not accessible from the luminous gateway. The metadata
for that buckets seemed ok, but listing was not possible. A local s3cmd
got "404 NoSuchKey". I exported and imported the metadata for one instance
and ran radosgw-admin --check. Now the bucket is listable but empty under
luminous and broken under jewel. The corresponding directory object still
contains the file in its omap.

I am afraid to corrupt my cluster so I stopped the upgrade for the other
gateways.

What could be the problem,and how may I solve that?

Any help is apreciated.

Regards,

Ingo Reimann
Dunkel GmbH
Robin H. Johnson
2018-01-30 18:17:29 UTC
Permalink
Raw Message
Post by Ingo Reimann
Some Buckets are not accessible from the luminous gateway. The metadata
for that buckets seemed ok, but listing was not possible. A local s3cmd
got "404 NoSuchKey". I exported and imported the metadata for one instance
and ran radosgw-admin --check. Now the bucket is listable but empty under
luminous and broken under jewel. The corresponding directory object still
contains the file in its omap.
I am afraid to corrupt my cluster so I stopped the upgrade for the other
gateways.
I have a couple of bugs open for possible the same issue:
https://tracker.ceph.com/issues/22756 - buckets showing as empty
http://tracker.ceph.com/issues/22714 - old AccessKeys not working

One more to come after more diagnosis my side, where some old files
don't work properly anymore (dropping off at a multiple of 512K)
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : ***@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Ingo Reimann
2018-01-31 06:39:02 UTC
Permalink
Raw Message
Hi Robin,

thanks for your reply.

Concerning "https://tracker.ceph.com/issues/22756 - buckets showing as
empty": Our cluster is rather old - argonaut, but the affected bucket and
user are created under jewel.

If you need more data, I may post it.

Best regards,
Ingo

-----Urspr√ľngliche Nachricht-----
Von: Robin H. Johnson [mailto:***@gentoo.org]
Gesendet: Dienstag, 30. Januar 2018 19:17
An: Ingo Reimann
Cc: ceph-users
Betreff: Re: [ceph-users] Broken Buckets after Jewel->Luminous Upgrade
Post by Ingo Reimann
Some Buckets are not accessible from the luminous gateway. The
metadata for that buckets seemed ok, but listing was not possible. A
local s3cmd got "404 NoSuchKey". I exported and imported the metadata
for one instance and ran radosgw-admin --check. Now the bucket is
listable but empty under luminous and broken under jewel. The
corresponding directory object still contains the file in its omap.
I am afraid to corrupt my cluster so I stopped the upgrade for the
other gateways.
I have a couple of bugs open for possible the same issue:
https://tracker.ceph.com/issues/22756 - buckets showing as empty
http://tracker.ceph.com/issues/22714 - old AccessKeys not working

One more to come after more diagnosis my side, where some old files don't
work properly anymore (dropping off at a multiple of 512K)


--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : ***@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP :
7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Robin H. Johnson
2018-01-31 06:56:49 UTC
Permalink
Raw Message
Post by Ingo Reimann
Hi Robin,
thanks for your reply.
Concerning "https://tracker.ceph.com/issues/22756 - buckets showing as
empty": Our cluster is rather old - argonaut, but the affected bucket and
user are created under jewel.
If you need more data, I may post it.
Yes, please, I'm even more interested now, as I only have customer
reports from old buckets! I'm wondering if it's slightly related to
issue 22714 for some other object that starts with an underscore.
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : ***@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Robin H. Johnson
2018-02-06 06:26:50 UTC
Permalink
Raw Message
Post by Ingo Reimann
What could be the problem,and how may I solve that?
For anybody else tracking this, the logs & debugging info are filed at
http://tracker.ceph.com/issues/22928
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : ***@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Loading...