[ceph-users] Luminous v12.2.10 released

[Abhishek Lekshmanan]

We're happy to announce the tenth bug fix release of the Luminous
v12.2.x long term stable release series. The previous release, v12.2.9,
introduced the PG hard-limit patches which were found to cause an issue
in certain upgrade scenarios, and this release was expedited to revert
those patches. If you already successfully upgraded to v12.2.9, you
should **not** upgrade to v12.2.10, but rather **wait** for a release in
which http://tracker.ceph.com/issues/36686 is addressed. All other users
are encouraged to upgrade to this release.

Notable Changes
---------------

* This release reverts the PG hard-limit patches added in v12.2.9 in which,
a partial upgrade during a recovery/backfill, can cause the osds on the
previous version, to fail with assert(trim_to <= info.last_complete). The
workaround for users is to upgrade and restart all OSDs to a version with the
pg hard limit, or only upgrade when all PGs are active+clean.

See also: http://tracker.ceph.com/issues/36686

As mentioned above if you've successfully upgraded to v12.2.9 DO NOT
upgrade to v12.2.10 until the linked tracker issue has been fixed.

* The bluestore_cache_* options are no longer needed. They are replaced
by osd_memory_target, defaulting to 4GB. BlueStore will expand
and contract its cache to attempt to stay within this
limit. Users upgrading should note this is a higher default
than the previous bluestore_cache_size of 1GB, so OSDs using
BlueStore will use more memory by default.

For more details, see BlueStore docs[1]


For the complete release notes with changelog, please check out the
release blog entry at:
http://ceph.com/releases/v12-2-10-luminous-released

Getting ceph:
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-12.2.10.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: 177915764b752804194937482a39e95e0ca3de94


[1]: http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#cache-size

--
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)

[Simon Ironside]

Post by Abhishek Lekshmanan
We're happy to announce the tenth bug fix release of the Luminous
v12.2.x long term stable release series. The previous release, v12.2.9,
introduced the PG hard-limit patches which were found to cause an issue
in certain upgrade scenarios, and this release was expedited to revert
those patches. If you already successfully upgraded to v12.2.9, you
should **not** upgrade to v12.2.10, but rather **wait** for a release in
which http://tracker.ceph.com/issues/36686 is addressed. All other users
are encouraged to upgrade to this release.

Is it safe for v12.2.9 users upgrade to v13.2.2 Mimic?

http://tracker.ceph.com/issues/36686 suggests a similar revert might be
on the cards for v13.2.3 so I'm not sure.

Thanks,
Simon

[Josh Durgin]

Post by Simon Ironside

Post by Abhishek Lekshmanan
We're happy to announce the tenth bug fix release of the Luminous
v12.2.x long term stable release series. The previous release, v12.2.9,
introduced the PG hard-limit patches which were found to cause an issue
in certain upgrade scenarios, and this release was expedited to revert
those patches. If you already successfully upgraded to v12.2.9, you
should **not** upgrade to v12.2.10, but rather **wait** for a release in
which http://tracker.ceph.com/issues/36686 is addressed. All other users
are encouraged to upgrade to this release.

Is it safe for v12.2.9 users upgrade to v13.2.2 Mimic?
http://tracker.ceph.com/issues/36686 suggests a similar revert might be
on the cards for v13.2.3 so I'm not sure.

Yes, 13.2.2 has the same pg hard limit code as 12.2.9, so that upgrade
is safe. The only danger is running a mixed-version cluster where some
of the osds have the pg hard limit code, and others do not.

13.2.3 will have a similar revert, so if you are running anything other
than 12.2.9 or 13.2.2 you can go directly to 13.2.3.

Josh

[Josh Durgin]

Post by Josh Durgin
13.2.3 will have a similar revert, so if you are running anything other
than 12.2.9 or 13.2.2 you can go directly to 13.2.3.

Correction: I misremembered here, we're not reverting these patches for
13.2.3, so 12.2.9 users can upgrade to 13.2.2 or later, but other
luminous users should avoid 13.2.2 or later for the time being, unless
they can accept some downtime during the upgrade.

See http://tracker.ceph.com/issues/36686#note-6 for more detail.

Josh

[Graham Allan]

Post by Abhishek Lekshmanan
We're happy to announce the tenth bug fix release of the Luminous
v12.2.x long term stable release series. The previous release, v12.2.9,
introduced the PG hard-limit patches which were found to cause an issue
in certain upgrade scenarios, and this release was expedited to revert
those patches. If you already successfully upgraded to v12.2.9, you
should **not** upgrade to v12.2.10, but rather **wait** for a release in
which http://tracker.ceph.com/issues/36686 is addressed. All other users
are encouraged to upgrade to this release.

I wonder if you can comment on upgrade policy for a mixed cluster - eg
where the majority is running 12.2.8 but a handful of newly-added osd
nodes were installed with 12.2.9. Should the 12.2.8 nodes be upgraded to
12.2.10 (this does sound like it should have no negative effects) and
just the 12.2.9 nodes kept to wait for a future release - or wait on all?

Thanks, Graham

[Josh Durgin]

Post by Graham Allan

Post by Abhishek Lekshmanan
We're happy to announce the tenth bug fix release of the Luminous
v12.2.x long term stable release series. The previous release, v12.2.9,
introduced the PG hard-limit patches which were found to cause an issue
in certain upgrade scenarios, and this release was expedited to revert
those patches. If you already successfully upgraded to v12.2.9, you
should **not** upgrade to v12.2.10, but rather **wait** for a release in
which http://tracker.ceph.com/issues/36686 is addressed. All other users
are encouraged to upgrade to this release.

I wonder if you can comment on upgrade policy for a mixed cluster - eg
where the majority is running 12.2.8 but a handful of newly-added osd
nodes were installed with 12.2.9. Should the 12.2.8 nodes be upgraded to
12.2.10 (this does sound like it should have no negative effects) and
just the 12.2.9 nodes kept to wait for a future release - or wait on all?

I'd suggest upgrading everything to 12.2.10. If you aren't hitting
crashes already with this mixed 12.2.9 + 12.2.8 cluster, a further
upgrade shouldn't cause any issues.

Josh

[Josh Durgin]

The only relevant component for this issue is OSDs. Upgrading the
monitors first as usual is fine. If your OSDs are all 12.2.8, moving
them to 12.2.10 has no chance of hitting this bug.

If you upgrade the OSDs to 13.2.2, which does have the PG hard limit
patches, you may hit the bug as noted here:

http://docs.ceph.com/docs/master/releases/mimic/#v13-2-2-mimic

Josh

On mixed node environment, does it matter what role ceph component has, in terms of updating them all to 12.2.10.
I have monitor nodes at 12.2.9 and OSD nodes at 12.2.8. I am preparing to upgrade to mimic 13.2.2 and will be running dist-upgrade+reboot prior to upgrade to 13.2.2, that would move them all to 12.2.10.
Is that ok?
And should I still update my monitor nodes first to 12.2.10 and then OSD nodes to 12.2.10?
Thank you,
Christa

Post by Graham Allan

Post by Abhishek Lekshmanan
We're happy to announce the tenth bug fix release of the Luminous
v12.2.x long term stable release series. The previous release, v12.2.9,
introduced the PG hard-limit patches which were found to cause an issue
in certain upgrade scenarios, and this release was expedited to revert
those patches. If you already successfully upgraded to v12.2.9, you
should **not** upgrade to v12.2.10, but rather **wait** for a release in
which http://tracker.ceph.com/issues/36686 is addressed. All other users
are encouraged to upgrade to this release.

I wonder if you can comment on upgrade policy for a mixed cluster - eg
where the majority is running 12.2.8 but a handful of newly-added osd
nodes were installed with 12.2.9. Should the 12.2.8 nodes be upgraded to
12.2.10 (this does sound like it should have no negative effects) and
just the 12.2.9 nodes kept to wait for a future release - or wait on all?

I'd suggest upgrading everything to 12.2.10. If you aren't hitting
crashes already with this mixed 12.2.9 + 12.2.8 cluster, a further
upgrade shouldn't cause any issues.
Josh
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Robert Sander]

Post by Abhishek Lekshmanan
As mentioned above if you've successfully upgraded to v12.2.9 DO NOT
upgrade to v12.2.10 until the linked tracker issue has been fixed.

What about clusters currently running 12.2.9 (because this was the
version in the repos when they got installed / last upgraded) where new
nodes are scheduled to setup?
Can the new nodes be installed with 12.2.10 and run with the other
12.2.9 nodes?
Should the new nodes be pinned to 12.2.9?

Regards
--
Robert Sander
Heinlein Support GmbH
Linux: Akademie - Support - Hosting
http://www.heinlein-support.de

Tel: [phone]
Fax: [phone]

Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
GeschÀftsfÌhrer: Peer Heinlein -- Sitz: Berlin

[Josh Durgin]

Post by Robert Sander

Post by Abhishek Lekshmanan
As mentioned above if you've successfully upgraded to v12.2.9 DO NOT
upgrade to v12.2.10 until the linked tracker issue has been fixed.

What about clusters currently running 12.2.9 (because this was the
version in the repos when they got installed / last upgraded) where new
nodes are scheduled to setup?
Can the new nodes be installed with 12.2.10 and run with the other
12.2.9 nodes?
Should the new nodes be pinned to 12.2.9?

To be safe, pin them to 12.2.9 until we have a safe upgrade path in a
future luminous release. Alternately you can restart them all at once as
12.2.10 if you don't mind a short loss of availability.

Josh