Discussion:
v0.94.7 Hammer released
(too old to reply)
Sage Weil
2016-05-13 14:21:51 UTC
Permalink
This Hammer point release fixes several minor bugs. It also includes a
backport of an improved ¡ceph osd reweight-by-utilization¢ command for
handling OSDs with higher-than-average utilizations.

We recommend that all hammer v0.94.x users upgrade.

For more detailed information, see the release announcement at

http://ceph.com/releases/v0-94-7-hammer-released/

or the complete changelog at

http://docs.ceph.com/docs/master/_downloads/v0.94.6.txt

Getting Ceph
------------

* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-0.94.7.tar.gz
* For packages, see http://ceph.com/docs/master/install/get-packages
* For ceph-deploy, see http://ceph.com/docs/master/install/install-ceph-deploy
Chris Dunlop
2016-05-16 06:20:09 UTC
Permalink
Post by Sage Weil
This Hammer point release fixes several minor bugs. It also includes a
backport of an improved ‘ceph osd reweight-by-utilization’ command for
handling OSDs with higher-than-average utilizations.
We recommend that all hammer v0.94.x users upgrade.
Per http://download.ceph.com/debian-hammer/pool/main/c/ceph/

ceph-common_0.94.7-1trusty_amd64.deb 11-May-2016 16:08 5959876
ceph-common_0.94.7-1xenial_amd64.deb 11-May-2016 15:54 6037236
ceph-common_0.94.7-1xenial_arm64.deb 11-May-2016 16:06 5843722
ceph-common_0.94.7-1~bpo80+1_amd64.deb 11-May-2016 16:08 6028036

Once again, no debian wheezy (~bpo70) version?

Ubuntu Precise missed out this time too.

Oddly, the date on the previously released wheezy version changed at the
same time as the 0.94.7 releases above, it was previously 15-Dec-2015 15:32:

ceph-common_0.94.5-1~bpo70+1_amd64.deb 11-May-2016 15:57 9868188


Cheers,

Chris
Emmanuel Lacour
2016-05-16 07:00:46 UTC
Permalink
Post by Chris Dunlop
Post by Sage Weil
This Hammer point release fixes several minor bugs. It also includes a
backport of an improved ‘ceph osd reweight-by-utilization’ command for
handling OSDs with higher-than-average utilizations.
We recommend that all hammer v0.94.x users upgrade.
Per http://download.ceph.com/debian-hammer/pool/main/c/ceph/
ceph-common_0.94.7-1trusty_amd64.deb 11-May-2016 16:08 5959876
ceph-common_0.94.7-1xenial_amd64.deb 11-May-2016 15:54 6037236
ceph-common_0.94.7-1xenial_arm64.deb 11-May-2016 16:06 5843722
ceph-common_0.94.7-1~bpo80+1_amd64.deb 11-May-2016 16:08 6028036
Once again, no debian wheezy (~bpo70) version?
+1

I see some git builds of a 0.94.6 here
http://gitbuilder.ceph.com/ceph-deb-wheezy-x86_64-basic/ref/
Dan van der Ster
2016-05-16 07:40:37 UTC
Permalink
Post by Chris Dunlop
Post by Sage Weil
This Hammer point release fixes several minor bugs. It also includes a
backport of an improved ‘ceph osd reweight-by-utilization’ command for
handling OSDs with higher-than-average utilizations.
We recommend that all hammer v0.94.x users upgrade.
Per http://download.ceph.com/debian-hammer/pool/main/c/ceph/
ceph-common_0.94.7-1trusty_amd64.deb 11-May-2016 16:08 5959876
ceph-common_0.94.7-1xenial_amd64.deb 11-May-2016 15:54 6037236
ceph-common_0.94.7-1xenial_arm64.deb 11-May-2016 16:06 5843722
ceph-common_0.94.7-1~bpo80+1_amd64.deb 11-May-2016 16:08 6028036
Once again, no debian wheezy (~bpo70) version?
Ubuntu Precise missed out this time too.
el6 is also missing.

-- dan
Dan van der Ster
2016-05-17 08:04:41 UTC
Permalink
Hi Sage et al,

I'm updating our pre-prod cluster from 0.94.6 to 0.94.7 and after
upgrading the ceph-mon's I'm getting loads of warnings like:

2016-05-17 10:01:29.314785 osd.76 [WRN] failed to encode map e103116
with expected crc

I've seen that error is whitelisted in the qa-suite:
https://github.com/ceph/ceph-qa-suite/pull/602/files

Is it really harmless? (This is the first time I've seen such a warning).

Thanks in advance!

Dan
Post by Sage Weil
This Hammer point release fixes several minor bugs. It also includes a
backport of an improved ‘ceph osd reweight-by-utilization’ command for
handling OSDs with higher-than-average utilizations.
We recommend that all hammer v0.94.x users upgrade.
For more detailed information, see the release announcement at
http://ceph.com/releases/v0-94-7-hammer-released/
or the complete changelog at
http://docs.ceph.com/docs/master/_downloads/v0.94.6.txt
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-0.94.7.tar.gz
* For packages, see http://ceph.com/docs/master/install/get-packages
* For ceph-deploy, see http://ceph.com/docs/master/install/install-ceph-deploy
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Max A. Krasilnikov
2016-05-17 08:44:59 UTC
Permalink
Hello!
Post by Dan van der Ster
Hi Sage et al,
I'm updating our pre-prod cluster from 0.94.6 to 0.94.7 and after
2016-05-17 10:01:29.314785 osd.76 [WRN] failed to encode map e103116
with expected crc
https://github.com/ceph/ceph-qa-suite/pull/602/files
Is it really harmless? (This is the first time I've seen such a warning).
I have the same warning using some jewel OSDs in hammer cluster (considering
step-by-step per-node upgrade). No problems, just warning in logs.
--
WBR, Max A. Krasilnikov
koukou73gr
2016-05-17 08:59:05 UTC
Permalink
Same here.

Warnings appeared for OSDs running the .6 version each time one of the
rest was restarted to the .7 version.

When the last .6 OSD host was upgraded, there where no more warnings
from the rest.

Cluster seems happy :)

-K.
Post by Dan van der Ster
Hi Sage et al,
I'm updating our pre-prod cluster from 0.94.6 to 0.94.7 and after
2016-05-17 10:01:29.314785 osd.76 [WRN] failed to encode map e103116
with expected crc
https://github.com/ceph/ceph-qa-suite/pull/602/files
Is it really harmless? (This is the first time I've seen such a warning).
Thanks in advance!
Dan
Sage Weil
2016-05-17 11:56:13 UTC
Permalink
Post by Dan van der Ster
Hi Sage et al,
I'm updating our pre-prod cluster from 0.94.6 to 0.94.7 and after
2016-05-17 10:01:29.314785 osd.76 [WRN] failed to encode map e103116
with expected crc
https://github.com/ceph/ceph-qa-suite/pull/602/files
Is it really harmless? (This is the first time I've seen such a warning).
Are you sure you were upgrading from v0.94.6? I don't see anything that
would trigger these warnings going from .6 to .7, which is strange.

That said, the errors are generally harmless--it just means the monitors
are running a different version of the code and the OSDs are pulling maps
directly from a mon to ensure they are all in sync. It's normal during
many upgrades, but not expected for this particular jump...

sage
Post by Dan van der Ster
Thanks in advance!
Dan
Post by Sage Weil
This Hammer point release fixes several minor bugs. It also includes a
backport of an improved ‘ceph osd reweight-by-utilization’ command for
handling OSDs with higher-than-average utilizations.
We recommend that all hammer v0.94.x users upgrade.
For more detailed information, see the release announcement at
http://ceph.com/releases/v0-94-7-hammer-released/
or the complete changelog at
http://docs.ceph.com/docs/master/_downloads/v0.94.6.txt
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-0.94.7.tar.gz
* For packages, see http://ceph.com/docs/master/install/get-packages
* For ceph-deploy, see http://ceph.com/docs/master/install/install-ceph-deploy
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Dan van der Ster
2016-05-17 12:07:21 UTC
Permalink
Post by Sage Weil
Post by Dan van der Ster
Hi Sage et al,
I'm updating our pre-prod cluster from 0.94.6 to 0.94.7 and after
2016-05-17 10:01:29.314785 osd.76 [WRN] failed to encode map e103116
with expected crc
https://github.com/ceph/ceph-qa-suite/pull/602/files
Is it really harmless? (This is the first time I've seen such a warning).
Are you sure you were upgrading from v0.94.6?
Absolutely. I first updated the mons, which I restarted into quorum
with 0.96.7. Then any changes to the osdmap triggered the failed to
encode warning.
The upgrade sequence went like this:

Update mons 0.94.6 to 0.94.7, restart, quorum. No warnings.
Update mds's 0.94.6 to 0.94.7, restart. Warnings from ~all osds.
Update osds 0.94.6 to 0.94.7, restart host by host. The 0.94.6 osds
printed warnings, the new OSDs did not.
Post by Sage Weil
I don't see anything that
would trigger these warnings going from .6 to .7, which is strange.
Could the osdmap GMT hitset changes have caused it? Commits Mar 24 here:

https://github.com/ceph/ceph/compare/v0.94.6...v0.94.7?expand=1
Post by Sage Weil
That said, the errors are generally harmless--it just means the monitors
are running a different version of the code and the OSDs are pulling maps
directly from a mon to ensure they are all in sync. It's normal during
many upgrades, but not expected for this particular jump...
Then I'm curious if others are getting this from 0.94.6 to 0.94.7.
For now I'm waiting to update our prod cluster.

Thanks!

Dan
Post by Sage Weil
sage
Post by Dan van der Ster
Thanks in advance!
Dan
Post by Sage Weil
This Hammer point release fixes several minor bugs. It also includes a
backport of an improved ‘ceph osd reweight-by-utilization’ command for
handling OSDs with higher-than-average utilizations.
We recommend that all hammer v0.94.x users upgrade.
For more detailed information, see the release announcement at
http://ceph.com/releases/v0-94-7-hammer-released/
or the complete changelog at
http://docs.ceph.com/docs/master/_downloads/v0.94.6.txt
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-0.94.7.tar.gz
* For packages, see http://ceph.com/docs/master/install/get-packages
* For ceph-deploy, see http://ceph.com/docs/master/install/install-ceph-deploy
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Christian Balzer
2016-05-17 12:14:35 UTC
Permalink
Hello,

for the record, I did the exact same sequence (no MDS) on my test cluster
with exactly the same results.

Didn't report it as I assumed it to be a more noisy (but harmless)
upgrade artifact.

Christian
Post by Dan van der Ster
Post by Sage Weil
Post by Dan van der Ster
Hi Sage et al,
I'm updating our pre-prod cluster from 0.94.6 to 0.94.7 and after
2016-05-17 10:01:29.314785 osd.76 [WRN] failed to encode map e103116
with expected crc
https://github.com/ceph/ceph-qa-suite/pull/602/files
Is it really harmless? (This is the first time I've seen such a warning).
Are you sure you were upgrading from v0.94.6?
Absolutely. I first updated the mons, which I restarted into quorum
with 0.96.7. Then any changes to the osdmap triggered the failed to
encode warning.
Update mons 0.94.6 to 0.94.7, restart, quorum. No warnings.
Update mds's 0.94.6 to 0.94.7, restart. Warnings from ~all osds.
Update osds 0.94.6 to 0.94.7, restart host by host. The 0.94.6 osds
printed warnings, the new OSDs did not.
Post by Sage Weil
I don't see anything that
would trigger these warnings going from .6 to .7, which is strange.
https://github.com/ceph/ceph/compare/v0.94.6...v0.94.7?expand=1
Post by Sage Weil
That said, the errors are generally harmless--it just means the
monitors are running a different version of the code and the OSDs are
pulling maps directly from a mon to ensure they are all in sync. It's
normal during many upgrades, but not expected for this particular
jump...
Then I'm curious if others are getting this from 0.94.6 to 0.94.7.
For now I'm waiting to update our prod cluster.
Thanks!
Dan
Post by Sage Weil
sage
Post by Dan van der Ster
Thanks in advance!
Dan
Post by Sage Weil
This Hammer point release fixes several minor bugs. It also
includes a backport of an improved ‘ceph osd
reweight-by-utilization’ command for handling OSDs with
higher-than-average utilizations.
We recommend that all hammer v0.94.x users upgrade.
For more detailed information, see the release announcement at
http://ceph.com/releases/v0-94-7-hammer-released/
or the complete changelog at
http://docs.ceph.com/docs/master/_downloads/v0.94.6.txt
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-0.94.7.tar.gz
* For packages, see http://ceph.com/docs/master/install/get-packages
* For ceph-deploy, see
http://ceph.com/docs/master/install/install-ceph-deploy
_______________________________________________ ceph-users mailing
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Christian Balzer Network/Systems Engineer
***@gol.com Global OnLine Japan/Rakuten Communications
http://www.gol.com/
Dan van der Ster
2016-05-24 08:02:23 UTC
Permalink
Hi all,

I'm mid-upgrade on a large cluster now. The upgrade is not going smoothly
-- it looks like the ceph-mon's are getting bombarded by so many of these
crc error warnings that they go into elections.

Did anyone upgrade a large cluster from 0.94.6 to 0.94.7 ? If not I'd
advise waiting until this is better understood.

Cheers, Dan
Post by Christian Balzer
Hello,
for the record, I did the exact same sequence (no MDS) on my test cluster
with exactly the same results.
Didn't report it as I assumed it to be a more noisy (but harmless)
upgrade artifact.
Christian
Post by Dan van der Ster
Post by Sage Weil
Post by Dan van der Ster
Hi Sage et al,
I'm updating our pre-prod cluster from 0.94.6 to 0.94.7 and after
2016-05-17 10:01:29.314785 osd.76 [WRN] failed to encode map e103116
with expected crc
https://github.com/ceph/ceph-qa-suite/pull/602/files
Is it really harmless? (This is the first time I've seen such a warning).
Are you sure you were upgrading from v0.94.6?
Absolutely. I first updated the mons, which I restarted into quorum
with 0.96.7. Then any changes to the osdmap triggered the failed to
encode warning.
Update mons 0.94.6 to 0.94.7, restart, quorum. No warnings.
Update mds's 0.94.6 to 0.94.7, restart. Warnings from ~all osds.
Update osds 0.94.6 to 0.94.7, restart host by host. The 0.94.6 osds
printed warnings, the new OSDs did not.
Post by Sage Weil
I don't see anything that
would trigger these warnings going from .6 to .7, which is strange.
https://github.com/ceph/ceph/compare/v0.94.6...v0.94.7?expand=1
Post by Sage Weil
That said, the errors are generally harmless--it just means the
monitors are running a different version of the code and the OSDs are
pulling maps directly from a mon to ensure they are all in sync. It's
normal during many upgrades, but not expected for this particular
jump...
Then I'm curious if others are getting this from 0.94.6 to 0.94.7.
For now I'm waiting to update our prod cluster.
Thanks!
Dan
Post by Sage Weil
sage
Post by Dan van der Ster
Thanks in advance!
Dan
Post by Sage Weil
This Hammer point release fixes several minor bugs. It also
includes a backport of an improved ‘ceph osd
reweight-by-utilization’ command for handling OSDs with
higher-than-average utilizations.
We recommend that all hammer v0.94.x users upgrade.
For more detailed information, see the release announcement at
http://ceph.com/releases/v0-94-7-hammer-released/
or the complete changelog at
http://docs.ceph.com/docs/master/_downloads/v0.94.6.txt
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-0.94.7.tar.gz
* For packages, see
http://ceph.com/docs/master/install/get-packages
Post by Dan van der Ster
Post by Sage Weil
Post by Dan van der Ster
Post by Sage Weil
* For ceph-deploy, see
http://ceph.com/docs/master/install/install-ceph-deploy
_______________________________________________ ceph-users mailing
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Christian Balzer Network/Systems Engineer
http://www.gol.com/
Dan van der Ster
2016-05-24 09:42:47 UTC
Permalink
Hi again,

We just finished the upgrade (5 mons, 1200 OSDs). As I mentioned, we
had loads of monitor elections and slow requests during the upgrades.
perf top showed the leader spending lots of time in LogMonitor::preprocess_log:

43.79% ceph-mon [.] LogMonitor::preprocess_log

To mitigate I tried a few things to minimize osd map changes: set
noout, update crush on start = false. I also increased the mon lease
timeouts:

ceph tell mon.* injectargs -- --mon_lease=15
--mon_lease_renew_interval=9 --mon_lease_ack_timeout=30

None of that really helped. But finally I did:

ceph tell osd.* injectargs -- --clog_to_monitors=false

which made things much better.

When I upgrade our 2nd cluster tomorrow, I'll set
clog_to_monitors=false before starting.

Cheers, Dan
Post by Dan van der Ster
Hi all,
I'm mid-upgrade on a large cluster now. The upgrade is not going smoothly --
it looks like the ceph-mon's are getting bombarded by so many of these crc
error warnings that they go into elections.
Did anyone upgrade a large cluster from 0.94.6 to 0.94.7 ? If not I'd advise
waiting until this is better understood.
Cheers, Dan
Post by Christian Balzer
Hello,
for the record, I did the exact same sequence (no MDS) on my test cluster
with exactly the same results.
Didn't report it as I assumed it to be a more noisy (but harmless)
upgrade artifact.
Christian
Post by Dan van der Ster
Post by Sage Weil
Post by Dan van der Ster
Hi Sage et al,
I'm updating our pre-prod cluster from 0.94.6 to 0.94.7 and after
2016-05-17 10:01:29.314785 osd.76 [WRN] failed to encode map e103116
with expected crc
https://github.com/ceph/ceph-qa-suite/pull/602/files
Is it really harmless? (This is the first time I've seen such a warning).
Are you sure you were upgrading from v0.94.6?
Absolutely. I first updated the mons, which I restarted into quorum
with 0.96.7. Then any changes to the osdmap triggered the failed to
encode warning.
Update mons 0.94.6 to 0.94.7, restart, quorum. No warnings.
Update mds's 0.94.6 to 0.94.7, restart. Warnings from ~all osds.
Update osds 0.94.6 to 0.94.7, restart host by host. The 0.94.6 osds
printed warnings, the new OSDs did not.
Post by Sage Weil
I don't see anything that
would trigger these warnings going from .6 to .7, which is strange.
https://github.com/ceph/ceph/compare/v0.94.6...v0.94.7?expand=1
Post by Sage Weil
That said, the errors are generally harmless--it just means the
monitors are running a different version of the code and the OSDs are
pulling maps directly from a mon to ensure they are all in sync. It's
normal during many upgrades, but not expected for this particular
jump...
Then I'm curious if others are getting this from 0.94.6 to 0.94.7.
For now I'm waiting to update our prod cluster.
Thanks!
Dan
Post by Sage Weil
sage
Post by Dan van der Ster
Thanks in advance!
Dan
Post by Sage Weil
This Hammer point release fixes several minor bugs. It also
includes a backport of an improved ‘ceph osd
reweight-by-utilization’ command for handling OSDs with
higher-than-average utilizations.
We recommend that all hammer v0.94.x users upgrade.
For more detailed information, see the release announcement at
http://ceph.com/releases/v0-94-7-hammer-released/
or the complete changelog at
http://docs.ceph.com/docs/master/_downloads/v0.94.6.txt
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-0.94.7.tar.gz
* For packages, see
http://ceph.com/docs/master/install/get-packages
* For ceph-deploy, see
http://ceph.com/docs/master/install/install-ceph-deploy
_______________________________________________ ceph-users mailing
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Christian Balzer Network/Systems Engineer
http://www.gol.com/
Loading...