Discussion:
[ceph-users] ceph 12.2.9 release
Dietmar Rieder
2018-11-07 09:27:31 UTC
Permalink
Hi,

I wonder if there is any release announcement for ceph 12.2.9 that I missed.
I just found the new packages on download.ceph.com, is this an official
release?

~ Dietmar
--
_________________________________________
D i e t m a r R i e d e r, Mag.Dr.
Innsbruck Medical University
Biocenter - Division for Bioinformatics
Web: http://www.icbi.at
Konstantin Shalygin
2018-11-07 10:59:49 UTC
Permalink
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I missed.
I just found the new packages on download.ceph.com, is this an official
release?
This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10



k
Simon Ironside
2018-11-07 13:58:14 UTC
Permalink
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I missed.
I just found the new packages on download.ceph.com, is this an official
release?
This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?

Simon
Thomas White
2018-11-07 14:22:13 UTC
Permalink
One of the Ceph clusters my team manages is on 12.2.9 on a Proxmox
environment seems to be running fine with simple x3 replication and RBD.
Would be interesting to know what issues have been encountered so far. All
our OSDs are simple filestore at present and our path to 12.2.9 was 10.2.7
-> 10.2.10 -> 12.2.9 in the past 2 weeks with no issues.

That said, it is disappointing these packages are making their way into
repositories without the proper announcements for an LTS release, especially
given this is enterprise orientated software.

Thomas

-----Original Message-----
From: ceph-users <ceph-users-***@lists.ceph.com> On Behalf Of Simon
Ironside
Sent: 07 November 2018 13:58
To: ceph-***@lists.ceph.com
Subject: Re: [ceph-users] ceph 12.2.9 release
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I missed.
I just found the new packages on download.ceph.com, is this an
official release?
This is because 12.2.9 have a several bugs. You should avoid to use
this release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?

Simon
Gregory Farnum
2018-11-07 15:39:59 UTC
Permalink
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed.
Post by Konstantin Shalygin
Post by Dietmar Rieder
I just found the new packages on download.ceph.com, is this an official
release?
This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
If you’ve already upgraded and have no issues then you won’t have any
trouble going forward — except perhaps on the next upgrade, if you do it
while the cluster is unhealthy.

I agree that it’s annoying when these issues make it out. We’ve had ongoing
discussions to try and improve the release process so it’s less drawn-out
and to prevent these upgrade issues from making it through testing, but
nobody has resolved it yet. If anybody has experience working with deb
repositories and handling releases, the Ceph upstream could use some
help... ;)
-Greg
Post by Dietmar Rieder
Simon
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Christoph Adomeit
2018-11-07 15:56:37 UTC
Permalink
Hello together,

we have upgraded to 12.2.9 because it was in the official repos.

Right after the update and some scrubs we have issues.

This morning after regular scrubs we had around 10% of all pgs inconstent:

pgs: 4036 active+clean
380 active+clean+inconsistent

After repairung these 380 pgs we again have:

1/93611534 objects unfound (0.000%)
28 active+clean+inconsistent
1 active+recovery_wait+degraded

Now we stopped repairing because it does not seem to solve the problem and more and more error messages are occuring. So far we did not see corruption but we do not feel well with the cluster.

What do you suggest, wait for 12.2.10 ? Roll Back to 12.2.8 ?

Is ist dangerous for our Data to leave the cluster running ?

I am sure we do not have hardware errors and that these errors came with the update to 12.2.9.

Thanks
Christoph
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed.
Post by Konstantin Shalygin
Post by Dietmar Rieder
I just found the new packages on download.ceph.com, is this an official
release?
This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
If you’ve already upgraded and have no issues then you won’t have any
trouble going forward — except perhaps on the next upgrade, if you do it
while the cluster is unhealthy.
I agree that it’s annoying when these issues make it out. We’ve had ongoing
discussions to try and improve the release process so it’s less drawn-out
and to prevent these upgrade issues from making it through testing, but
nobody has resolved it yet. If anybody has experience working with deb
repositories and handling releases, the Ceph upstream could use some
help... ;)
-Greg
Post by Dietmar Rieder
Simon
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Gregory Farnum
2018-11-07 16:15:22 UTC
Permalink
The specific bug you are known at risk for when installing the 12.2.9
packages is http://tracker.ceph.com/issues/36686.

It only triggers when PGs are not active+clean and are running different
minor versions. (Even more specifically, it seems to only show up when
doing backfill from an OSD running new code to an OSD running old code
during the upgrade process.)

If you have encountered other issues, there are no special troubleshooting
steps I'm aware of; follow the standard advice.
-Greg

On Wed, Nov 7, 2018 at 8:08 AM Christoph Adomeit <
Post by Christoph Adomeit
Hello together,
we have upgraded to 12.2.9 because it was in the official repos.
Right after the update and some scrubs we have issues.
pgs: 4036 active+clean
380 active+clean+inconsistent
1/93611534 objects unfound (0.000%)
28 active+clean+inconsistent
1 active+recovery_wait+degraded
Now we stopped repairing because it does not seem to solve the problem and
more and more error messages are occuring. So far we did not see corruption
but we do not feel well with the cluster.
What do you suggest, wait for 12.2.10 ? Roll Back to 12.2.8 ?
Is ist dangerous for our Data to leave the cluster running ?
I am sure we do not have hardware errors and that these errors came with
the update to 12.2.9.
Thanks
Christoph
Post by Gregory Farnum
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed.
Post by Konstantin Shalygin
Post by Dietmar Rieder
I just found the new packages on download.ceph.com, is this an
official
Post by Gregory Farnum
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
release?
This is because 12.2.9 have a several bugs. You should avoid to use
this
Post by Gregory Farnum
Post by Dietmar Rieder
Post by Konstantin Shalygin
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
If you’ve already upgraded and have no issues then you won’t have any
trouble going forward — except perhaps on the next upgrade, if you do it
while the cluster is unhealthy.
I agree that it’s annoying when these issues make it out. We’ve had
ongoing
Post by Gregory Farnum
discussions to try and improve the release process so it’s less drawn-out
and to prevent these upgrade issues from making it through testing, but
nobody has resolved it yet. If anybody has experience working with deb
repositories and handling releases, the Ceph upstream could use some
help... ;)
-Greg
Post by Dietmar Rieder
Simon
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Ashley Merrick
2018-11-07 16:16:05 UTC
Permalink
I am seeing this on the latest mimic on my test cluster aswel.

Every automatic deep-scrub comes back as inconsistent, but doing another
manual scrub comes back as fine and clear each time.

Not sure if related or not..

On Wed, 7 Nov 2018 at 11:57 PM, Christoph Adomeit <
Post by Christoph Adomeit
Hello together,
we have upgraded to 12.2.9 because it was in the official repos.
Right after the update and some scrubs we have issues.
pgs: 4036 active+clean
380 active+clean+inconsistent
1/93611534 objects unfound (0.000%)
28 active+clean+inconsistent
1 active+recovery_wait+degraded
Now we stopped repairing because it does not seem to solve the problem and
more and more error messages are occuring. So far we did not see corruption
but we do not feel well with the cluster.
What do you suggest, wait for 12.2.10 ? Roll Back to 12.2.8 ?
Is ist dangerous for our Data to leave the cluster running ?
I am sure we do not have hardware errors and that these errors came with
the update to 12.2.9.
Thanks
Christoph
Post by Gregory Farnum
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed.
Post by Konstantin Shalygin
Post by Dietmar Rieder
I just found the new packages on download.ceph.com, is this an
official
Post by Gregory Farnum
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
release?
This is because 12.2.9 have a several bugs. You should avoid to use
this
Post by Gregory Farnum
Post by Dietmar Rieder
Post by Konstantin Shalygin
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
If you’ve already upgraded and have no issues then you won’t have any
trouble going forward — except perhaps on the next upgrade, if you do it
while the cluster is unhealthy.
I agree that it’s annoying when these issues make it out. We’ve had
ongoing
Post by Gregory Farnum
discussions to try and improve the release process so it’s less drawn-out
and to prevent these upgrade issues from making it through testing, but
nobody has resolved it yet. If anybody has experience working with deb
repositories and handling releases, the Ceph upstream could use some
help... ;)
-Greg
Post by Dietmar Rieder
Simon
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
David Turner
2018-11-07 17:43:37 UTC
Permalink
My big question is that we've had a few of these releases this year that
are bugged and shouldn't be upgraded to... They don't have any release
notes or announcement and the only time this comes out is when users
finally ask about it weeks later. Why is this not proactively announced to
avoid a problematic release and hopefully prevent people from installing
it? It would be great if there was an actual release notes saying not to
upgrade to this version or something.
Post by Ashley Merrick
I am seeing this on the latest mimic on my test cluster aswel.
Every automatic deep-scrub comes back as inconsistent, but doing another
manual scrub comes back as fine and clear each time.
Not sure if related or not..
On Wed, 7 Nov 2018 at 11:57 PM, Christoph Adomeit <
Post by Christoph Adomeit
Hello together,
we have upgraded to 12.2.9 because it was in the official repos.
Right after the update and some scrubs we have issues.
pgs: 4036 active+clean
380 active+clean+inconsistent
1/93611534 objects unfound (0.000%)
28 active+clean+inconsistent
1 active+recovery_wait+degraded
Now we stopped repairing because it does not seem to solve the problem
and more and more error messages are occuring. So far we did not see
corruption but we do not feel well with the cluster.
What do you suggest, wait for 12.2.10 ? Roll Back to 12.2.8 ?
Is ist dangerous for our Data to leave the cluster running ?
I am sure we do not have hardware errors and that these errors came with
the update to 12.2.9.
Thanks
Christoph
Post by Gregory Farnum
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that
I
Post by Gregory Farnum
Post by Dietmar Rieder
missed.
Post by Konstantin Shalygin
Post by Dietmar Rieder
I just found the new packages on download.ceph.com, is this an
official
Post by Gregory Farnum
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
release?
This is because 12.2.9 have a several bugs. You should avoid to use
this
Post by Gregory Farnum
Post by Dietmar Rieder
Post by Konstantin Shalygin
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
If you’ve already upgraded and have no issues then you won’t have any
trouble going forward — except perhaps on the next upgrade, if you do it
while the cluster is unhealthy.
I agree that it’s annoying when these issues make it out. We’ve had
ongoing
Post by Gregory Farnum
discussions to try and improve the release process so it’s less
drawn-out
Post by Gregory Farnum
and to prevent these upgrade issues from making it through testing, but
nobody has resolved it yet. If anybody has experience working with deb
repositories and handling releases, the Ceph upstream could use some
help... ;)
-Greg
Post by Dietmar Rieder
Simon
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Janne Johansson
2018-11-08 08:02:04 UTC
Permalink
My big question is that we've had a few of these releases this year that are bugged and shouldn't be upgraded to... They don't have any release notes or announcement and the only time this comes out is when users finally ask about it weeks later. Why is this not proactively announced to avoid a problematic release and hopefully prevent people from installing it? It would be great if there was an actual release notes saying not to upgrade to this version or something.
I think the big question is why do these packages end up publicly so
that scripts, updates and anyone not actively trying to hold back get
exposed to them, then we are somehow supposed to notice that the
accompanying release notes are lacking and then from that divinate
that we shouldn't have upgraded into this release at all. This seems
all backwards in most possible ways.

I'm not even upset about releases having bugs, stuff happens, but the
way people are forced into it, then it's somehow your fault for
running ceph-deploy or yum/apt upgrade against official release-repos.
It's almost as if it was meant to push people into slow-moving dists
like Blue^H^H^H^HRedhat with ceph on top.
--
May the most significant bit of your life be positive.
Alfredo Deza
2018-11-08 12:40:58 UTC
Permalink
Post by Janne Johansson
My big question is that we've had a few of these releases this year that are bugged and shouldn't be upgraded to... They don't have any release notes or announcement and the only time this comes out is when users finally ask about it weeks later. Why is this not proactively announced to avoid a problematic release and hopefully prevent people from installing it? It would be great if there was an actual release notes saying not to upgrade to this version or something.
I think the big question is why do these packages end up publicly so
that scripts, updates and anyone not actively trying to hold back get
exposed to them, then we are somehow supposed to notice that the
accompanying release notes are lacking and then from that divinate
that we shouldn't have upgraded into this release at all. This seems
all backwards in most possible ways.
I'm not even upset about releases having bugs, stuff happens, but the
way people are forced into it, then it's somehow your fault for
running ceph-deploy or yum/apt upgrade against official release-repos.
It isn't your fault (or anyone in the community), we don't have a good
system in place for community repos to manage it in a way that would
help when problems like this come up.

We are in a much better place than a few years ago though, when
packages had to be placed manually when creating repos, but more work
is needed to address multi-version support in deb repos
and yanking known bad versions out of the official/latest release location.
Post by Janne Johansson
It's almost as if it was meant to push people into slow-moving dists
like Blue^H^H^H^HRedhat with ceph on top.
--
May the most significant bit of your life be positive.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Christoph Adomeit
2018-11-08 07:57:10 UTC
Permalink
So my question regarding the latest ceph releases still is:

Where do all these scrub errors come from and do we have to worry about ?
Post by Ashley Merrick
I am seeing this on the latest mimic on my test cluster aswel.
Every automatic deep-scrub comes back as inconsistent, but doing another
manual scrub comes back as fine and clear each time.
Not sure if related or not..
On Wed, 7 Nov 2018 at 11:57 PM, Christoph Adomeit <
Post by Christoph Adomeit
Hello together,
we have upgraded to 12.2.9 because it was in the official repos.
Right after the update and some scrubs we have issues.
pgs: 4036 active+clean
380 active+clean+inconsistent
1/93611534 objects unfound (0.000%)
28 active+clean+inconsistent
1 active+recovery_wait+degraded
Now we stopped repairing because it does not seem to solve the problem and
more and more error messages are occuring. So far we did not see corruption
but we do not feel well with the cluster.
What do you suggest, wait for 12.2.10 ? Roll Back to 12.2.8 ?
Is ist dangerous for our Data to leave the cluster running ?
I am sure we do not have hardware errors and that these errors came with
the update to 12.2.9.
Thanks
Christoph
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed.
Post by Konstantin Shalygin
Post by Dietmar Rieder
I just found the new packages on download.ceph.com, is this an
official
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
release?
This is because 12.2.9 have a several bugs. You should avoid to use
this
Post by Dietmar Rieder
Post by Konstantin Shalygin
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
If you’ve already upgraded and have no issues then you won’t have any
trouble going forward — except perhaps on the next upgrade, if you do it
while the cluster is unhealthy.
I agree that it’s annoying when these issues make it out. We’ve had
ongoing
discussions to try and improve the release process so it’s less drawn-out
and to prevent these upgrade issues from making it through testing, but
nobody has resolved it yet. If anybody has experience working with deb
repositories and handling releases, the Ceph upstream could use some
help... ;)
-Greg
Post by Dietmar Rieder
Simon
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Was macht ein Clown im Büro ? Faxen
Christoph Adomeit
GATWORKS GmbH
Reststrauch 191
41199 Moenchengladbach
Sitz: Moenchengladbach
Amtsgericht Moenchengladbach, HRB 6303
Geschaeftsfuehrer:
Christoph Adomeit, Hans Wilhelm Terstappen

***@gatworks.de Internetloesungen vom Feinsten
Fon. +49 2166 9149-32 Fax. +49 2166 9149-10
Kevin Olbrich
2018-11-07 15:57:07 UTC
Permalink
Post by Gregory Farnum
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed.
Post by Konstantin Shalygin
Post by Dietmar Rieder
I just found the new packages on download.ceph.com, is this an
official
Post by Konstantin Shalygin
Post by Dietmar Rieder
release?
This is because 12.2.9 have a several bugs. You should avoid to use
this
Post by Konstantin Shalygin
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
If you’ve already upgraded and have no issues then you won’t have any
trouble going forward — except perhaps on the next upgrade, if you do it
while the cluster is unhealthy.
I agree that it’s annoying when these issues make it out. We’ve had
ongoing discussions to try and improve the release process so it’s less
drawn-out and to prevent these upgrade issues from making it through
testing, but nobody has resolved it yet. If anybody has experience working
with deb repositories and handling releases, the Ceph upstream could use
some help... ;)
-Greg
We solve this problem by hosting two repos. One for staging and QA and one
for production.
Every release gets to staging (for example directly after building a scm
tag).

If QA passed, the stage repo is turned into the prod one.
Using symlinks, it would be possible to switch back if problems occure.
Example: https://incoming.debian.org/

Currently I would be unable to deploy new nodes if I use the official
mirrors as apt is unable to use older versions (which does work on yum/dnf).
Thats why we are implementing "mirror-sync" / rsync with a copy of the repo
and the desired packages until such solution is available.

Kevin
Post by Gregory Farnum
Post by Dietmar Rieder
Simon
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Ken Dreyer
2018-11-07 21:13:06 UTC
Permalink
We solve this problem by hosting two repos. One for staging and QA and one for production.
Every release gets to staging (for example directly after building a scm tag).
If QA passed, the stage repo is turned into the prod one.
Using symlinks, it would be possible to switch back if problems occure.
Example: https://incoming.debian.org/
With the CentOS Storage SIG's cbs.centos.org , we have the ability to
tag builds as "-candidate", "-testing", and "-released". I think that
mechanism could help here, so brave users can run "testing" early
before it goes out to the entire world in "released".

We would have to build out something like that for Ubuntu, maybe
copying around the binaries.

- Ken
Simon Ironside
2018-11-07 16:10:30 UTC
Permalink
Post by Dietmar Rieder
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9
that I missed.
Post by Konstantin Shalygin
Post by Dietmar Rieder
I just found the new packages on download.ceph.com
<http://download.ceph.com>, is this an official
Post by Konstantin Shalygin
Post by Dietmar Rieder
release?
This is because 12.2.9 have a several bugs. You should avoid to
use this
Post by Konstantin Shalygin
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
If you’ve already upgraded and have no issues then you won’t have any
trouble going forward — except perhaps on the next upgrade, if you do
it while the cluster is unhealthy.
Thanks, the upgrade went fine and I've no known issues. The only warning
I have is about too many PGs per OSD which is my fault not ceph's. I
trust that doesn't count as a reason not to proceed to 13.2.2?
Post by Dietmar Rieder
I agree that it’s annoying when these issues make it out. We’ve had
ongoing discussions to try and improve the release process so it’s
less drawn-out and to prevent these upgrade issues from making it
through testing, but nobody has resolved it yet. If anybody has
experience working with deb repositories and handling releases, the
Ceph upstream could use some help... ;)
Totally, I get that this happens from time to time but once a bad
release is known why not just delete the affected packages from the
official repos? That seems to me to be a really easy step to take
especially if release announcements haven't been sent, docs.ceph.com
hasn't been updated yet etc. I reposync --newest-only the RPMs from the
official repos to my own then update my ceph hosts from there which is
how I ended up with 12.2.9.

Simon
Ricardo J. Barberis
2018-11-07 22:38:44 UTC
Permalink
On 07/11/2018 10:59, Konstantin Shalygin wrote: > >> I wonder if there
is any release announcement for ceph 12.2.9 that I > >> missed. I just found
the new packages on download.ceph.com, is this an > >> official release?
Post by Konstantin Shalygin
Post by Dietmar Rieder
Post by Konstantin Shalygin
This is because 12.2.9 have a several bugs. You should avoid to use
this > > release and wait for 12.2.10 > > Argh! What's it doing in the repos
then?? I've just upgraded to it! > What are the bugs? Is there a thread about
them? > > Simon [...]

Content analysis details: (-1.0 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 SURBL_BLOCKED ADMINISTRATOR NOTICE: The query to SURBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: li.org]
-1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
-0.0 BAYES_40 BODY: Bayes spam probability is 20 to 40%
[score: 0.3702]
X-AntiAbuse: This header was added to track abuse,
please include it with any abuse report
X-AntiAbuse: Primary Hostname - c137.dattaweb.com
X-AntiAbuse: Original Domain - lists.ceph.com
X-AntiAbuse: Originator/Caller UID/GID - [502 502] / [502 502]
X-AntiAbuse: Sender Address Domain - palmtx.com.ar
Subject: Re: [ceph-users] ceph 12.2.9 release
X-BeenThere: ceph-***@lists.ceph.com
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <ceph-users-ceph.com>
List-Unsubscribe: <http://lists.ceph.com/options.cgi/ceph-users-ceph.com>,
<mailto:ceph-users-***@lists.ceph.com?subject=unsubscribe>
List-Archive: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/>
List-Post: <mailto:ceph-***@lists.ceph.com>
List-Help: <mailto:ceph-users-***@lists.ceph.com?subject=help>
List-Subscribe: <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>,
<mailto:ceph-users-***@lists.ceph.com?subject=subscribe>
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed. I just found the new packages on download.ceph.com, is this an
official release?
This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
Simon
Is it safe to downgrade from 12.2.9 to 12.2.8?

Or should we just wait for 12.2.10?

Thanks,
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
Neha Ojha
2018-11-07 23:28:03 UTC
Permalink
For those on 12.2.9 -

If you have successfully upgraded to 12.2.9, there is no reason for
you to downgrade, since the bug appears while upgrading to 12.2.9 -
http://tracker.ceph.com/issues/36686. We suggest you to not upgrade to
12.2.10, which reverts the feature that caused this bug. Also, 12.2.10
does not have much in-store except for the revert. We are working on a
clean upgrade path for this feature and will announce it when it is
ready.

For those who haven't upgraded to 12.2.9 -

Please avoid this release and wait for 12.2.10.

More information here:
https://www.spinics.net/lists/ceph-devel/msg43509.html,
https://www.spinics.net/lists/ceph-users/msg49112.html

Again, sorry about the inconvenience and hope this helps!

Thanks,
Neha

On Wed, Nov 7, 2018 at 2:38 PM, Ricardo J. Barberis
Post by Ricardo J. Barberis
Post by Simon Ironside
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed. I just found the new packages on download.ceph.com, is this an
official release?
This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
Simon
Is it safe to downgrade from 12.2.9 to 12.2.8?
Or should we just wait for 12.2.10?
Thanks,
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Ricardo J. Barberis
2018-11-08 20:16:41 UTC
Permalink
Hi Neha, thank you for the info.

I'd like to clarify that we didn't actually upgrade to 12.2.9, we just
installed 4 more OSD servers and those got 12.2.9, so we have a mixture
of 12.2.9 and 12.2.8.

Should we:
- keep as is and wait for 12.2.10+ before proceeding?
- downgrade our newest OSDs from 12.2.9 to 12.2.8?
- upgrade everithing to 12.2.9?


Our current setup (we still have disks to add as OSDs):

# ceph versions
{
"mon": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 5
},
"mgr": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 5
},
"osd": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 75,
"ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) luminous (stable)": 16
},
"mds": {},
"overall": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 85,
"ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) luminous (stable)": 16
}
}
Post by Neha Ojha
For those on 12.2.9 -
If you have successfully upgraded to 12.2.9, there is no reason for
you to downgrade, since the bug appears while upgrading to 12.2.9 -
http://tracker.ceph.com/issues/36686. We suggest you to not upgrade to
12.2.10, which reverts the feature that caused this bug. Also, 12.2.10
does not have much in-store except for the revert. We are working on a
clean upgrade path for this feature and will announce it when it is
ready.
For those who haven't upgraded to 12.2.9 -
Please avoid this release and wait for 12.2.10.
https://www.spinics.net/lists/ceph-devel/msg43509.html,
https://www.spinics.net/lists/ceph-users/msg49112.html
Again, sorry about the inconvenience and hope this helps!
Thanks,
Neha
On Wed, Nov 7, 2018 at 2:38 PM, Ricardo J. Barberis
Post by Ricardo J. Barberis
Post by Simon Ironside
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed. I just found the new packages on download.ceph.com, is this
an official release?
This is because 12.2.9 have a several bugs. You should avoid to use
this release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
Simon
Is it safe to downgrade from 12.2.9 to 12.2.8?
Or should we just wait for 12.2.10?
Thanks,
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
Neha Ojha
2018-11-08 20:37:51 UTC
Permalink
On Thu, Nov 8, 2018 at 12:16 PM, Ricardo J. Barberis
Post by Ricardo J. Barberis
Hi Neha, thank you for the info.
I'd like to clarify that we didn't actually upgrade to 12.2.9, we just
installed 4 more OSD servers and those got 12.2.9, so we have a mixture
of 12.2.9 and 12.2.8.
- keep as is and wait for 12.2.10+ before proceeding?
- downgrade our newest OSDs from 12.2.9 to 12.2.8?
- upgrade everithing to 12.2.9?
I think the best way to proceed is to upgrade the newest OSDs and
restart all OSDs with 12.2.9, or only upgrade when all PGs are
active+clean, if restart is not an option.
Post by Ricardo J. Barberis
# ceph versions
{
"mon": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 5
},
"mgr": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 5
},
"osd": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 75,
"ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) luminous (stable)": 16
},
"mds": {},
"overall": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 85,
"ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) luminous (stable)": 16
}
}
Post by Neha Ojha
For those on 12.2.9 -
If you have successfully upgraded to 12.2.9, there is no reason for
you to downgrade, since the bug appears while upgrading to 12.2.9 -
http://tracker.ceph.com/issues/36686. We suggest you to not upgrade to
12.2.10, which reverts the feature that caused this bug. Also, 12.2.10
does not have much in-store except for the revert. We are working on a
clean upgrade path for this feature and will announce it when it is
ready.
For those who haven't upgraded to 12.2.9 -
Please avoid this release and wait for 12.2.10.
https://www.spinics.net/lists/ceph-devel/msg43509.html,
https://www.spinics.net/lists/ceph-users/msg49112.html
Again, sorry about the inconvenience and hope this helps!
Thanks,
Neha
On Wed, Nov 7, 2018 at 2:38 PM, Ricardo J. Barberis
Post by Ricardo J. Barberis
Post by Simon Ironside
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed. I just found the new packages on download.ceph.com, is this
an official release?
This is because 12.2.9 have a several bugs. You should avoid to use
this release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
Simon
Is it safe to downgrade from 12.2.9 to 12.2.8?
Or should we just wait for 12.2.10?
Thanks,
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
Graham Allan
2018-11-08 23:18:02 UTC
Permalink
Thanks for the extra info, but I find the question more nuanced than that.

For example in my case I ended up with 12.2.9 on my last handful of
newly-installed servers (and this is simply using yum rather than
explicitly ceph-deploy).

We are replacing hardware nodes so the cluster is actively backfilling
across nodes, which fulfills Greg's criterion of "(the bug) only
triggers when PGs are not active+clean". Fortunately we are mostly
backfilling from older to newer minor versions (12.2.8 to 12.2.9), so
the problem may not manifest itself, but I'm not going to be in a
position to have all pgs active+clean until the backfills are complete.
Perhaps at that point it might be safe to upgrade the remaining osds to
the same 12.2.9? The advice to also not upgrade to 12.9.10 is a bit
disturbing since it sounds like the feature revert might make it
incompatible with what we have.

Graham
Post by Neha Ojha
For those on 12.2.9 -
If you have successfully upgraded to 12.2.9, there is no reason for
you to downgrade, since the bug appears while upgrading to 12.2.9 -
http://tracker.ceph.com/issues/36686. We suggest you to not upgrade to
12.2.10, which reverts the feature that caused this bug. Also, 12.2.10
does not have much in-store except for the revert. We are working on a
clean upgrade path for this feature and will announce it when it is
ready.
For those who haven't upgraded to 12.2.9 -
Please avoid this release and wait for 12.2.10.
https://www.spinics.net/lists/ceph-devel/msg43509.html,
https://www.spinics.net/lists/ceph-users/msg49112.html
Again, sorry about the inconvenience and hope this helps!
Thanks,
Neha
On Wed, Nov 7, 2018 at 2:38 PM, Ricardo J. Barberis
Post by Ricardo J. Barberis
Post by Simon Ironside
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed. I just found the new packages on download.ceph.com, is this an
official release?
This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10
Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?
Simon
Is it safe to downgrade from 12.2.9 to 12.2.8?
Or should we just wait for 12.2.10?
Thanks,
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Graham Allan
Minnesota Supercomputing Institute - ***@umn.edu
Matthew Vernon
2018-11-23 14:46:28 UTC
Permalink
Post by Neha Ojha
For those who haven't upgraded to 12.2.9 -
Please avoid this release and wait for 12.2.10.
Any idea when 12.2.10 is going to be here, please?

Regards,

Matthew
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Matthew Vernon
2018-11-07 14:05:27 UTC
Permalink
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I missed.
I just found the new packages on download.ceph.com, is this an official
release?
This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10
It seems that maybe something isn't quite right in the release
infrastructure, then? The 12.2.8 packages are still available, but e.g.
debian-luminous's Packages file is pointing to the 12.2.9 (broken) packages.

Could the Debian/Ubuntu repos only have their releases updated (as
opposed to what's in the pool) for safe/official releases? It's one
thing letting people find pre-release things if they go looking, but
ISTM that arranging that a mis-timed apt-get update/upgrade might
install known-broken packages is ... unfortunate.

Regards,

Matthew
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Ricardo J. Barberis
2018-11-07 22:20:12 UTC
Permalink
On 07/11/2018 10:59, Konstantin Shalygin wrote: > >> I wonder if there
is any release announcement for ceph 12.2.9 that I missed. > >> I [...]

Content analysis details: (-1.0 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 SURBL_BLOCKED ADMINISTRATOR NOTICE: The query to SURBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: gtdinternet.com]
-1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
-0.0 BAYES_20 BODY: Bayes spam probability is 5 to 20%
[score: 0.1964]
X-AntiAbuse: This header was added to track abuse,
please include it with any abuse report
X-AntiAbuse: Primary Hostname - c137.dattaweb.com
X-AntiAbuse: Original Domain - lists.ceph.com
X-AntiAbuse: Originator/Caller UID/GID - [502 502] / [502 502]
X-AntiAbuse: Sender Address Domain - palmtx.com.ar
Subject: Re: [ceph-users] ceph 12.2.9 release
X-BeenThere: ceph-***@lists.ceph.com
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <ceph-users-ceph.com>
List-Unsubscribe: <http://lists.ceph.com/options.cgi/ceph-users-ceph.com>,
<mailto:ceph-users-***@lists.ceph.com?subject=unsubscribe>
List-Archive: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/>
List-Post: <mailto:ceph-***@lists.ceph.com>
List-Help: <mailto:ceph-users-***@lists.ceph.com?subject=help>
List-Subscribe: <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>,
<mailto:ceph-users-***@lists.ceph.com?subject=subscribe>
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I missed.
I just found the new packages on download.ceph.com, is this an official
release?
This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10
It seems that maybe something isn't quite right in the release
infrastructure, then? The 12.2.8 packages are still available, but e.g.
debian-luminous's Packages file is pointing to the 12.2.9 (broken) packages.
Could the Debian/Ubuntu repos only have their releases updated (as
opposed to what's in the pool) for safe/official releases? It's one
thing letting people find pre-release things if they go looking, but
ISTM that arranging that a mis-timed apt-get update/upgrade might
install known-broken packages is ... unfortunate.
Regards,
Matthew
Not only Debian/Ubuntu, CentOS also:

# LANG=C yum list ceph-common
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
* elrepo: repos.mia.lax-noc.com
* epel: epel.gtdinternet.com
8 packages excluded due to repository priority protections
Installed Packages
ceph-common.x86_64 2:12.2.8-0.el7 @Ceph
Available Packages
ceph-common.x86_64 2:12.2.9-0.el7 Ceph
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
Dietmar Rieder
2018-11-07 14:37:54 UTC
Permalink
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I missed.
I just found the new packages on download.ceph.com, is this an official
release?
This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10
Thanks a lot!

~Dietmar
--
_________________________________________
D i e t m a r R i e d e r, Mag.Dr.
Innsbruck Medical University
Biocenter - Division for Bioinformatics
Email: ***@i-med.ac.at
Web: http://www.icbi.at
Marc Roos
2018-11-07 14:16:36 UTC
Permalink
I don't see the problem. I am installing only the ceph updates when
others have done this and are running several weeks without problems. I
have noticed this 12.2.9 availability also, did not see any release
notes, so why install it? Especially with recent issues of other
releases.

That being said, this ceph is often a major part of a production
environment. And I am surprised how easily buggy releases are finding
their way to the public. We had this 12.2.5 now 12.2.9 there was also
something with upgrading to mimic. I would not expect this from a LTS
version (or a redhat product).





-----Original Message-----
From: Matthew Vernon [mailto:***@sanger.ac.uk]
Sent: woensdag 7 november 2018 15:05
To: Konstantin Shalygin; ceph-***@lists.ceph.com
Subject: Re: [ceph-users] ceph 12.2.9 release
Post by Konstantin Shalygin
Post by Dietmar Rieder
I wonder if there is any release announcement for ceph 12.2.9 that I
missed.
Post by Konstantin Shalygin
Post by Dietmar Rieder
I just found the new packages on download.ceph.com, is this an
official release?
This is because 12.2.9 have a several bugs. You should avoid to use
this release and wait for 12.2.10
It seems that maybe something isn't quite right in the release
infrastructure, then? The 12.2.8 packages are still available, but e.g.
debian-luminous's Packages file is pointing to the 12.2.9 (broken)
packages.

Could the Debian/Ubuntu repos only have their releases updated (as
opposed to what's in the pool) for safe/official releases? It's one
thing letting people find pre-release things if they go looking, but
ISTM that arranging that a mis-timed apt-get update/upgrade might
install known-broken packages is ... unfortunate.

Regards,

Matthew


--
The Wellcome Sanger Institute is operated by Genome Research Limited,
a charity registered in England with number 1021457 and a company
registered in England with number 2742969, whose registered office is
215 Euston Road, London, NW1 2BE.
Matthew Vernon
2018-11-07 14:28:30 UTC
Permalink
Post by Marc Roos
I don't see the problem. I am installing only the ceph updates when
others have done this and are running several weeks without problems. I
have noticed this 12.2.9 availability also, did not see any release
notes, so why install it? Especially with recent issues of other
releases.
Relevantly, if you want to upgrade to Luminous in many of the obvious
ways, you'll end up with 12.2.9.

Regards,

Matthew
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Ricardo J. Barberis
2018-11-07 22:22:36 UTC
Permalink
On 07/11/2018 14:16, Marc Roos wrote: > > > > > > I don't see the problem.
I am installing only the ceph updates when > > others have done this and
are running several weeks without problems. I > > have noticed this 12.2.9
availability also, did not see any release > > notes, so why install it?
Especially with recent issues of other > > releases. > > Relevantly, if you
want to upgrade to Luminous in many of the obvious > ways, you'll end up
with 12.2.9. > > Regards, > > Matthew [...]

Content analysis details: (-1.0 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
0.0 SURBL_BLOCKED ADMINISTRATOR NOTICE: The query to SURBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: li.org]
-1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
X-AntiAbuse: This header was added to track abuse,
please include it with any abuse report
X-AntiAbuse: Primary Hostname - c137.dattaweb.com
X-AntiAbuse: Original Domain - lists.ceph.com
X-AntiAbuse: Originator/Caller UID/GID - [502 502] / [502 502]
X-AntiAbuse: Sender Address Domain - palmtx.com.ar
Subject: Re: [ceph-users] ceph 12.2.9 release
X-BeenThere: ceph-***@lists.ceph.com
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: <ceph-users-ceph.com>
List-Unsubscribe: <http://lists.ceph.com/options.cgi/ceph-users-ceph.com>,
<mailto:ceph-users-***@lists.ceph.com?subject=unsubscribe>
List-Archive: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/>
List-Post: <mailto:ceph-***@lists.ceph.com>
List-Help: <mailto:ceph-users-***@lists.ceph.com?subject=help>
List-Subscribe: <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>,
<mailto:ceph-users-***@lists.ceph.com?subject=subscribe>
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Post by Marc Roos
I don't see the problem. I am installing only the ceph updates when
others have done this and are running several weeks without problems. I
have noticed this 12.2.9 availability also, did not see any release
notes, so why install it? Especially with recent issues of other
releases.
Relevantly, if you want to upgrade to Luminous in many of the obvious
ways, you'll end up with 12.2.9.
Regards,
Matthew
Also relevant: if you use ceph-deploy like I do con CentOS 7, it installs the latest version available, so I inadvertedly ended up with 12.2.9 on my last four servers.

Thanks,
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
Valmar Kuristik
2018-11-08 08:06:49 UTC
Permalink
I'll second that.

We are in progress of upgrading after just receiving new hardware, and
it looks like right now, digging information on what to do and exactly
how will take literally hundreds of times more time and effort than the
upgrade itself, once you know.
Post by Janne Johansson
My big question is that we've had a few of these releases this year that are bugged and shouldn't be upgraded to... They don't have any release notes or announcement and the only time this comes out is when users finally ask about it weeks later. Why is this not proactively announced to avoid a problematic release and hopefully prevent people from installing it? It would be great if there was an actual release notes saying not to upgrade to this version or something.
I think the big question is why do these packages end up publicly so
that scripts, updates and anyone not actively trying to hold back get
exposed to them, then we are somehow supposed to notice that the
accompanying release notes are lacking and then from that divinate
that we shouldn't have upgraded into this release at all. This seems
all backwards in most possible ways.
I'm not even upset about releases having bugs, stuff happens, but the
way people are forced into it, then it's somehow your fault for
running ceph-deploy or yum/apt upgrade against official release-repos.
It's almost as if it was meant to push people into slow-moving dists
like Blue^H^H^H^HRedhat with ceph on top.
Marc Roos
2018-11-08 09:17:43 UTC
Permalink
And that is why I don't like ceph-deploy. Unless you have maybe hundreds
of disks, I don’t see why you cannot install it "manually".




-----Original Message-----
From: Ricardo J. Barberis [mailto:***@palmtx.com.ar]
Sent: woensdag 7 november 2018 23:23
To: ceph-***@lists.ceph.com
Subject: Re: [ceph-users] ceph 12.2.9 release
Post by Matthew Vernon
Post by Marc Roos
I don't see the problem. I am installing only the ceph updates when
others have done this and are running several weeks without
problems. I have noticed this 12.2.9 availability also, did not see
any release notes, so why install it? Especially with recent issues
of other releases.
Relevantly, if you want to upgrade to Luminous in many of the obvious
ways, you'll end up with 12.2.9.
Regards,
Matthew
Also relevant: if you use ceph-deploy like I do con CentOS 7, it
installs the latest version available, so I inadvertedly ended up with
12.2.9 on my last four servers.

Thanks,
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/ Usuario LFS Nº 5121:
http://www.linuxfromscratch.org/ Senior SysAdmin / IT Architect -
www.DonWeb.com _______________________________________________
ceph-users mailing list
ceph-***@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Simon Ironside
2018-11-08 09:28:32 UTC
Permalink
Post by Marc Roos
And that is why I don't like ceph-deploy. Unless you have maybe hundreds
of disks, I don’t see why you cannot install it "manually".
Also relevant: if you use ceph-deploy like I do con CentOS 7, it
installs the latest version available, so I inadvertedly ended up with
12.2.9 on my last four servers.
I use ceph-deploy (on RHEL 7) but with my own repos. I still fell into
the 12.2.9 trap but that's because that's the package version I'd
mirrored. If I'd downloaded 12.2.8 instead it would've been fine with
ceph-deploy.

Simon
Matthew Vernon
2018-11-08 09:35:32 UTC
Permalink
Post by Marc Roos
And that is why I don't like ceph-deploy. Unless you have maybe hundreds
of disks, I don’t see why you cannot install it "manually".
...as the recent ceph survey showed, plenty of people have hundreds of
disks! Ceph is meant to be operated at scale, which is why many admins
will have automation (ceph-ansible, etc.) in place.

[our test clusters are 180 OSDs...]

Regards,

Matthew
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Martin Verges
2018-11-08 10:06:48 UTC
Permalink
Sorry to say this, but that's why there's the croit management
interface (free community edition feature).
You don't have to worry about problems that are absolutely critical
for reliable and stable operation. It doesn't matter if you run a
cluster with 10 or 1000 hard disks, it just has to run!

On the 12.11. on the Ceph Day in Berlin I can give you information
directly about it.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: ***@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Post by Matthew Vernon
Post by Marc Roos
And that is why I don't like ceph-deploy. Unless you have maybe hundreds
of disks, I don’t see why you cannot install it "manually".
...as the recent ceph survey showed, plenty of people have hundreds of
disks! Ceph is meant to be operated at scale, which is why many admins will
have automation (ceph-ansible, etc.) in place.
[our test clusters are 180 OSDs...]
Regards,
Matthew
--
The Wellcome Sanger Institute is operated by Genome Research Limited, a
charity registered in England with number 1021457 and a company registered
in England with number 2742969, whose registered office is 215 Euston Road,
London, NW1 2BE. _______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Martin Verges
2018-11-08 11:23:01 UTC
Permalink
This post might be inappropriate. Click to display it.
Ricardo J. Barberis
2018-11-08 20:33:14 UTC
Permalink
Post by Marc Roos
And that is why I don't like ceph-deploy. Unless you have maybe hundreds
of disks, I don’t see why you cannot install it "manually".
We do have another cluster with 600+ disks, this one has 91 so far.

We actually started using ceph-deploy on this cluster to simplify our old
method of installation, which would have done the same thing, actually, since
it also uses the official repos.

We might have noticed the different version with our old method, but I can't
be sure.

Cheers,
Post by Marc Roos
-----Original Message-----
Sent: woensdag 7 november 2018 23:23
Subject: Re: [ceph-users] ceph 12.2.9 release
Post by Matthew Vernon
Post by Marc Roos
I don't see the problem. I am installing only the ceph updates when
others have done this and are running several weeks without
problems. I have noticed this 12.2.9 availability also, did not see
any release notes, so why install it? Especially with recent issues
of other releases.
Relevantly, if you want to upgrade to Luminous in many of the obvious
ways, you'll end up with 12.2.9.
Regards,
Matthew
Also relevant: if you use ceph-deploy like I do con CentOS 7, it
installs the latest version available, so I inadvertedly ended up with
12.2.9 on my last four servers.
Thanks,
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
Marc Roos
2018-11-08 10:06:06 UTC
Permalink
I know ceph is meant to operate at scale, that’s why we are all here.
But if you have a 180 disk cluster, you have 6-9 nodes, is nothing when
you add a node. I would just do the manual install and especially with a
production environment, considering all the 'little' bugs surfacing
here. I doubt if ceph-deploy is even much faster.




-----Original Message-----
From: Matthew Vernon [mailto:***@sanger.ac.uk]
Sent: donderdag 8 november 2018 10:36
To: ceph-***@lists.ceph.com
Cc: Marc Roos
Subject: Re: [ceph-users] ceph 12.2.9 release
Post by Marc Roos
And that is why I don't like ceph-deploy. Unless you have maybe
hundreds of disks, I don’t see why you cannot install it "manually".
...as the recent ceph survey showed, plenty of people have hundreds of
disks! Ceph is meant to be operated at scale, which is why many admins
will have automation (ceph-ansible, etc.) in place.

[our test clusters are 180 OSDs...]

Regards,

Matthew


--
The Wellcome Sanger Institute is operated by Genome Research Limited,
a charity registered in England with number 1021457 and a company
registered in England with number 2742969, whose registered office is
215 Euston Road, London, NW1 2BE.
Marc Roos
2018-11-08 10:23:51 UTC
Permalink
Hmmmm interesting maybe,

- You can use this separate from the commandline?
- And if I modify something from the commandline, these changes are
visible in the webinterface?
- I can easily remove/add this webinterface? I mean sometimes you have
these tools that just customize the whole environment, that it is
difficult to revert it.



-----Original Message-----
From: Martin Verges [mailto:***@croit.io]
Sent: donderdag 8 november 2018 11:07
To: Matthew Vernon
Cc: ceph-users
Subject: Re: [ceph-users] ceph 12.2.9 release

Sorry to say this, but that's why there's the croit management interface
(free community edition feature).
You don't have to worry about problems that are absolutely critical for
reliable and stable operation. It doesn't matter if you run a cluster
with 10 or 1000 hard disks, it just has to run!

On the 12.11. on the Ceph Day in Berlin I can give you information
directly about it.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: ***@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht
Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx
Post by Matthew Vernon
Post by Marc Roos
And that is why I don't like ceph-deploy. Unless you have maybe
hundreds of disks, I don’t see why you cannot install it "manually".
...as the recent ceph survey showed, plenty of people have hundreds of
disks! Ceph is meant to be operated at scale, which is why many admins
will have automation (ceph-ansible, etc.) in place.
[our test clusters are 180 OSDs...]
Regards,
Matthew
--
The Wellcome Sanger Institute is operated by Genome Research Limited,
a charity registered in England with number 1021457 and a company
registered in England with number 2742969, whose registered office is
215 Euston Road, London, NW1 2BE.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-***@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Loading...