[ceph-users] can we drop support of centos/rhel 7.4?

Discussion:

kefu chai

2018-09-14 02:48:23 UTC

hi ceph-{maintainers,users,developers},

recently, i ran into an issue[0] which popped up when we build Ceph on
centos 7.5, but test it on centos 7.4. as we know, the gperftools-libs
package provides the tcmalloc allocator shared library, but centos 7.4
and centos 7.5 ship different version of gperftools-{devel,libs}. the
former ships 2.4, and the latter 2.6.1.

the crux is that the tcmalloc in gperftools 2.6.1 implements more
standard compliant C++ APIs, which were missing in gperftools 2.4.
that's why we have failures like:

ceph-osd: symbol lookup error: ceph-osd: undefined symbol: _ZdaPvm

when testing Ceph on centos 7.4.

my question is: is it okay to drop the support of centos/rhel 7.4? so
we will solely build and test the supported Ceph releases (luminous,
mimic) on 7.5 ?

thanks,

--
[0] http://tracker.ceph.com/issues/35969

--
Regards
Kefu Chai

John Spray

2018-09-14 09:12:30 UTC

Permalink

Post by kefu chai
hi ceph-{maintainers,users,developers},
recently, i ran into an issue[0] which popped up when we build Ceph on
centos 7.5, but test it on centos 7.4. as we know, the gperftools-libs
package provides the tcmalloc allocator shared library, but centos 7.4
and centos 7.5 ship different version of gperftools-{devel,libs}. the
former ships 2.4, and the latter 2.6.1.
the crux is that the tcmalloc in gperftools 2.6.1 implements more
standard compliant C++ APIs, which were missing in gperftools 2.4.
ceph-osd: symbol lookup error: ceph-osd: undefined symbol: _ZdaPvm
when testing Ceph on centos 7.4.
my question is: is it okay to drop the support of centos/rhel 7.4? so
we will solely build and test the supported Ceph releases (luminous,
mimic) on 7.5 ?

My preference would be to target the latest minor release (i.e. 7.5)
of the major release. We don't test on CentOS 7.1, 7.2 etc, so I
don't think we need to give 7.4 any special treatment.

John

Post by kefu chai
thanks,
--
[0] http://tracker.ceph.com/issues/35969
--
Regards
Kefu Chai

David Turner

2018-09-14 13:25:56 UTC

Permalink

Release dates
RHEL 7.4 - July 2017
Luminous 12.2.0 - August 2017
CentOS 7.4 - September 2017
RHEL 7.5 - April 2018
CentOS 7.5 - May 2018
Mimic 13.2.0 - June 2018

In the world of sysadmins it takes time to let new releases/OS's simmer
before beginning to test them let alone upgrading to them. It is not
possible to tell all companies that use CentOS that we have to move to a
new OS upgrade 5 months after it is released. We are still testing if
CentOS 7.5 works in our infrastructure in general let alone being up and
running on it. The kernel upgrades alone are a big change now to mention
the obvious package version changes. We don't even have the OK to install
it in staging. Once we do, and we have the time to start testing it,
...among our other tasks, we can start regression testing our use case in
staging before thinking about upgrading prod.

That time frame isn't really so bad if everything is working great for
ceph, but what if we're waiting on 12.2.9 and 13.2.2 for a bugfix that's
giving us grief? Now we are not only dealing with the bugs, but now we have
to regression test an OS upgrade, update our package management, and make
sure our new deployments will have this version... And then we can start
regression testing the new release that hopefully fixes the bugs we're
dealing with...

What about backporting the API standards to the CentOS 7.4 version of
gperftools-libs?

I've noticed little package issues like this in the past, but assumed that
was because most development was done on Ubuntu instead of RHEL. We had to
set our repos to a newer version of CentOS than we were running or willing
to upgrade to just for a single package we needed. If y'all are really
thinking of only supporting/testing the latest dot release of the latest
major version of RHEL, then you might have just given me the fuel to be
able to finally convince my company into allowing us to be the first
application in 9,000 servers to not run CentOS. I've been trying to get
them to allow it for a while because of the previous package issues, but I
hadn't put much effort into it because I thought/hoped those problems might
be behind us...

Do y'all not test ceph on 7.3 right now? This email thread really might be
enough to get us off of CentOS for Ceph.

Post by John Spray

My preference would be to target the latest minor release (i.e. 7.5)
of the major release. We don't test on CentOS 7.1, 7.2 etc, so I
don't think we need to give 7.4 any special treatment.
John

Post by kefu chai
thanks,
--
[0] http://tracker.ceph.com/issues/35969
--
Regards
Kefu Chai

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

David Turner

2018-09-14 13:29:59 UTC

Permalink

It's odd to me because this feels like the opposite direction of the rest
of Ceph. Making management and operating Ceph simpler and easier. Requiring
fast OS upgrades on dot releases of Ceph versions is not that direction at
all.

Post by David Turner
Release dates
RHEL 7.4 - July 2017
Luminous 12.2.0 - August 2017
CentOS 7.4 - September 2017
RHEL 7.5 - April 2018
CentOS 7.5 - May 2018
Mimic 13.2.0 - June 2018
In the world of sysadmins it takes time to let new releases/OS's simmer
before beginning to test them let alone upgrading to them. It is not
possible to tell all companies that use CentOS that we have to move to a
new OS upgrade 5 months after it is released. We are still testing if
CentOS 7.5 works in our infrastructure in general let alone being up and
running on it. The kernel upgrades alone are a big change now to mention
the obvious package version changes. We don't even have the OK to install
it in staging. Once we do, and we have the time to start testing it,
...among our other tasks, we can start regression testing our use case in
staging before thinking about upgrading prod.
That time frame isn't really so bad if everything is working great for
ceph, but what if we're waiting on 12.2.9 and 13.2.2 for a bugfix that's
giving us grief? Now we are not only dealing with the bugs, but now we have
to regression test an OS upgrade, update our package management, and make
sure our new deployments will have this version... And then we can start
regression testing the new release that hopefully fixes the bugs we're
dealing with...
What about backporting the API standards to the CentOS 7.4 version of
gperftools-libs?
I've noticed little package issues like this in the past, but assumed that
was because most development was done on Ubuntu instead of RHEL. We had to
set our repos to a newer version of CentOS than we were running or willing
to upgrade to just for a single package we needed. If y'all are really
thinking of only supporting/testing the latest dot release of the latest
major version of RHEL, then you might have just given me the fuel to be
able to finally convince my company into allowing us to be the first
application in 9,000 servers to not run CentOS. I've been trying to get
them to allow it for a while because of the previous package issues, but I
hadn't put much effort into it because I thought/hoped those problems might
be behind us...
Do y'all not test ceph on 7.3 right now? This email thread really might be
enough to get us off of CentOS for Ceph.

Post by John Spray

My preference would be to target the latest minor release (i.e. 7.5)
of the major release. We don't test on CentOS 7.1, 7.2 etc, so I
don't think we need to give 7.4 any special treatment.
John

Post by kefu chai
thanks,
--
[0] http://tracker.ceph.com/issues/35969
--
Regards
Kefu Chai

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

John Spray

2018-09-14 14:07:38 UTC

Permalink

From the comments on http://tracker.ceph.com/issues/35969, I think
Kefu is still proposing to work around this in Ceph's build (i.e. to
fix this so that our packages still work on centOS 7.4).

Ideally, distros maintain ABI compatibility such that the packages we
test on one minor version will also work on another -- that hasn't
happened in the 7.4->7.5 transition for gperftools. That's annoying,
but it's also kind of a special case, and we hopefully won't encounter
issues like this particularly frequently within a major release. If
we move away from using 7.4 for the main build/test cycle, that
doesn't mean we wouldn't also be accepting fixes to keep it working on
older releases (although it of course relies on someone noticing
if/when it breaks).

Post by David Turner
What about backporting the API standards to the CentOS 7.4 version of gperftools-libs?
I've noticed little package issues like this in the past, but assumed that was because most development was done on Ubuntu instead of RHEL. We had to set our repos to a newer version of CentOS than we were running or willing to upgrade to just for a single package we needed. If y'all are really thinking of only supporting/testing the latest dot release of the latest major version of RHEL, then you might have just given me the fuel to be able to finally convince my company into allowing us to be the first application in 9,000 servers to not run CentOS. I've been trying to get them to allow it for a while because of the previous package issues, but I hadn't put much effort into it because I thought/hoped those problems might be behind us...
Do y'all not test ceph on 7.3 right now? This email thread really might be enough to get us off of CentOS for Ceph.

There is a set of permutations in qa/distros, used in
qa/suites/buildpackages/ -- I'm not sure exactly what's run when
though (possibly some only at release time?), perhaps someone more
familiar with exactly what tests are run before a release could chime
in.

John

Post by David Turner

Post by John Spray

My preference would be to target the latest minor release (i.e. 7.5)
of the major release. We don't test on CentOS 7.1, 7.2 etc, so I
don't think we need to give 7.4 any special treatment.
John

Post by kefu chai
thanks,
--
[0] http://tracker.ceph.com/issues/35969
--
Regards
Kefu Chai

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

kefu chai

2018-09-14 14:27:44 UTC

Permalink

Post by John Spray

Yeah, David, I hear you =) i just wanted to explorer all options
before working on a workaround on Ceph side.

Post by John Spray
From the comments on http://tracker.ceph.com/issues/35969, I think
Kefu is still proposing to work around this in Ceph's build (i.e. to
fix this so that our packages still work on centOS 7.4).

True, that's the plan A prime. =)

Post by John Spray
Ideally, distros maintain ABI compatibility such that the packages we
test on one minor version will also work on another -- that hasn't
happened in the 7.4->7.5 transition for gperftools. That's annoying,
but it's also kind of a special case, and we hopefully won't encounter
issues like this particularly frequently within a major release. If
we move away from using 7.4 for the main build/test cycle, that
doesn't mean we wouldn't also be accepting fixes to keep it working on
older releases (although it of course relies on someone noticing
if/when it breaks).

Post by David Turner
What about backporting the API standards to the CentOS 7.4 version of gperftools-libs?

we was trying to asking the downstream maintainers to update
gperftools-libs in latest CentOS. i guess that's why we have 2.6 in
CentOS 7.5. =D anyway, no worries, i will fix this issue on Ceph as i
proposed in http://tracker.ceph.com/issues/35969 .

Post by John Spray

Post by David Turner
I've noticed little package issues like this in the past, but assumed that was because most development was done on Ubuntu instead of RHEL. We had to set our repos to a newer version of CentOS than we were running or willing to upgrade to just for a single package we needed. If y'all are really thinking of only supporting/testing the latest dot release of the latest major version of RHEL, then you might have just given me the fuel to be able to finally convince my company into allowing us to be the first application in 9,000 servers to not run CentOS. I've been trying to get them to allow it for a while because of the previous package issues, but I hadn't put much effort into it because I thought/hoped those problems might be behind us...
Do y'all not test ceph on 7.3 right now? This email thread really might be enough to get us off of CentOS for Ceph.

by looking at the qa/ directory, i found that supported-random-distro%
a very popular facet:

- master (nautilus in future):
https://github.com/ceph/ceph/tree/master/qa/distros/supported-random-distro%24
-- centos 7.4, rhel 7.5, ubuntu {18.04, 16.04}
- mimic: https://github.com/ceph/ceph/tree/mimic/qa/distros/supported-random-distro%24
-- centos 7.4, rhel 7.5, ubuntu {18.04, 16.04}
- luminous: https://github.com/ceph/ceph/tree/luminous/qa/distros/supported
-- centos 7.4, , ubuntu {14.04, 16.04}

Post by John Spray
John

Post by David Turner

Post by John Spray

My preference would be to target the latest minor release (i.e. 7.5)
of the major release. We don't test on CentOS 7.1, 7.2 etc, so I
don't think we need to give 7.4 any special treatment.
John

Post by kefu chai
thanks,
--
[0] http://tracker.ceph.com/issues/35969
--
Regards
Kefu Chai

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--
Regards
Kefu Chai

Marc Roos

2018-09-14 13:35:27 UTC

Permalink

I agree. I was on centos7.4 and updated to I think luminous 12.2.7, and
had something not working related to some python dependancy. This was
resolved by upgrading to centos7.5

-----Original Message-----
From: David Turner [mailto:***@gmail.com]
Sent: vrijdag 14 september 2018 15:30
To: John Spray
Cc: Ceph Development; ceph-***@lists.ceph.com;
ceph-***@ceph.com
Subject: Re: [ceph-users] can we drop support of centos/rhel 7.4?

It's odd to me because this feels like the opposite direction of the
rest of Ceph. Making management and operating Ceph simpler and easier.
Requiring fast OS upgrades on dot releases of Ceph versions is not that
direction at all.

On Fri, Sep 14, 2018, 9:25 AM David Turner <***@gmail.com>
wrote:

Release dates
RHEL 7.4 - July 2017
Luminous 12.2.0 - August 2017
CentOS 7.4 - September 2017
RHEL 7.5 - April 2018
CentOS 7.5 - May 2018
Mimic 13.2.0 - June 2018

In the world of sysadmins it takes time to let new releases/OS's
simmer before beginning to test them let alone upgrading to them. It is
not possible to tell all companies that use CentOS that we have to move
to a new OS upgrade 5 months after it is released. We are still testing
if CentOS 7.5 works in our infrastructure in general let alone being up
and running on it. The kernel upgrades alone are a big change now to
mention the obvious package version changes. We don't even have the OK
to install it in staging. Once we do, and we have the time to start
testing it, ...among our other tasks, we can start regression testing
our use case in staging before thinking about upgrading prod.

That time frame isn't really so bad if everything is working great
for ceph, but what if we're waiting on 12.2.9 and 13.2.2 for a bugfix
that's giving us grief? Now we are not only dealing with the bugs, but
now we have to regression test an OS upgrade, update our package
management, and make sure our new deployments will have this version...
And then we can start regression testing the new release that hopefully
fixes the bugs we're dealing with...

What about backporting the API standards to the CentOS 7.4 version
of gperftools-libs?

I've noticed little package issues like this in the past, but
assumed that was because most development was done on Ubuntu instead of
RHEL. We had to set our repos to a newer version of CentOS than we were
running or willing to upgrade to just for a single package we needed. If
y'all are really thinking of only supporting/testing the latest dot
release of the latest major version of RHEL, then you might have just
given me the fuel to be able to finally convince my company into
allowing us to be the first application in 9,000 servers to not run
CentOS. I've been trying to get them to allow it for a while because of
the previous package issues, but I hadn't put much effort into it
because I thought/hoped those problems might be behind us...

Do y'all not test ceph on 7.3 right now? This email thread really
might be enough to get us off of CentOS for Ceph.

Post by kefu chai
hi ceph-{maintainers,users,developers},
recently, i ran into an issue[0] which popped up when we

build Ceph on

Post by kefu chai
centos 7.5, but test it on centos 7.4. as we know, the

gperftools-libs

Post by kefu chai
package provides the tcmalloc allocator shared library, but

centos 7.4

Post by kefu chai
and centos 7.5 ship different version of

gperftools-{devel,libs}. the

Post by kefu chai
former ships 2.4, and the latter 2.6.1.
the crux is that the tcmalloc in gperftools 2.6.1 implements

Post by kefu chai
standard compliant C++ APIs, which were missing in

gperftools 2.4.
_ZdaPvm

Post by kefu chai
when testing Ceph on centos 7.4.
my question is: is it okay to drop the support of

centos/rhel 7.4? so

Post by kefu chai
we will solely build and test the supported Ceph releases

(luminous,

Post by kefu chai
mimic) on 7.5 ?

My preference would be to target the latest minor release
(i.e. 7.5)
of the major release. We don't test on CentOS 7.1, 7.2 etc,
so I
don't think we need to give 7.4 any special treatment.

John

Post by kefu chai
thanks,
--
[0] http://tracker.ceph.com/issues/35969
--
Regards
Kefu Chai

_______________________________________________
ceph-users mailing list
ceph-***@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Ken Dreyer

2018-09-24 15:39:17 UTC

Permalink

Post by kefu chai
my question is: is it okay to drop the support of centos/rhel 7.4? so
we will solely build and test the supported Ceph releases (luminous,
mimic) on 7.5 ?

CentOS itself does not support old point releases, and I don't think
we should imply that we do either.

From #centos-devel earlier:

< fidencio> TrevorH: avij: do you know whether there's an offical EOL
announcement for 6.9?
< avij> fidencio: it happens every time there's a 6.(x+1) release
< avij> it's always implied
< avij> RH has some support for past RHEL releases, but there's no such
mechanism in CentOS

- Ken

kefu chai

2018-09-25 12:17:19 UTC

Permalink

Post by Ken Dreyer

Post by kefu chai
my question is: is it okay to drop the support of centos/rhel 7.4? so
we will solely build and test the supported Ceph releases (luminous,
mimic) on 7.5 ?

CentOS itself does not support old point releases, and I don't think
we should imply that we do either.
< fidencio> TrevorH: avij: do you know whether there's an offical EOL
announcement for 6.9?
< avij> fidencio: it happens every time there's a 6.(x+1) release
< avij> it's always implied
< avij> RH has some support for past RHEL releases, but there's no such
mechanism in CentOS

thanks for the insights, Ken. as suggested by Brad, the issue was
fixed by bumping up the required gperftools-lib version. assuming both
CentOS/RHEL 7.4 and 7.5 have access to the gperftools-lib whose
version is the same with that of gperftools-devel, which we use to
build Ceph rpm packages.

--
Regards
Kefu Chai