Discussion:
Repeated failures in RGW in Ceph 12.1.4
(too old to reply)
Bryan Banister
2017-08-30 14:44:40 UTC
Permalink
Not sure what's happening but we started to but a decent load on the RGWs we have setup and we were seeing failures with the following kind of fingerprint:

2017-08-29 17:06:22.072361 7ffdc501a700 1 rgw realm reloader: Frontends paused
2017-08-29 17:06:22.072359 7fffacbe9700 1 civetweb: 0x555556add000: 7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT /blah?partNumber=8&uploadId=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0 - Minio (linux; amd64) minio-go/3.0.0
2017-08-29 17:06:22.072438 7fffcb426700 0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
2017-08-29 17:06:23.689610 7ffdc501a700 1 rgw realm reloader: Store closed
2017-08-29 17:06:24.117630 7ffdc501a700 1 failed to decode the mdlog history: buffer::end_of_buffer
2017-08-29 17:06:24.117635 7ffdc501a700 1 failed to read mdlog history: (5) Input/output error
2017-08-29 17:06:24.118711 7ffdc501a700 1 rgw realm reloader: Creating new store
2017-08-29 17:06:24.118901 7ffdc501a700 1 mgrc service_daemon_register rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz,distro=rhel,distro_description=Red Hat Enterprise Linux Server 7.3 (Maipo),distro_version=7.3,frontend_config#0=civetweb port=80 num_threads=1024,frontend_type#0=civetweb,hos
tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT 2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}
2017-08-29 17:06:24.118925 7ffdc501a700 1 rgw realm reloader: Finishing initialization of new store
2017-08-29 17:06:24.118927 7ffdc501a700 1 rgw realm reloader: - REST subsystem init
2017-08-29 17:06:24.118943 7ffdc501a700 1 rgw realm reloader: - user subsystem init
2017-08-29 17:06:24.118947 7ffdc501a700 1 rgw realm reloader: - user subsystem init
2017-08-29 17:06:24.118950 7ffdc501a700 1 rgw realm reloader: - usage subsystem init
2017-08-29 17:06:24.118985 7ffdc501a700 1 rgw realm reloader: Resuming frontends with new realm configuration.
2017-08-29 17:06:24.119018 7fffad3ea700 1 ====== starting new request req=0x7fffad3e4190 =====
2017-08-29 17:06:24.119039 7fffacbe9700 1 ====== starting new request req=0x7fffacbe3190 =====
2017-08-29 17:06:24.120163 7fffacbe9700 1 ====== req done req=0x7fffacbe3190 op status=0 http_status=403 ======
2017-08-29 17:06:24.120200 7fffad3ea700 1 ====== req done req=0x7fffad3e4190 op status=0 http_status=403 ======

Any help understanding how to fix this would be greatly appreciated!
-Bryan

________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
Bryan Banister
2017-08-30 18:44:12 UTC
Permalink
Looks like my RGW users also got deleted/lost?!

[***@carf-ceph-osd01 ~]# radosgw-admin user list
[]

Yikes!!

Any thoughts?
-Bryan

From: ceph-users [mailto:ceph-users-***@lists.ceph.com] On Behalf Of Bryan Banister
Sent: Wednesday, August 30, 2017 9:45 AM
To: ceph-***@lists.ceph.com
Subject: [ceph-users] Repeated failures in RGW in Ceph 12.1.4

Note: External Email
________________________________
Not sure what's happening but we started to but a decent load on the RGWs we have setup and we were seeing failures with the following kind of fingerprint:

2017-08-29 17:06:22.072361 7ffdc501a700 1 rgw realm reloader: Frontends paused
2017-08-29 17:06:22.072359 7fffacbe9700 1 civetweb: 0x555556add000: 7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT /blah?partNumber=8&uploadId=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0 - Minio (linux; amd64) minio-go/3.0.0
2017-08-29 17:06:22.072438 7fffcb426700 0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
2017-08-29 17:06:23.689610 7ffdc501a700 1 rgw realm reloader: Store closed
2017-08-29 17:06:24.117630 7ffdc501a700 1 failed to decode the mdlog history: buffer::end_of_buffer
2017-08-29 17:06:24.117635 7ffdc501a700 1 failed to read mdlog history: (5) Input/output error
2017-08-29 17:06:24.118711 7ffdc501a700 1 rgw realm reloader: Creating new store
2017-08-29 17:06:24.118901 7ffdc501a700 1 mgrc service_daemon_register rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz,distro=rhel,distro_description=Red Hat Enterprise Linux Server 7.3 (Maipo),distro_version=7.3,frontend_config#0=civetweb port=80 num_threads=1024,frontend_type#0=civetweb,hos
tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT 2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}
2017-08-29 17:06:24.118925 7ffdc501a700 1 rgw realm reloader: Finishing initialization of new store
2017-08-29 17:06:24.118927 7ffdc501a700 1 rgw realm reloader: - REST subsystem init
2017-08-29 17:06:24.118943 7ffdc501a700 1 rgw realm reloader: - user subsystem init
2017-08-29 17:06:24.118947 7ffdc501a700 1 rgw realm reloader: - user subsystem init
2017-08-29 17:06:24.118950 7ffdc501a700 1 rgw realm reloader: - usage subsystem init
2017-08-29 17:06:24.118985 7ffdc501a700 1 rgw realm reloader: Resuming frontends with new realm configuration.
2017-08-29 17:06:24.119018 7fffad3ea700 1 ====== starting new request req=0x7fffad3e4190 =====
2017-08-29 17:06:24.119039 7fffacbe9700 1 ====== starting new request req=0x7fffacbe3190 =====
2017-08-29 17:06:24.120163 7fffacbe9700 1 ====== req done req=0x7fffacbe3190 op status=0 http_status=403 ======
2017-08-29 17:06:24.120200 7fffad3ea700 1 ====== req done req=0x7fffad3e4190 op status=0 http_status=403 ======

Any help understanding how to fix this would be greatly appreciated!
-Bryan

________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.

________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
Yehuda Sadeh-Weinraub
2017-08-30 20:30:20 UTC
Permalink
On Wed, Aug 30, 2017 at 5:44 PM, Bryan Banister
Not sure what’s happening but we started to but a decent load on the RGWs we
have setup and we were seeing failures with the following kind of
2017-08-29 17:06:22.072361 7ffdc501a700 1 rgw realm reloader: Frontends paused
Are you modifying configuration? Could be that something is sending
HUP singal to the radosgw process. We disabled this behavior (process
dynamic reconfig after HUP) in 12.2.0.

Yehuda
7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT
/blah?partNumber=8&uploadId=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0
- Minio (linux; amd64) minio-go/3.0.0
2017-08-29 17:06:22.072438 7fffcb426700 0 ERROR: failed to clone shard,
completion_mgr.get_next() returned ret=-125
2017-08-29 17:06:23.689610 7ffdc501a700 1 rgw realm reloader: Store closed
2017-08-29 17:06:24.117630 7ffdc501a700 1 failed to decode the mdlog
history: buffer::end_of_buffer
2017-08-29 17:06:24.117635 7ffdc501a700 1 failed to read mdlog history: (5)
Input/output error
2017-08-29 17:06:24.118711 7ffdc501a700 1 rgw realm reloader: Creating new store
2017-08-29 17:06:24.118901 7ffdc501a700 1 mgrc service_daemon_register
rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4
(a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R)
Enterprise Linux Server 7.3
(Maipo),distro_version=7.3,frontend_config#0=civetweb port=80
num_threads=1024,frontend_type#0=civetweb,hos
tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT
2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}
2017-08-29 17:06:24.118925 7ffdc501a700 1 rgw realm reloader: Finishing
initialization of new store
2017-08-29 17:06:24.118927 7ffdc501a700 1 rgw realm reloader: - REST subsystem init
2017-08-29 17:06:24.118943 7ffdc501a700 1 rgw realm reloader: - user subsystem init
2017-08-29 17:06:24.118947 7ffdc501a700 1 rgw realm reloader: - user subsystem init
2017-08-29 17:06:24.118950 7ffdc501a700 1 rgw realm reloader: - usage subsystem init
2017-08-29 17:06:24.118985 7ffdc501a700 1 rgw realm reloader: Resuming
frontends with new realm configuration.
2017-08-29 17:06:24.119018 7fffad3ea700 1 ====== starting new request
req=0x7fffad3e4190 =====
2017-08-29 17:06:24.119039 7fffacbe9700 1 ====== starting new request
req=0x7fffacbe3190 =====
2017-08-29 17:06:24.120163 7fffacbe9700 1 ====== req done
req=0x7fffacbe3190 op status=0 http_status=403 ======
2017-08-29 17:06:24.120200 7fffad3ea700 1 ====== req done
req=0x7fffad3e4190 op status=0 http_status=403 ======
Any help understanding how to fix this would be greatly appreciated!
-Bryan
________________________________
Note: This email is for the confidential use of the named addressee(s) only
and may contain proprietary, confidential or privileged information. If you
are not the intended recipient, you are hereby notified that any review,
dissemination or copying of this email is strictly prohibited, and to please
notify the sender immediately and destroy this email and any attachments.
Email transmission cannot be guaranteed to be secure or error-free. The
Company, therefore, does not make any guarantees as to the completeness or
accuracy of this email or any attachments. This email is for informational
purposes only and does not constitute a recommendation, offer, request or
solicitation of any kind to buy, sell, subscribe, redeem or perform any type
of transaction of a financial product.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Bryan Banister
2017-08-30 20:42:10 UTC
Permalink
We are not sending a HUP signal that we know about. We were not modifying our configuration. However all user accounts in the RGW were lost!
-Bryan

-----Original Message-----
From: Yehuda Sadeh-Weinraub [mailto:***@redhat.com]
Sent: Wednesday, August 30, 2017 3:30 PM
To: Bryan Banister <***@jumptrading.com>
Cc: ceph-***@lists.ceph.com
Subject: Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4

Note: External Email
-------------------------------------------------

On Wed, Aug 30, 2017 at 5:44 PM, Bryan Banister
Not sure what’s happening but we started to but a decent load on the RGWs we
have setup and we were seeing failures with the following kind of
2017-08-29 17:06:22.072361 7ffdc501a700 1 rgw realm reloader: Frontends paused
Are you modifying configuration? Could be that something is sending
HUP singal to the radosgw process. We disabled this behavior (process
dynamic reconfig after HUP) in 12.2.0.

Yehuda
7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT
/blah?partNumber=8&uploadId=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0
- Minio (linux; amd64) minio-go/3.0.0
2017-08-29 17:06:22.072438 7fffcb426700 0 ERROR: failed to clone shard,
completion_mgr.get_next() returned ret=-125
2017-08-29 17:06:23.689610 7ffdc501a700 1 rgw realm reloader: Store closed
2017-08-29 17:06:24.117630 7ffdc501a700 1 failed to decode the mdlog
history: buffer::end_of_buffer
2017-08-29 17:06:24.117635 7ffdc501a700 1 failed to read mdlog history: (5)
Input/output error
2017-08-29 17:06:24.118711 7ffdc501a700 1 rgw realm reloader: Creating new store
2017-08-29 17:06:24.118901 7ffdc501a700 1 mgrc service_daemon_register
rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4
(a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R)
Enterprise Linux Server 7.3
(Maipo),distro_version=7.3,frontend_config#0=civetweb port=80
num_threads=1024,frontend_type#0=civetweb,hos
tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT
2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}
2017-08-29 17:06:24.118925 7ffdc501a700 1 rgw realm reloader: Finishing
initialization of new store
2017-08-29 17:06:24.118927 7ffdc501a700 1 rgw realm reloader: - REST subsystem init
2017-08-29 17:06:24.118943 7ffdc501a700 1 rgw realm reloader: - user subsystem init
2017-08-29 17:06:24.118947 7ffdc501a700 1 rgw realm reloader: - user subsystem init
2017-08-29 17:06:24.118950 7ffdc501a700 1 rgw realm reloader: - usage subsystem init
2017-08-29 17:06:24.118985 7ffdc501a700 1 rgw realm reloader: Resuming
frontends with new realm configuration.
2017-08-29 17:06:24.119018 7fffad3ea700 1 ====== starting new request
req=0x7fffad3e4190 =====
2017-08-29 17:06:24.119039 7fffacbe9700 1 ====== starting new request
req=0x7fffacbe3190 =====
2017-08-29 17:06:24.120163 7fffacbe9700 1 ====== req done
req=0x7fffacbe3190 op status=0 http_status=403 ======
2017-08-29 17:06:24.120200 7fffad3ea700 1 ====== req done
req=0x7fffad3e4190 op status=0 http_status=403 ======
Any help understanding how to fix this would be greatly appreciated!
-Bryan
________________________________
Note: This email is for the confidential use of the named addressee(s) only
and may contain proprietary, confidential or privileged information. If you
are not the intended recipient, you are hereby notified that any review,
dissemination or copying of this email is strictly prohibited, and to please
notify the sender immediately and destroy this email and any attachments.
Email transmission cannot be guaranteed to be secure or error-free. The
Company, therefore, does not make any guarantees as to the completeness or
accuracy of this email or any attachments. This email is for informational
purposes only and does not constitute a recommendation, offer, request or
solicitation of any kind to buy, sell, subscribe, redeem or perform any type
of transaction of a financial product.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
Bryan Banister
2017-08-30 22:09:41 UTC
Permalink
Looking at the systemd service it does show that twice, at roughly the same time and one day apart, the service did receive a HUP signal:

Aug 29 16:31:02 carf-ceph-osd02 radosgw[130050]: 2017-08-29 16:31:02.528559 7fffc641c700 -1 received signal: Hangup from PID: 73176 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0

Aug 30 16:32:02 carf-ceph-osd02 radosgw[130050]: 2017-08-30 16:32:02.529825 7fffc641c700 -1 received signal: Hangup from PID: 48062 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0



Any idea what would do this?



I'll be updating the version to 12.2.0 shortly,

-Bryan



-----Original Message-----
From: ceph-users [mailto:ceph-users-***@lists.ceph.com] On Behalf Of Bryan Banister
Sent: Wednesday, August 30, 2017 3:42 PM
To: Yehuda Sadeh-Weinraub <***@redhat.com>
Cc: ceph-***@lists.ceph.com
Subject: Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4



Note: External Email

-------------------------------------------------



We are not sending a HUP signal that we know about. We were not modifying our configuration. However all user accounts in the RGW were lost!

-Bryan



-----Original Message-----

From: Yehuda Sadeh-Weinraub [mailto:***@redhat.com]

Sent: Wednesday, August 30, 2017 3:30 PM

To: Bryan Banister <***@jumptrading.com<mailto:***@jumptrading.com>>

Cc: ceph-***@lists.ceph.com<mailto:ceph-***@lists.ceph.com>

Subject: Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4



Note: External Email

-------------------------------------------------



On Wed, Aug 30, 2017 at 5:44 PM, Bryan Banister
Not sure what’s happening but we started to but a decent load on the RGWs we
have setup and we were seeing failures with the following kind of
2017-08-29 17:06:22.072361 7ffdc501a700 1 rgw realm reloader: Frontends
paused
Are you modifying configuration? Could be that something is sending

HUP singal to the radosgw process. We disabled this behavior (process

dynamic reconfig after HUP) in 12.2.0.



Yehuda
7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT
/blah?partNumber=8&uploadId=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0
- Minio (linux; amd64) minio-go/3.0.0
2017-08-29 17:06:22.072438 7fffcb426700 0 ERROR: failed to clone shard,
completion_mgr.get_next() returned ret=-125
2017-08-29 17:06:23.689610 7ffdc501a700 1 rgw realm reloader: Store closed
2017-08-29 17:06:24.117630 7ffdc501a700 1 failed to decode the mdlog
history: buffer::end_of_buffer
2017-08-29 17:06:24.117635 7ffdc501a700 1 failed to read mdlog history: (5)
Input/output error
2017-08-29 17:06:24.118711 7ffdc501a700 1 rgw realm reloader: Creating new
store
2017-08-29 17:06:24.118901 7ffdc501a700 1 mgrc service_daemon_register
rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4
(a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R)
Enterprise Linux Server 7.3
(Maipo),distro_version=7.3,frontend_config#0=civetweb port=80
num_threads=1024,frontend_type#0=civetweb,hos
tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT
2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}
2017-08-29 17:06:24.118925 7ffdc501a700 1 rgw realm reloader: Finishing
initialization of new store
2017-08-29 17:06:24.118927 7ffdc501a700 1 rgw realm reloader: - REST
subsystem init
2017-08-29 17:06:24.118943 7ffdc501a700 1 rgw realm reloader: - user
subsystem init
2017-08-29 17:06:24.118947 7ffdc501a700 1 rgw realm reloader: - user
subsystem init
2017-08-29 17:06:24.118950 7ffdc501a700 1 rgw realm reloader: - usage
subsystem init
2017-08-29 17:06:24.118985 7ffdc501a700 1 rgw realm reloader: Resuming
frontends with new realm configuration.
2017-08-29 17:06:24.119018 7fffad3ea700 1 ====== starting new request
req=0x7fffad3e4190 =====
2017-08-29 17:06:24.119039 7fffacbe9700 1 ====== starting new request
req=0x7fffacbe3190 =====
2017-08-29 17:06:24.120163 7fffacbe9700 1 ====== req done
req=0x7fffacbe3190 op status=0 http_status=403 ======
2017-08-29 17:06:24.120200 7fffad3ea700 1 ====== req done
req=0x7fffad3e4190 op status=0 http_status=403 ======
Any help understanding how to fix this would be greatly appreciated!
-Bryan
________________________________
Note: This email is for the confidential use of the named addressee(s) only
and may contain proprietary, confidential or privileged information. If you
are not the intended recipient, you are hereby notified that any review,
dissemination or copying of this email is strictly prohibited, and to please
notify the sender immediately and destroy this email and any attachments.
Email transmission cannot be guaranteed to be secure or error-free. The
Company, therefore, does not make any guarantees as to the completeness or
accuracy of this email or any attachments. This email is for informational
purposes only and does not constitute a recommendation, offer, request or
solicitation of any kind to buy, sell, subscribe, redeem or perform any type
of transaction of a financial product.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
________________________________



Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.

_______________________________________________

ceph-users mailing list

ceph-***@lists.ceph.com<mailto:ceph-***@lists.ceph.com>

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
Ben Hines
2017-08-30 22:40:01 UTC
Permalink
The daily log rotation.

-Ben
Post by Bryan Banister
Looking at the systemd service it does show that twice, at roughly the
Aug 29 16:31:02 carf-ceph-osd02 radosgw[130050]: 2017-08-29
16:31:02.528559 7fffc641c700 -1 received signal: Hangup from PID: 73176
task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse
radosgw UID: 0
Aug 30 16:32:02 carf-ceph-osd02 radosgw[130050]: 2017-08-30
16:32:02.529825 7fffc641c700 -1 received signal: Hangup from PID: 48062
task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse
radosgw UID: 0
Any idea what would do this?
I'll be updating the version to 12.2.0 shortly,
-Bryan
-----Original Message-----
Sent: Wednesday, August 30, 2017 3:42 PM
Subject: Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4
Note: External Email
-------------------------------------------------
We are not sending a HUP signal that we know about. We were not modifying
our configuration. However all user accounts in the RGW were lost!
-Bryan
-----Original Message-----
Sent: Wednesday, August 30, 2017 3:30 PM
Subject: Re: [ceph-users] Repeated failures in RGW in Ceph 12.1.4
Note: External Email
-------------------------------------------------
On Wed, Aug 30, 2017 at 5:44 PM, Bryan Banister
Not sure what’s happening but we started to but a decent load on the
RGWs we
have setup and we were seeing failures with the following kind of
2017-08-29 17:06:22.072361 7ffdc501a700 1 rgw realm reloader: Frontends
paused
Are you modifying configuration? Could be that something is sending
HUP singal to the radosgw process. We disabled this behavior (process
dynamic reconfig after HUP) in 12.2.0.
Yehuda
7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT
/blah?partNumber=8&uploadId=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1"
1 0
- Minio (linux; amd64) minio-go/3.0.0
2017-08-29 17:06:22.072438 7fffcb426700 0 ERROR: failed to clone shard,
completion_mgr.get_next() returned ret=-125
2017-08-29 17:06:23.689610 7ffdc501a700 1 rgw realm reloader: Store
closed
2017-08-29 17:06:24.117630 7ffdc501a700 1 failed to decode the mdlog
history: buffer::end_of_buffer
(5)
Input/output error
2017-08-29 17:06:24.118711 7ffdc501a700 1 rgw realm reloader: Creating
new
store
2017-08-29 17:06:24.118901 7ffdc501a700 1 mgrc service_daemon_register
rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version
12.1.4
(a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R)
Enterprise Linux Server 7.3
(Maipo),distro_version=7.3,frontend_config#0=civetweb port=80
num_threads=1024,frontend_type#0=civetweb,hos
tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT
2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_
swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=
b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,
zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}
2017-08-29 17:06:24.118925 7ffdc501a700 1 rgw realm reloader: Finishing
initialization of new store
2017-08-29 17:06:24.118927 7ffdc501a700 1 rgw realm reloader: - REST
subsystem init
2017-08-29 17:06:24.118943 7ffdc501a700 1 rgw realm reloader: - user
subsystem init
2017-08-29 17:06:24.118947 7ffdc501a700 1 rgw realm reloader: - user
subsystem init
2017-08-29 17:06:24.118950 7ffdc501a700 1 rgw realm reloader: - usage
subsystem init
2017-08-29 17:06:24.118985 7ffdc501a700 1 rgw realm reloader: Resuming
frontends with new realm configuration.
2017-08-29 17:06:24.119018 7fffad3ea700 1 ====== starting new request
req=0x7fffad3e4190 =====
2017-08-29 17:06:24.119039 7fffacbe9700 1 ====== starting new request
req=0x7fffacbe3190 =====
2017-08-29 17:06:24.120163 7fffacbe9700 1 ====== req done
req=0x7fffacbe3190 op status=0 http_status=403 ======
2017-08-29 17:06:24.120200 7fffad3ea700 1 ====== req done
req=0x7fffad3e4190 op status=0 http_status=403 ======
Any help understanding how to fix this would be greatly appreciated!
-Bryan
________________________________
Note: This email is for the confidential use of the named addressee(s)
only
and may contain proprietary, confidential or privileged information. If
you
are not the intended recipient, you are hereby notified that any review,
dissemination or copying of this email is strictly prohibited, and to
please
notify the sender immediately and destroy this email and any attachments.
Email transmission cannot be guaranteed to be secure or error-free. The
Company, therefore, does not make any guarantees as to the completeness
or
accuracy of this email or any attachments. This email is for
informational
purposes only and does not constitute a recommendation, offer, request or
solicitation of any kind to buy, sell, subscribe, redeem or perform any
type
of transaction of a financial product.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
________________________________
Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited, and
to please notify the sender immediately and destroy this email and any
attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to the
completeness or accuracy of this email or any attachments. This email is
for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
------------------------------
Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited, and
to please notify the sender immediately and destroy this email and any
attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to the
completeness or accuracy of this email or any attachments. This email is
for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Loading...