[ceph-users] RadosGW S3ResponseError: 405 Method Not Allowed

Discussion:

Steffen W Sørensen

10 years ago

Hi,

Newbie to RadosGW+Ceph, but learning...
Got a running Ceph Cluster working with rbd+CephFS clients. Now I'm trying to verify a RadosGW S3 api, but seems to have an issue with RadosGW access.

I get the error (not found anything searching so far...):

S3ResponseError: 405 Method Not Allowed

when trying to access the rgw.

Apache vhost access log file says:

10.20.0.29 - - [27/Feb/2015:14:09:04 +0100] "GET / HTTP/1.1" 405 27 "-" "Boto/2.34.0 Python/2.6.6 Linux/2.6.32-504.8.1.el6.x86_64"

and Apache's general error_log file says:

[Fri Feb 27 14:09:04 2015] [warn] FastCGI: 10.20.0.29 GET http://{fqdn}:8005/ auth AWS WL4EJJYTLVYXEHNR6QSA:X6XR4z7Gr9qTMNDphTNlRUk3gfc=

RadosGW seems to launch and run fine, though /var/log/messages at launches says:

Feb 27 14:12:34 rgw kernel: radosgw[14985]: segfault at e0 ip 0000003fb36cb1dc sp 00007fffde221410 error 4 in librados.so.2.0.0[3fb3200000+6d0000]

# ps -fuapache
UID PID PPID C STIME TTY TIME CMD
apache 15113 15111 0 14:07 ? 00:00:00 /usr/sbin/fcgi-
apache 15114 15111 0 14:07 ? 00:00:00 /usr/sbin/httpd
apache 15115 15111 0 14:07 ? 00:00:00 /usr/sbin/httpd
apache 15116 15111 0 14:07 ? 00:00:00 /usr/sbin/httpd
apache 15117 15111 0 14:07 ? 00:00:00 /usr/sbin/httpd
apache 15118 15111 0 14:07 ? 00:00:00 /usr/sbin/httpd
apache 15119 15111 0 14:07 ? 00:00:00 /usr/sbin/httpd
apache 15120 15111 0 14:07 ? 00:00:00 /usr/sbin/httpd
apache 15121 15111 0 14:07 ? 00:00:00 /usr/sbin/httpd
apache 15224 1 1 14:12 ? 00:00:25 /usr/bin/radosgw -n client.radosgw.owmblob

RadosGW create my FastCGI socket and a default .asok, (not sure why/what default socket are meant for) as well as the configured log file though it never logs anything...

# tail -18 /etc/ceph/ceph.conf:

[client.radosgw.owmblob]
keyring = /etc/ceph/ceph.client.radosgw.keyring
host = rgw
rgw data = /var/lib/ceph/radosgw/ceph-rgw
log file = /var/log/radosgw/client.radosgw.owmblob.log
debug rgw = 20
rgw enable log rados = true
rgw enable ops log = true
rgw enable apis = s3
rgw cache enabled = true
rgw cache lru size = 10000
rgw socket path = /var/run/ceph/ceph.radosgw.owmblob.fastcgi.sock
;#rgw host = localhost
;#rgw port = 8004
rgw dns name = {fqdn}
rgw print continue = true
rgw thread pool size = 20

Turned out /etc/init.d/ceph-radosgw didn't chown $USER even when log_file didn't exist,
assuming radosgw creates this log file when opening it, only it creates it as root not $USER, thus not output, manually chowning it and restarting GW gives output ala:

2015-02-27 15:25:14.464112 7fef463e9700 20 enqueued request req=0x25dea40
2015-02-27 15:25:14.465750 7fef463e9700 20 RGWWQ:
2015-02-27 15:25:14.465786 7fef463e9700 20 req: 0x25dea40
2015-02-27 15:25:14.465864 7fef463e9700 10 allocated request req=0x25e3050
2015-02-27 15:25:14.466214 7fef431e4700 20 dequeued request req=0x25dea40
2015-02-27 15:25:14.466677 7fef431e4700 20 RGWWQ: empty
2015-02-27 15:25:14.467888 7fef431e4700 20 CONTENT_LENGTH=0
2015-02-27 15:25:14.467922 7fef431e4700 20 DOCUMENT_ROOT=/var/www/html
2015-02-27 15:25:14.467941 7fef431e4700 20 FCGI_ROLE=RESPONDER
2015-02-27 15:25:14.467958 7fef431e4700 20 GATEWAY_INTERFACE=CGI/1.1
2015-02-27 15:25:14.467976 7fef431e4700 20 HTTP_ACCEPT_ENCODING=identity
2015-02-27 15:25:14.469476 7fef431e4700 20 HTTP_AUTHORIZATION=AWS WL4EJJYTLVYXEHNR6QSA:OAT0zVItGyp98T5mALeHz4p1fcg=
2015-02-27 15:25:14.469516 7fef431e4700 20 HTTP_DATE=Fri, 27 Feb 2015 14:25:14 GMT
2015-02-27 15:25:14.469533 7fef431e4700 20 HTTP_HOST={fqdn}:8005
2015-02-27 15:25:14.469550 7fef431e4700 20 HTTP_USER_AGENT=Boto/2.34.0 Python/2.6.6 Linux/2.6.32-504.8.1.el6.x86_64
2015-02-27 15:25:14.469571 7fef431e4700 20 PATH=/sbin:/usr/sbin:/bin:/usr/bin
2015-02-27 15:25:14.469589 7fef431e4700 20 QUERY_STRING=
2015-02-27 15:25:14.469607 7fef431e4700 20 REMOTE_ADDR=10.20.0.29
2015-02-27 15:25:14.469624 7fef431e4700 20 REMOTE_PORT=34386
2015-02-27 15:25:14.469641 7fef431e4700 20 REQUEST_METHOD=GET
2015-02-27 15:25:14.469658 7fef431e4700 20 REQUEST_URI=/
2015-02-27 15:25:14.469677 7fef431e4700 20 SCRIPT_FILENAME=/var/www/html/s3gw.fcgi
2015-02-27 15:25:14.469694 7fef431e4700 20 SCRIPT_NAME=/
2015-02-27 15:25:14.469711 7fef431e4700 20 SCRIPT_URI=http://{fqdn}:8005/
2015-02-27 15:25:14.469730 7fef431e4700 20 SCRIPT_URL=/
2015-02-27 15:25:14.469748 7fef431e4700 20 SERVER_ADDR=10.20.0.29
2015-02-27 15:25:14.469765 7fef431e4700 20 SERVER_ADMIN={email}
2015-02-27 15:25:14.469782 7fef431e4700 20 SERVER_NAME={fqdn}
2015-02-27 15:25:14.469801 7fef431e4700 20 SERVER_PORT=8005
2015-02-27 15:25:14.469818 7fef431e4700 20 SERVER_PROTOCOL=HTTP/1.1
2015-02-27 15:25:14.469835 7fef431e4700 20 SERVER_SIGNATURE=
2015-02-27 15:25:14.469852 7fef431e4700 20 SERVER_SOFTWARE=Apache/2.2.22 (Fedora)
2015-02-27 15:25:14.469882 7fef431e4700 1 ====== starting new request req=0x25dea40 =====
2015-02-27 15:25:14.470558 7fef431e4700 2 req 1:0.000679::GET /::initializing
2015-02-27 15:25:14.470669 7fef431e4700 10 host={fqdn}:8005 rgw_dns_name={fqdn}
2015-02-27 15:25:14.471810 7fef431e4700 2 req 1:0.001933::GET /::http status=405

any clues what I'm missing out would be appreciated, TIA!

/Steffen

Steffen W Sørensen

10 years ago

Permalink

Hi

Seems there's a minor flaw in CentOS/RHEL niit script:

line 91 reads:

daemon --user="$user" "$RADOSGW -n $name"

should ImHO be:

daemon --user="$user" $RADOSGW -n $name

to avoid /etc/rc.d/init.d/functions:__pids_var_run line 151 complain in dirname :)

/Steffen

Yehuda Sadeh-Weinraub

10 years ago

Permalink

----- Original Message -----

...

405 would usually mean one of two things. Either you sent the request to a misconfigured apache and it didn't get to the gateway at all, or the gateway got the request but couldn't match it to a known operation. It seems that your request did find its way to the gateway, but the question here is why doesn't it match to a known operation. This really looks like a valid list all buckets request, so I'm not sure what's happening.
I'd look at two things first. One is the '{fqdn}' string, which I'm not sure whether that's the actual string that you have, or whether you just replaced it for the sake of anonymity. The second is the port number, which should be fine, but maybe the fact that it appears as part of the script uri triggers some issue.

Yehuda

/Steffen
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Steffen W Sørensen

10 years ago

Permalink

Post by Yehuda Sadeh-Weinraub
I'd look at two things first. One is the '{fqdn}' string, which I'm not sure whether that's the actual string that you have, or whether you just replaced it for the sake of anonymity. The second is the port number, which should be fine, but maybe the fact that it appears as part of the script uri triggers some issue.

When launching radosgw it logs this:

...
2015-02-27 18:33:58.663960 7f200b67a8a0 20 rados->read obj-ofs=0 read_ofs=0 read_len=524288
2015-02-27 18:33:58.675821 7f200b67a8a0 20 rados->read r=0 bl.length=678
2015-02-27 18:33:58.676532 7f200b67a8a0 10 cache put: name=.rgw.root+zone_info.default
2015-02-27 18:33:58.676573 7f200b67a8a0 10 moving .rgw.root+zone_info.default to cache LRU end
2015-02-27 18:33:58.677415 7f200b67a8a0 2 zone default is master
2015-02-27 18:33:58.677666 7f200b67a8a0 20 get_obj_state: rctx=0x2a85cd0 obj=.rgw.root:region_map state=0x2a86498 s->prefetch_data=0
2015-02-27 18:33:58.677760 7f200b67a8a0 10 cache get: name=.rgw.root+region_map : miss
2015-02-27 18:33:58.709411 7f200b67a8a0 10 cache put: name=.rgw.root+region_map
2015-02-27 18:33:58.709846 7f200b67a8a0 10 adding .rgw.root+region_map to cache LRU end
2015-02-27 18:33:58.957336 7f1ff17f2700 2 garbage collection: start
2015-02-27 18:33:58.959189 7f1ff0df1700 20 BucketsSyncThread: start
2015-02-27 18:33:58.985486 7f200b67a8a0 0 framework: fastcgi
2015-02-27 18:33:58.985778 7f200b67a8a0 0 framework: civetweb
2015-02-27 18:33:58.985879 7f200b67a8a0 0 framework conf key: port, val: 7480
2015-02-27 18:33:58.986462 7f200b67a8a0 0 starting handler: civetweb
2015-02-27 18:33:59.032173 7f1fc3fff700 20 UserSyncThread: start
2015-02-27 18:33:59.214739 7f200b67a8a0 0 starting handler: fastcgi
2015-02-27 18:33:59.286723 7f1fb59e8700 10 allocated request req=0x2aa1b20
2015-02-27 18:34:00.533188 7f1fc3fff700 20 RGWRados::pool_iterate: got {my user name}
2015-02-27 18:34:01.038190 7f1ff17f2700 2 garbage collection: stop
2015-02-27 18:34:01.670780 7f1fc3fff700 20 RGWUserStatsCache: sync user={my user name}
2015-02-27 18:34:01.687730 7f1fc3fff700 0 ERROR: can't read user header: ret=-2
2015-02-27 18:34:01.689734 7f1fc3fff700 0 ERROR: sync_user() failed, user={my user name} ret=-2

Why does it seem to find my radosgw defined user name as a pool and what might bring it to fail to read user header?

/Steffen

Yehuda Sadeh-Weinraub

10 years ago

Permalink

----- Original Message -----

...

That's just a red herring. It tries to sync the user stats, but it can't because quota is not enabled (iirc). We should probably get rid of these messages as they're pretty confusing.

Yehuda

Steffen W Sørensen

10 years ago

Permalink

That's the old way of defining pools. The new way involves in defining a zone and placement targets for that zone. Then you can have different default placement targets for different users.

Anu URL/pointers to better understand such matters?

Do you have any special config in your ceph.conf? E.g., did you modify the rgw_enable_apis configurable by any chance?

# tail -20 /etc/ceph/ceph.conf

[client.radosgw.owmblob]
keyring = /etc/ceph/ceph.client.radosgw.keyring
host = rgw
user = apache
rgw data = /var/lib/ceph/radosgw/ceph-rgw
log file = /var/log/radosgw/client.radosgw.owmblob.log
debug rgw = 20
rgw enable log rados = true
rgw enable ops log = true
rgw enable apis = s3
rgw cache enabled = true
rgw cache lru size = 10000
rgw socket path = /var/run/ceph/ceph.radosgw.owmblob.fastcgi.sock
;#rgw host = localhost
;#rgw port = 8004
rgw dns name = {fqdn}
rgw print continue = true
rgw thread pool size = 20

What is the purpose of the data directory btw?

/Steffen

Steffen W Sørensen

10 years ago

Permalink

Post by Steffen W SÃ¸rensen
rgw enable apis = s3

Commenting this out makes it work :)

[***@rgw tests3]# ./lsbuckets.py
[***@rgw tests3]# ./lsbuckets.py
my-new-bucket 2015-02-27T17:49:04.000Z
[***@rgw tests3]#

...
2015-02-27 18:49:22.601578 7f48f2bdd700 20 rgw_create_bucket returned ret=-17 bucket=my-new-bucket(@{i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[default.5234475.2])
2015-02-27 18:49:22.625672 7f48f2bdd700 2 req 4:0.350444:s3:PUT /my-new-bucket/:create_bucket:http status=200
2015-02-27 18:49:22.625758 7f48f2bdd700 1 ====== req done req=0x7f4938007810 http_status=200 ======
...

Why I just wants a S3 API available not admin API?

/Steffen

Steffen W Sørensen

10 years ago

Permalink

Post by Steffen W SÃ¸rensen

Post by Steffen W SÃ¸rensen
rgw enable apis = s3

Commenting this out makes it work :)

Thanks for helping on this initial issue!

Post by Steffen W SÃ¸rensen
my-new-bucket 2015-02-27T17:49:04.000Z
...
2015-02-27 18:49:22.625672 7f48f2bdd700 2 req 4:0.350444:s3:PUT /my-new-bucket/:create_bucket:http status=200
2015-02-27 18:49:22.625758 7f48f2bdd700 1 ====== req done req=0x7f4938007810 http_status=200 ======
...

Into which pool does such user data (buckets and objects) gets stored and possible howto direct user data into a dedicated pool?

[***@rgw ~]# rados df
pool name category KB objects clones degraded unfound rd rd KB wr wr KB
.intent-log - 0 0 0 0 0 0 0 0 0
.log - 1 1 0 0 0 0 0 2 2
.rgw - 1 4 0 0 0 17 14 10 4
.rgw.buckets - 0 0 0 0 0 0 0 0 0
.rgw.buckets.extra - 0 0 0 0 0 0 0 0 0
.rgw.buckets.index - 0 1 0 0 0 2 0 3 0
.rgw.control - 0 8 0 0 0 0 0 0 0
.rgw.gc - 0 32 0 0 0 8302 8302 5556 0
.rgw.root - 1 3 0 0 0 929 618 3 3
.usage - 0 0 0 0 0 0 0 0 0
.users - 1 1 0 0 0 6 4 5 3
.users.email - 1 1 0 0 0 3 2 5 3
.users.swift - 0 0 0 0 0 0 0 0 0
.users.uid - 1 2 0 0 0 65 54 16 4

Assume a bucket is a naming container for objects in a pool maybe similar to a directory with files.

/Steffen

Steffen W Sørensen

10 years ago

Permalink

Post by Steffen W SÃ¸rensen
Into which pool does such user data (buckets and objects) gets stored and possible howto direct user data into a dedicated pool?
pool name category KB objects clones degraded unfound rd rd KB wr wr KB
.intent-log - 0 0 0 0 0 0 0 0 0
.log - 1 1 0 0 0 0 0 2 2
.rgw - 1 4 0 0 0 17 14 10 4
.rgw.buckets - 0 0 0 0 0 0 0 0 0
.rgw.buckets.extra - 0 0 0 0 0 0 0 0 0
.rgw.buckets.index - 0 1 0 0 0 2 0 3 0
.rgw.control - 0 8 0 0 0 0 0 0 0
.rgw.gc - 0 32 0 0 0 8302 8302 5556 0
.rgw.root - 1 3 0 0 0 929 618 3 3
.usage - 0 0 0 0 0 0 0 0 0
.users - 1 1 0 0 0 6 4 5 3
.users.email - 1 1 0 0 0 3 2 5 3
.users.swift - 0 0 0 0 0 0 0 0 0
.users.uid - 1 2 0 0 0 65 54 16 4

...

So it's mapped into a zone (at least on my Giant version 0.87)
and in my simple non-federated config it's in the default region+zone:

[***@rgw ~]# radosgw-admin region list
{ "default_info": { "default_region": "default"},
"regions": [
"default"]}
[***@rgw ~]# radosgw-admin zone list
{ "zones": [
"default"]}

[***@rgw ~]# radosgw-admin region get
{ "name": "default",
"api_name": "",
"is_master": "true",
"endpoints": [],
"master_zone": "",
"zones": [
{ "name": "default",
"endpoints": [],
"log_meta": "false",
"log_data": "false"}],
"placement_targets": [
{ "name": "default-placement",
"tags": []}],
"default_placement": "default-placement"}

[***@rgw ~]# radosgw-admin zone get
{ "domain_root": ".rgw",
"control_pool": ".rgw.control",
"gc_pool": ".rgw.gc",
"log_pool": ".log",
"intent_log_pool": ".intent-log",
"usage_log_pool": ".usage",
"user_keys_pool": ".users",
"user_email_pool": ".users.email",
"user_swift_pool": ".users.swift",
"user_uid_pool": ".users.uid",
"system_key": { "access_key": "",
"secret_key": ""},
"placement_pools": [
{ "key": "default-placement",
"val": { "index_pool": ".rgw.buckets.index",
"data_pool": ".rgw.buckets",
"data_extra_pool": ".rgw.buckets.extra"}}]}

and my user if associated with the default region+zone, thus it's data goes into .rgw.buckets + .rgw.buckets.index [+ .rgw.buckets.extra]
Buckets seems a naming container at the radosgw level, above the underlying Ceph pool abstraction level, 'just' providing object persistence for radosgw abstraction/object FS on top of Ceph Pools... I think.

So more users associated with same region+zone can share buckets+objects?

Would be nice with a drawing showing abstractions at the different levels possible woth links to details on administration at different levels :)
Lot of stuff to grasp for a newbie just in the need of a S3 service for an App usage :)

/Steffen

Steffen W Sørensen

10 years ago

Permalink

Sorry forgot to send to the list...

...