zhbingyin
9 years ago
In multipart upload of S3, the checksum algorithm is different from that in swift.
Including the following page for your convenience.
http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb
åå§é®ä»¶
å件人:Saverio ***@gmail.com
æ¶ä»¶äºº:***@h3c.com
æé:ceph-***@lists.ceph.comceph-***@lists.ceph.com
åéæ¶éŽ:2016幎5æ11æ¥(åšäž)â22:15
äž»é¢:Re: [ceph-users] RadosGW - Problems running the S3 and SWIFT API atthe same time
It does not work also the way around: If I upload a file with the swift client with the -S options to force swift to make multipart: swift upload -S 1000000 multipart 180.mp4 Then I am not able to read the file with S3 s3cmd get s3://multipart/180.mp4 download: 's3://multipart/180.mp4' - './180.mp4' [1 of 1] download: 's3://multipart/180.mp4' - './180.mp4' [1 of 1] 38818503 of 38818503 100% in 1s 27.32 MB/s done WARNING: MD5 signatures do not match: computed=961f154cc78c7bf1be3b4009c29e5a68, received=d41d8cd98f00b204e9800998ecf8427e Saverio 2016-05-11 16:07 GMT+02:00 Saverio Proto ***@gmail.com: Thank you. It is exactly a problem with multipart. So I tried two clients (s3cmd and rclone). When you upload a file in S3 using multipart, you are not able to read anymore this object with the SWIFT API because the md5 check fails. Saverio 2016-05-09 12:00 GMT+02:00 Xusangdi ***@h3c.com: Hi, I'm not running a cluster as yours, but I don't think the issue is caused by you using 2 APIs at the same time. IIRC the dash thing is append by S3 multipart upload, with a following digit indicating the number of parts. You may want to check this reported in s3cmd community: https://sourceforge.net/p/s3tools/bugs/123/ and some basic info from Amazon: http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html Hope this helps :D Regards, ---Sandy -----Original Message----- From: ceph-users [mailto:ceph-users-***@lists.ceph.com] On Behalf Of Saverio Proto Sent: Monday, May 09, 2016 4:42 PM To: ceph-***@lists.ceph.com Subject: Re: [ceph-users] RadosGW - Problems running the S3 and SWIFT API at the same time I try to simplify the question to get some feedback. Is anyone running the RadosGW in production with S3 and SWIFT API active at the same time ? thank you ! Saverio 2016-05-06 11:39 GMT+02:00 Saverio Proto ***@gmail.com: Hello, We have been running the Rados GW with the S3 API and we did not have problems for more than a year. We recently enabled also the SWIFT API for our users. radosgw --version ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403) The idea is that each user of the system is free of choosing the S3 client or the SWIFT client to access the same container/buckets. Please tell us if this is possible by design or if we are doing something wrong. We have now a problem that some files wrote in the past with S3, cannot be read with the SWIFT API because the md5sum always fails. I am able to reproduce the bug in this way: We have this file googlebooks-fre-all-2gram-20120701-ts.gz and we know the correct md5 is 1c8113d2bd21232688221ec74dccff3a You can download the same file here: https://www.dropbox.com/s/auq16vdv2maw4p7/googlebooks-fre-all-2gram-20 120701-ts.gz?dl=0 rclone mkdir lss3:bugreproduce rclone copy googlebooks-fre-all-2gram-20120701-ts.gz lss3:bugreproduce The file is successfully uploaded. At this point I can succesfully download again the file: rclone copy lss3:bugreproduce/googlebooks-fre-all-2gram-20120701-ts.gz test.gz but not with swift: swift download googlebooks-ngrams-gz fre/googlebooks-fre-all-2gram-20120701-ts.gz Error downloading object 'googlebooks-ngrams-gz/fre/googlebooks-fre-all-2gram-20120701-ts.gz': u'Error downloading fre/googlebooks-fre-all-2gram-20120701-ts.gz: md5sum != etag, 1c8113d2bd21232688221ec74dccff3a != 1a209a31b4ac3eb923fac5e8d194d9d3-2' Also I found strange the dash character '-' at the end of the md5 that is trying to compare. Of course upload a file with the swift client and redownloading the same file just works. Should I open a bug for the radosgw on http://tracker.ceph.com/ ? thank you Saverio _______________________________________________ ceph-users mailing list ceph-***@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ------------------------------------------------------------------------------------------------------------------------------------- æ¬é®ä»¶åå ¶é件å«ææå·åäžéä¿¡ææ¯æéå ¬åžçä¿å¯ä¿¡æ¯ïŒä» éäºåéç»äžé¢å°åäžååº ç䞪人æ矀ç»ãçŠæ¢ä»»äœå ¶ä»äººä»¥ä»»äœåœ¢åŒäœ¿çšïŒå æ¬äœäžéäºå šéšæéšåå°æ³é²ãå€å¶ã ææ£åïŒæ¬é®ä»¶äžçä¿¡æ¯ãåŠææšéæ¶äºæ¬é®ä»¶ïŒè¯·æšç«å³çµè¯æé®ä»¶éç¥å件人并å é€æ¬ é®ä»¶ïŒ This e-mail and its attachments contain confidential information from H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
Including the following page for your convenience.
http://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb
åå§é®ä»¶
å件人:Saverio ***@gmail.com
æ¶ä»¶äºº:***@h3c.com
æé:ceph-***@lists.ceph.comceph-***@lists.ceph.com
åéæ¶éŽ:2016幎5æ11æ¥(åšäž)â22:15
äž»é¢:Re: [ceph-users] RadosGW - Problems running the S3 and SWIFT API atthe same time
It does not work also the way around: If I upload a file with the swift client with the -S options to force swift to make multipart: swift upload -S 1000000 multipart 180.mp4 Then I am not able to read the file with S3 s3cmd get s3://multipart/180.mp4 download: 's3://multipart/180.mp4' - './180.mp4' [1 of 1] download: 's3://multipart/180.mp4' - './180.mp4' [1 of 1] 38818503 of 38818503 100% in 1s 27.32 MB/s done WARNING: MD5 signatures do not match: computed=961f154cc78c7bf1be3b4009c29e5a68, received=d41d8cd98f00b204e9800998ecf8427e Saverio 2016-05-11 16:07 GMT+02:00 Saverio Proto ***@gmail.com: Thank you. It is exactly a problem with multipart. So I tried two clients (s3cmd and rclone). When you upload a file in S3 using multipart, you are not able to read anymore this object with the SWIFT API because the md5 check fails. Saverio 2016-05-09 12:00 GMT+02:00 Xusangdi ***@h3c.com: Hi, I'm not running a cluster as yours, but I don't think the issue is caused by you using 2 APIs at the same time. IIRC the dash thing is append by S3 multipart upload, with a following digit indicating the number of parts. You may want to check this reported in s3cmd community: https://sourceforge.net/p/s3tools/bugs/123/ and some basic info from Amazon: http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html Hope this helps :D Regards, ---Sandy -----Original Message----- From: ceph-users [mailto:ceph-users-***@lists.ceph.com] On Behalf Of Saverio Proto Sent: Monday, May 09, 2016 4:42 PM To: ceph-***@lists.ceph.com Subject: Re: [ceph-users] RadosGW - Problems running the S3 and SWIFT API at the same time I try to simplify the question to get some feedback. Is anyone running the RadosGW in production with S3 and SWIFT API active at the same time ? thank you ! Saverio 2016-05-06 11:39 GMT+02:00 Saverio Proto ***@gmail.com: Hello, We have been running the Rados GW with the S3 API and we did not have problems for more than a year. We recently enabled also the SWIFT API for our users. radosgw --version ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403) The idea is that each user of the system is free of choosing the S3 client or the SWIFT client to access the same container/buckets. Please tell us if this is possible by design or if we are doing something wrong. We have now a problem that some files wrote in the past with S3, cannot be read with the SWIFT API because the md5sum always fails. I am able to reproduce the bug in this way: We have this file googlebooks-fre-all-2gram-20120701-ts.gz and we know the correct md5 is 1c8113d2bd21232688221ec74dccff3a You can download the same file here: https://www.dropbox.com/s/auq16vdv2maw4p7/googlebooks-fre-all-2gram-20 120701-ts.gz?dl=0 rclone mkdir lss3:bugreproduce rclone copy googlebooks-fre-all-2gram-20120701-ts.gz lss3:bugreproduce The file is successfully uploaded. At this point I can succesfully download again the file: rclone copy lss3:bugreproduce/googlebooks-fre-all-2gram-20120701-ts.gz test.gz but not with swift: swift download googlebooks-ngrams-gz fre/googlebooks-fre-all-2gram-20120701-ts.gz Error downloading object 'googlebooks-ngrams-gz/fre/googlebooks-fre-all-2gram-20120701-ts.gz': u'Error downloading fre/googlebooks-fre-all-2gram-20120701-ts.gz: md5sum != etag, 1c8113d2bd21232688221ec74dccff3a != 1a209a31b4ac3eb923fac5e8d194d9d3-2' Also I found strange the dash character '-' at the end of the md5 that is trying to compare. Of course upload a file with the swift client and redownloading the same file just works. Should I open a bug for the radosgw on http://tracker.ceph.com/ ? thank you Saverio _______________________________________________ ceph-users mailing list ceph-***@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ------------------------------------------------------------------------------------------------------------------------------------- æ¬é®ä»¶åå ¶é件å«ææå·åäžéä¿¡ææ¯æéå ¬åžçä¿å¯ä¿¡æ¯ïŒä» éäºåéç»äžé¢å°åäžååº ç䞪人æ矀ç»ãçŠæ¢ä»»äœå ¶ä»äººä»¥ä»»äœåœ¢åŒäœ¿çšïŒå æ¬äœäžéäºå šéšæéšåå°æ³é²ãå€å¶ã ææ£åïŒæ¬é®ä»¶äžçä¿¡æ¯ãåŠææšéæ¶äºæ¬é®ä»¶ïŒè¯·æšç«å³çµè¯æé®ä»¶éç¥å件人并å é€æ¬ é®ä»¶ïŒ This e-mail and its attachments contain confidential information from H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!