[ceph-users] RGW snapshots

[Mike Bryant]

Hi,
is there any way to create snapshots of individual buckets, that can
be restored from piecemeal?
i.e. if someone deletes objects by mistake?

Cheers
Mike


--
Mike Bryant | Systems Administrator | Ocado Technology
mike.bryant at ocado.com | 01707 382148 | www.ocadotechnology.com
--
Notice: This email is confidential and may contain copyright material of
Ocado Limited (the "Company"). Opinions and views expressed in this message
may not necessarily reflect the opinions and views of the Company.

If you are not the intended recipient, please notify us immediately and
delete all copies of this message. Please note that it is your
responsibility to scan this message for viruses.

Company reg. no. 3875000.

Ocado Limited
Titan Court
3 Bishops Square
Hatfield Business Park
Hatfield
Herts
AL10 9NE

[Craig Lewis]

I've looked into this a bit, and the best I've come up with is to
snapshot all of the RGW pools. I asked a similar question before:
http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

I am planning to have a 2nd cluster for disaster recovery, with some
in-house geo-replication.

I haven't actually tried this yet. I just setup my development cluster,
and this is on my list of things to test. The basic idea:

* Disable geo-replication
* Snapshot the Disaster Recovery cluster manually
* Rollback all of the RGW pools to the snapshot I want to restore from
* Manually restore objects from the Disaster Recovery cluster to the
Production Cluster, probably using s3cmd
* Return all of the RGW pools to the most recent snapshot
* Re-enable geo-replication


I have several layers of safety above this, so this process is meant to
be a last resort after several layers of human+code errors. In theory,
it shouldn't ever happen, but we all know how that goes.


I would like to discuss how RadosGW snapshots might work, but there
doesn't seem to be much interest at this time. The ability to use
RadosGW snapshots is somewhat niche.




*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email clewis at centraldesktop.com <mailto:clewis at centraldesktop.com>

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website <http://www.centraldesktop.com/> | Twitter
<http://www.twitter.com/centraldesktop> | Facebook
<http://www.facebook.com/CentralDesktop> | LinkedIn
<http://www.linkedin.com/groups?gid=147417> | Blog
<http://cdblog.centraldesktop.com/>

Post by Mike Bryant
Hi,
is there any way to create snapshots of individual buckets, that can
be restored from piecemeal?
i.e. if someone deletes objects by mistake?
Cheers
Mike
--
Mike Bryant | Systems Administrator | Ocado Technology
mike.bryant at ocado.com | 01707 382148 | www.ocadotechnology.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20130624/74046aea/attachment.htm>

[Anatol]

Post by Craig Lewis
I've looked into this a bit, and the
best I've come up with is to snapshot all of the RGW pools.? I

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

Post by Craig Lewis
I am planning to have a 2nd cluster for disaster recovery, with
some in-house geo-replication.?
I haven't actually tried this yet.? I just setup my development
cluster, and this is on my list of things to test.? The basic
Disable geo-replication
Snapshot the Disaster Recovery cluster manually
Rollback all of the RGW pools to the snapshot I want to
restore from
Manually restore objects from the Disaster Recovery cluster
to the Production Cluster, probably using s3cmd
Return all of the RGW pools to the most recent snapshot
Re-enable geo-replication

Hi,

I am facing the same issue (I want to create backups of buckets, and I would
like to do this from snapshots so that I can get a consistent backup without
stopping write operations).

Any news on how this worked out? Any new features that make this process
simpler?

Thanks!

[Craig Lewis]

I'm planning to add RGW Snapshots to Giant:
https://wiki.ceph.com/Planning/Blueprints/Giant/rgw%3A_Snapshots .

I'm still getting my development environment setup, so I don't have
anything on github yet. If you're interested in testing, my repo is
github.com/clewis/ceph

*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email clewis at centraldesktop.com <mailto:clewis at centraldesktop.com>

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website <http://www.centraldesktop.com/> | Twitter
<http://www.twitter.com/centraldesktop> | Facebook
<http://www.facebook.com/CentralDesktop> | LinkedIn
<http://www.linkedin.com/groups?gid=147417> | Blog
<http://cdblog.centraldesktop.com/>

Post by Craig Lewis

Post by Craig Lewis
I've looked into this a bit, and the
best I've come up with is to snapshot all of the RGW pools. I

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

Post by Craig Lewis
I am planning to have a 2nd cluster for disaster recovery, with
some in-house geo-replication.
I haven't actually tried this yet. I just setup my development
cluster, and this is on my list of things to test. The basic
Disable geo-replication
Snapshot the Disaster Recovery cluster manually
Rollback all of the RGW pools to the snapshot I want to
restore from
Manually restore objects from the Disaster Recovery cluster
to the Production Cluster, probably using s3cmd
Return all of the RGW pools to the most recent snapshot
Re-enable geo-replication

Hi,
I am facing the same issue (I want to create backups of buckets, and I would
like to do this from snapshots so that I can get a consistent backup without
stopping write operations).
Any news on how this worked out? Any new features that make this process
simpler?
Thanks!
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140319/1fefde5f/attachment.htm>

[Craig Lewis]

I'm planning to add RGW Snapshots to Giant:
https://wiki.ceph.com/Planning/Blueprints/Giant/rgw%3A_Snapshots .

I'm still getting my development environment setup, so I don't have
anything on github yet. If you're interested in testing, my repo is
github.com/clewis/ceph

*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email clewis at centraldesktop.com <mailto:clewis at centraldesktop.com>

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website <http://www.centraldesktop.com/> | Twitter
<http://www.twitter.com/centraldesktop> | Facebook
<http://www.facebook.com/CentralDesktop> | LinkedIn
<http://www.linkedin.com/groups?gid=147417> | Blog
<http://cdblog.centraldesktop.com/>

Post by Craig Lewis

Post by Craig Lewis
I've looked into this a bit, and the
best I've come up with is to snapshot all of the RGW pools. I

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

Post by Craig Lewis
I am planning to have a 2nd cluster for disaster recovery, with
some in-house geo-replication.
I haven't actually tried this yet. I just setup my development
cluster, and this is on my list of things to test. The basic
Disable geo-replication
Snapshot the Disaster Recovery cluster manually
Rollback all of the RGW pools to the snapshot I want to
restore from
Manually restore objects from the Disaster Recovery cluster
to the Production Cluster, probably using s3cmd
Return all of the RGW pools to the most recent snapshot
Re-enable geo-replication

Hi,
I am facing the same issue (I want to create backups of buckets, and I would
like to do this from snapshots so that I can get a consistent backup without
stopping write operations).
Any news on how this worked out? Any new features that make this process
simpler?
Thanks!
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140319/1fefde5f/attachment-0002.htm>

[Craig Lewis]

I'm planning to add RGW Snapshots to Giant:
https://wiki.ceph.com/Planning/Blueprints/Giant/rgw%3A_Snapshots .

I'm still getting my development environment setup, so I don't have
anything on github yet. If you're interested in testing, my repo is
github.com/clewis/ceph

*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email clewis at centraldesktop.com <mailto:clewis at centraldesktop.com>

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website <http://www.centraldesktop.com/> | Twitter
<http://www.twitter.com/centraldesktop> | Facebook
<http://www.facebook.com/CentralDesktop> | LinkedIn
<http://www.linkedin.com/groups?gid=147417> | Blog
<http://cdblog.centraldesktop.com/>

Post by Craig Lewis

Post by Craig Lewis
I've looked into this a bit, and the
best I've come up with is to snapshot all of the RGW pools. I

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

Post by Craig Lewis
I am planning to have a 2nd cluster for disaster recovery, with
some in-house geo-replication.
I haven't actually tried this yet. I just setup my development
cluster, and this is on my list of things to test. The basic
Disable geo-replication
Snapshot the Disaster Recovery cluster manually
Rollback all of the RGW pools to the snapshot I want to
restore from
Manually restore objects from the Disaster Recovery cluster
to the Production Cluster, probably using s3cmd
Return all of the RGW pools to the most recent snapshot
Re-enable geo-replication

Hi,
I am facing the same issue (I want to create backups of buckets, and I would
like to do this from snapshots so that I can get a consistent backup without
stopping write operations).
Any news on how this worked out? Any new features that make this process
simpler?
Thanks!
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140319/1fefde5f/attachment-0003.htm>

[Craig Lewis]

I'm planning to add RGW Snapshots to Giant:
https://wiki.ceph.com/Planning/Blueprints/Giant/rgw%3A_Snapshots .

I'm still getting my development environment setup, so I don't have
anything on github yet. If you're interested in testing, my repo is
github.com/clewis/ceph

*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email clewis at centraldesktop.com <mailto:clewis at centraldesktop.com>

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website <http://www.centraldesktop.com/> | Twitter
<http://www.twitter.com/centraldesktop> | Facebook
<http://www.facebook.com/CentralDesktop> | LinkedIn
<http://www.linkedin.com/groups?gid=147417> | Blog
<http://cdblog.centraldesktop.com/>

Post by Craig Lewis

Post by Craig Lewis
I've looked into this a bit, and the
best I've come up with is to snapshot all of the RGW pools. I

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

Post by Craig Lewis
I am planning to have a 2nd cluster for disaster recovery, with
some in-house geo-replication.
I haven't actually tried this yet. I just setup my development
cluster, and this is on my list of things to test. The basic
Disable geo-replication
Snapshot the Disaster Recovery cluster manually
Rollback all of the RGW pools to the snapshot I want to
restore from
Manually restore objects from the Disaster Recovery cluster
to the Production Cluster, probably using s3cmd
Return all of the RGW pools to the most recent snapshot
Re-enable geo-replication

Hi,
I am facing the same issue (I want to create backups of buckets, and I would
like to do this from snapshots so that I can get a consistent backup without
stopping write operations).
Any news on how this worked out? Any new features that make this process
simpler?
Thanks!
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140319/1fefde5f/attachment-0004.htm>

[Anatol]

Post by Craig Lewis
I've looked into this a bit, and the
best I've come up with is to snapshot all of the RGW pools.? I

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

Post by Craig Lewis
I am planning to have a 2nd cluster for disaster recovery, with
some in-house geo-replication.?
I haven't actually tried this yet.? I just setup my development
cluster, and this is on my list of things to test.? The basic
Disable geo-replication
Snapshot the Disaster Recovery cluster manually
Rollback all of the RGW pools to the snapshot I want to
restore from
Manually restore objects from the Disaster Recovery cluster
to the Production Cluster, probably using s3cmd
Return all of the RGW pools to the most recent snapshot
Re-enable geo-replication

Hi,

I am facing the same issue (I want to create backups of buckets, and I would
like to do this from snapshots so that I can get a consistent backup without
stopping write operations).

Any news on how this worked out? Any new features that make this process
simpler?

Thanks!

[Anatol]

Post by Craig Lewis
I've looked into this a bit, and the
best I've come up with is to snapshot all of the RGW pools.? I

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

Post by Craig Lewis
I am planning to have a 2nd cluster for disaster recovery, with
some in-house geo-replication.?
I haven't actually tried this yet.? I just setup my development
cluster, and this is on my list of things to test.? The basic
Disable geo-replication
Snapshot the Disaster Recovery cluster manually
Rollback all of the RGW pools to the snapshot I want to
restore from
Manually restore objects from the Disaster Recovery cluster
to the Production Cluster, probably using s3cmd
Return all of the RGW pools to the most recent snapshot
Re-enable geo-replication

Hi,

I am facing the same issue (I want to create backups of buckets, and I would
like to do this from snapshots so that I can get a consistent backup without
stopping write operations).

Any news on how this worked out? Any new features that make this process
simpler?

Thanks!

[Anatol]

Post by Craig Lewis
I've looked into this a bit, and the
best I've come up with is to snapshot all of the RGW pools.? I

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

Post by Craig Lewis
I am planning to have a 2nd cluster for disaster recovery, with
some in-house geo-replication.?
I haven't actually tried this yet.? I just setup my development
cluster, and this is on my list of things to test.? The basic
Disable geo-replication
Snapshot the Disaster Recovery cluster manually
Rollback all of the RGW pools to the snapshot I want to
restore from
Manually restore objects from the Disaster Recovery cluster
to the Production Cluster, probably using s3cmd
Return all of the RGW pools to the most recent snapshot
Re-enable geo-replication

Hi,

I am facing the same issue (I want to create backups of buckets, and I would
like to do this from snapshots so that I can get a consistent backup without
stopping write operations).

Any news on how this worked out? Any new features that make this process
simpler?

Thanks!

[Mike Bryant]

Hi,
is there any way to create snapshots of individual buckets, that can
be restored from piecemeal?
i.e. if someone deletes objects by mistake?

Cheers
Mike


--
Mike Bryant | Systems Administrator | Ocado Technology
mike.bryant at ocado.com | 01707 382148 | www.ocadotechnology.com
--
Notice: This email is confidential and may contain copyright material of
Ocado Limited (the "Company"). Opinions and views expressed in this message
may not necessarily reflect the opinions and views of the Company.

If you are not the intended recipient, please notify us immediately and
delete all copies of this message. Please note that it is your
responsibility to scan this message for viruses.

Company reg. no. 3875000.

Ocado Limited
Titan Court
3 Bishops Square
Hatfield Business Park
Hatfield
Herts
AL10 9NE

[Craig Lewis]

I've looked into this a bit, and the best I've come up with is to
snapshot all of the RGW pools. I asked a similar question before:
http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

I am planning to have a 2nd cluster for disaster recovery, with some
in-house geo-replication.

I haven't actually tried this yet. I just setup my development cluster,
and this is on my list of things to test. The basic idea:

* Disable geo-replication
* Snapshot the Disaster Recovery cluster manually
* Rollback all of the RGW pools to the snapshot I want to restore from
* Manually restore objects from the Disaster Recovery cluster to the
Production Cluster, probably using s3cmd
* Return all of the RGW pools to the most recent snapshot
* Re-enable geo-replication


I have several layers of safety above this, so this process is meant to
be a last resort after several layers of human+code errors. In theory,
it shouldn't ever happen, but we all know how that goes.


I would like to discuss how RadosGW snapshots might work, but there
doesn't seem to be much interest at this time. The ability to use
RadosGW snapshots is somewhat niche.




*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email clewis at centraldesktop.com <mailto:clewis at centraldesktop.com>

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website <http://www.centraldesktop.com/> | Twitter
<http://www.twitter.com/centraldesktop> | Facebook
<http://www.facebook.com/CentralDesktop> | LinkedIn
<http://www.linkedin.com/groups?gid=147417> | Blog
<http://cdblog.centraldesktop.com/>

Post by Mike Bryant
Hi,
is there any way to create snapshots of individual buckets, that can
be restored from piecemeal?
i.e. if someone deletes objects by mistake?
Cheers
Mike
--
Mike Bryant | Systems Administrator | Ocado Technology
mike.bryant at ocado.com | 01707 382148 | www.ocadotechnology.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20130624/74046aea/attachment-0002.htm>

[Mike Bryant]

Hi,
is there any way to create snapshots of individual buckets, that can
be restored from piecemeal?
i.e. if someone deletes objects by mistake?

Cheers
Mike


--
Mike Bryant | Systems Administrator | Ocado Technology
mike.bryant at ocado.com | 01707 382148 | www.ocadotechnology.com
--
Notice: This email is confidential and may contain copyright material of
Ocado Limited (the "Company"). Opinions and views expressed in this message
may not necessarily reflect the opinions and views of the Company.

If you are not the intended recipient, please notify us immediately and
delete all copies of this message. Please note that it is your
responsibility to scan this message for viruses.

Company reg. no. 3875000.

Ocado Limited
Titan Court
3 Bishops Square
Hatfield Business Park
Hatfield
Herts
AL10 9NE

[Craig Lewis]

I've looked into this a bit, and the best I've come up with is to
snapshot all of the RGW pools. I asked a similar question before:
http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

I am planning to have a 2nd cluster for disaster recovery, with some
in-house geo-replication.

I haven't actually tried this yet. I just setup my development cluster,
and this is on my list of things to test. The basic idea:

* Disable geo-replication
* Snapshot the Disaster Recovery cluster manually
* Rollback all of the RGW pools to the snapshot I want to restore from
* Manually restore objects from the Disaster Recovery cluster to the
Production Cluster, probably using s3cmd
* Return all of the RGW pools to the most recent snapshot
* Re-enable geo-replication


I have several layers of safety above this, so this process is meant to
be a last resort after several layers of human+code errors. In theory,
it shouldn't ever happen, but we all know how that goes.


I would like to discuss how RadosGW snapshots might work, but there
doesn't seem to be much interest at this time. The ability to use
RadosGW snapshots is somewhat niche.




*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email clewis at centraldesktop.com <mailto:clewis at centraldesktop.com>

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website <http://www.centraldesktop.com/> | Twitter
<http://www.twitter.com/centraldesktop> | Facebook
<http://www.facebook.com/CentralDesktop> | LinkedIn
<http://www.linkedin.com/groups?gid=147417> | Blog
<http://cdblog.centraldesktop.com/>

Post by Mike Bryant
Hi,
is there any way to create snapshots of individual buckets, that can
be restored from piecemeal?
i.e. if someone deletes objects by mistake?
Cheers
Mike
--
Mike Bryant | Systems Administrator | Ocado Technology
mike.bryant at ocado.com | 01707 382148 | www.ocadotechnology.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20130624/74046aea/attachment-0003.htm>

[Mike Bryant]

Hi,
is there any way to create snapshots of individual buckets, that can
be restored from piecemeal?
i.e. if someone deletes objects by mistake?

Cheers
Mike


--
Mike Bryant | Systems Administrator | Ocado Technology
mike.bryant at ocado.com | 01707 382148 | www.ocadotechnology.com
--
Notice: This email is confidential and may contain copyright material of
Ocado Limited (the "Company"). Opinions and views expressed in this message
may not necessarily reflect the opinions and views of the Company.

If you are not the intended recipient, please notify us immediately and
delete all copies of this message. Please note that it is your
responsibility to scan this message for viruses.

Company reg. no. 3875000.

Ocado Limited
Titan Court
3 Bishops Square
Hatfield Business Park
Hatfield
Herts
AL10 9NE

[Craig Lewis]

I've looked into this a bit, and the best I've come up with is to
snapshot all of the RGW pools. I asked a similar question before:
http://comments.gmane.org/gmane.comp.file-systems.ceph.user/855

I am planning to have a 2nd cluster for disaster recovery, with some
in-house geo-replication.

I haven't actually tried this yet. I just setup my development cluster,
and this is on my list of things to test. The basic idea:

* Disable geo-replication
* Snapshot the Disaster Recovery cluster manually
* Rollback all of the RGW pools to the snapshot I want to restore from
* Manually restore objects from the Disaster Recovery cluster to the
Production Cluster, probably using s3cmd
* Return all of the RGW pools to the most recent snapshot
* Re-enable geo-replication


I have several layers of safety above this, so this process is meant to
be a last resort after several layers of human+code errors. In theory,
it shouldn't ever happen, but we all know how that goes.


I would like to discuss how RadosGW snapshots might work, but there
doesn't seem to be much interest at this time. The ability to use
RadosGW snapshots is somewhat niche.




*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email clewis at centraldesktop.com <mailto:clewis at centraldesktop.com>

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website <http://www.centraldesktop.com/> | Twitter
<http://www.twitter.com/centraldesktop> | Facebook
<http://www.facebook.com/CentralDesktop> | LinkedIn
<http://www.linkedin.com/groups?gid=147417> | Blog
<http://cdblog.centraldesktop.com/>

Post by Mike Bryant
Hi,
is there any way to create snapshots of individual buckets, that can
be restored from piecemeal?
i.e. if someone deletes objects by mistake?
Cheers
Mike
--
Mike Bryant | Systems Administrator | Ocado Technology
mike.bryant at ocado.com | 01707 382148 | www.ocadotechnology.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20130624/74046aea/attachment-0004.htm>