Discussion:
[ceph-users] Ceph or Gluster for implementing big NAS
Premysl Kouril
2018-11-12 11:42:46 UTC
Permalink
Hi,

We are planning to build NAS solution which will be primarily used via NFS
and CIFS and workloads ranging from various archival application to more
“real-time processing”. The NAS will not be used as a block storage for
virtual machines, so the access really will always be file oriented.

We are considering primarily two designs and I’d like to kindly ask for any
thoughts, views, insights, experiences.

Both designs utilize “distributed storage software at some level”. Both
designs would be built from commodity servers and should scale as we grow.
Both designs involve virtualization for instantiating "access virtual
machines" which will be serving the NFS and CIFS protocol - so in this
sense the access layer is decoupled from the data layer itself.

First design is based on a distributed filesystem like Gluster or CephFS.
We would deploy this software on those commodity servers and mount the
resultant filesystem on the “access virtual machines” and they would be
serving the mounted filesystem via NFS/CIFS.

Second design is based on distributed block storage using CEPH. So we would
build distributed block storage on those commodity servers, and then, via
virtualization (like OpenStack Cinder) we would allocate the block storage
into the access VM. Inside the access VM we would deploy ZFS which would
aggregate block storage into a single filesystem. And this filesystem would
be served via NFS/CIFS from the very same VM.


Any advices and insights highly appreciated


Cheers,

Prema
Kevin Olbrich
2018-11-12 14:01:12 UTC
Permalink
Hi!

ZFS won't play nice on ceph. Best would be to mount CephFS directly with
the ceph-fuse driver on the endpoint.
If you definitely want to put a storage gateway between the data and the
compute nodes, then go with nfs-ganesha which can export CephFS directly
without local ("proxy") mount.

I had such a setup with nfs and switched to mount CephFS directly. If using
NFS with the same data, you must make sure your HA works well to avoid data
corruption.
With ceph-fuse you directly connect to the cluster, one component less that
breaks.

Kevin

Am Mo., 12. Nov. 2018 um 12:44 Uhr schrieb Premysl Kouril <
Post by Premysl Kouril
Hi,
We are planning to build NAS solution which will be primarily used via NFS
and CIFS and workloads ranging from various archival application to more
“real-time processing”. The NAS will not be used as a block storage for
virtual machines, so the access really will always be file oriented.
We are considering primarily two designs and I’d like to kindly ask for
any thoughts, views, insights, experiences.
Both designs utilize “distributed storage software at some level”. Both
designs would be built from commodity servers and should scale as we grow.
Both designs involve virtualization for instantiating "access virtual
machines" which will be serving the NFS and CIFS protocol - so in this
sense the access layer is decoupled from the data layer itself.
First design is based on a distributed filesystem like Gluster or CephFS.
We would deploy this software on those commodity servers and mount the
resultant filesystem on the “access virtual machines” and they would be
serving the mounted filesystem via NFS/CIFS.
Second design is based on distributed block storage using CEPH. So we
would build distributed block storage on those commodity servers, and then,
via virtualization (like OpenStack Cinder) we would allocate the block
storage into the access VM. Inside the access VM we would deploy ZFS which
would aggregate block storage into a single filesystem. And this filesystem
would be served via NFS/CIFS from the very same VM.
Any advices and insights highly appreciated
Cheers,
Prema
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Dan van der Ster
2018-11-12 14:07:34 UTC
Permalink
We've done ZFS on RBD in a VM, exported via NFS, for a couple years.
It's very stable and if your use-case permits you can set zfs
sync=disabled to get very fast write performance that's tough to beat.

But if you're building something new today and have *only* the NAS
use-case then it would make better sense to try CephFS first and see
if it works for you.

-- Dan
Hi!
ZFS won't play nice on ceph. Best would be to mount CephFS directly with the ceph-fuse driver on the endpoint.
If you definitely want to put a storage gateway between the data and the compute nodes, then go with nfs-ganesha which can export CephFS directly without local ("proxy") mount.
I had such a setup with nfs and switched to mount CephFS directly. If using NFS with the same data, you must make sure your HA works well to avoid data corruption.
With ceph-fuse you directly connect to the cluster, one component less that breaks.
Kevin
Hi,
We are planning to build NAS solution which will be primarily used via NFS and CIFS and workloads ranging from various archival application to more “real-time processing”. The NAS will not be used as a block storage for virtual machines, so the access really will always be file oriented.
We are considering primarily two designs and I’d like to kindly ask for any thoughts, views, insights, experiences.
Both designs utilize “distributed storage software at some level”. Both designs would be built from commodity servers and should scale as we grow. Both designs involve virtualization for instantiating "access virtual machines" which will be serving the NFS and CIFS protocol - so in this sense the access layer is decoupled from the data layer itself.
First design is based on a distributed filesystem like Gluster or CephFS. We would deploy this software on those commodity servers and mount the resultant filesystem on the “access virtual machines” and they would be serving the mounted filesystem via NFS/CIFS.
Second design is based on distributed block storage using CEPH. So we would build distributed block storage on those commodity servers, and then, via virtualization (like OpenStack Cinder) we would allocate the block storage into the access VM. Inside the access VM we would deploy ZFS which would aggregate block storage into a single filesystem. And this filesystem would be served via NFS/CIFS from the very same VM.
Any advices and insights highly appreciated
Cheers,
Prema
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Kevin Olbrich
2018-11-12 14:16:55 UTC
Permalink
Hi Dan,

ZFS without sync would be very much identical to ext2/ext4 without journals
or XFS with barriers disabled.
The ARC cache in ZFS is awesome but disbaling sync on ZFS is a very high
risk (using ext4 with kvm-mode unsafe would be similar I think).

Also, ZFS only works as expected with scheduler set to noop as it is
optimized to consume whole, non-shared devices.

Just my 2 cents ;-)

Kevin


Am Mo., 12. Nov. 2018 um 15:08 Uhr schrieb Dan van der Ster <
Post by Dan van der Ster
We've done ZFS on RBD in a VM, exported via NFS, for a couple years.
It's very stable and if your use-case permits you can set zfs
sync=disabled to get very fast write performance that's tough to beat.
But if you're building something new today and have *only* the NAS
use-case then it would make better sense to try CephFS first and see
if it works for you.
-- Dan
Post by Kevin Olbrich
Hi!
ZFS won't play nice on ceph. Best would be to mount CephFS directly with
the ceph-fuse driver on the endpoint.
Post by Kevin Olbrich
If you definitely want to put a storage gateway between the data and the
compute nodes, then go with nfs-ganesha which can export CephFS directly
without local ("proxy") mount.
Post by Kevin Olbrich
I had such a setup with nfs and switched to mount CephFS directly. If
using NFS with the same data, you must make sure your HA works well to
avoid data corruption.
Post by Kevin Olbrich
With ceph-fuse you directly connect to the cluster, one component less
that breaks.
Post by Kevin Olbrich
Kevin
Am Mo., 12. Nov. 2018 um 12:44 Uhr schrieb Premysl Kouril <
Post by Premysl Kouril
Hi,
We are planning to build NAS solution which will be primarily used via
NFS and CIFS and workloads ranging from various archival application to
more “real-time processing”. The NAS will not be used as a block storage
for virtual machines, so the access really will always be file oriented.
Post by Kevin Olbrich
Post by Premysl Kouril
We are considering primarily two designs and I’d like to kindly ask for
any thoughts, views, insights, experiences.
Post by Kevin Olbrich
Post by Premysl Kouril
Both designs utilize “distributed storage software at some level”. Both
designs would be built from commodity servers and should scale as we grow.
Both designs involve virtualization for instantiating "access virtual
machines" which will be serving the NFS and CIFS protocol - so in this
sense the access layer is decoupled from the data layer itself.
Post by Kevin Olbrich
Post by Premysl Kouril
First design is based on a distributed filesystem like Gluster or
CephFS. We would deploy this software on those commodity servers and mount
the resultant filesystem on the “access virtual machines” and they would be
serving the mounted filesystem via NFS/CIFS.
Post by Kevin Olbrich
Post by Premysl Kouril
Second design is based on distributed block storage using CEPH. So we
would build distributed block storage on those commodity servers, and then,
via virtualization (like OpenStack Cinder) we would allocate the block
storage into the access VM. Inside the access VM we would deploy ZFS which
would aggregate block storage into a single filesystem. And this filesystem
would be served via NFS/CIFS from the very same VM.
Post by Kevin Olbrich
Post by Premysl Kouril
Any advices and insights highly appreciated
Cheers,
Prema
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Premysl Kouril
2018-11-12 14:36:14 UTC
Permalink
Hi Kevin,

I should have also said, that we are internally inclined towards the
"monster VM" approach due to seemingly simpler architecture (data
distribution on block layer rather than on file system layer). So my
original question is more about comparing the two approaches (distribution
on block layer vs distribution on filesystem layer). "Monster VM" approach
being the one where we just keep mounting block volumes to a single VM with
normal non-distributed filesystem and then exporting via NFS/CIFS.

Regards,
Prema
Post by Kevin Olbrich
Hi Dan,
ZFS without sync would be very much identical to ext2/ext4 without
journals or XFS with barriers disabled.
The ARC cache in ZFS is awesome but disbaling sync on ZFS is a very high
risk (using ext4 with kvm-mode unsafe would be similar I think).
Also, ZFS only works as expected with scheduler set to noop as it is
optimized to consume whole, non-shared devices.
Just my 2 cents ;-)
Kevin
Am Mo., 12. Nov. 2018 um 15:08 Uhr schrieb Dan van der Ster <
Post by Dan van der Ster
We've done ZFS on RBD in a VM, exported via NFS, for a couple years.
It's very stable and if your use-case permits you can set zfs
sync=disabled to get very fast write performance that's tough to beat.
But if you're building something new today and have *only* the NAS
use-case then it would make better sense to try CephFS first and see
if it works for you.
-- Dan
Post by Kevin Olbrich
Hi!
ZFS won't play nice on ceph. Best would be to mount CephFS directly
with the ceph-fuse driver on the endpoint.
Post by Kevin Olbrich
If you definitely want to put a storage gateway between the data and
the compute nodes, then go with nfs-ganesha which can export CephFS
directly without local ("proxy") mount.
Post by Kevin Olbrich
I had such a setup with nfs and switched to mount CephFS directly. If
using NFS with the same data, you must make sure your HA works well to
avoid data corruption.
Post by Kevin Olbrich
With ceph-fuse you directly connect to the cluster, one component less
that breaks.
Post by Kevin Olbrich
Kevin
Am Mo., 12. Nov. 2018 um 12:44 Uhr schrieb Premysl Kouril <
Post by Premysl Kouril
Hi,
We are planning to build NAS solution which will be primarily used via
NFS and CIFS and workloads ranging from various archival application to
more “real-time processing”. The NAS will not be used as a block storage
for virtual machines, so the access really will always be file oriented.
Post by Kevin Olbrich
Post by Premysl Kouril
We are considering primarily two designs and I’d like to kindly ask
for any thoughts, views, insights, experiences.
Post by Kevin Olbrich
Post by Premysl Kouril
Both designs utilize “distributed storage software at some level”.
Both designs would be built from commodity servers and should scale as we
grow. Both designs involve virtualization for instantiating "access virtual
machines" which will be serving the NFS and CIFS protocol - so in this
sense the access layer is decoupled from the data layer itself.
Post by Kevin Olbrich
Post by Premysl Kouril
First design is based on a distributed filesystem like Gluster or
CephFS. We would deploy this software on those commodity servers and mount
the resultant filesystem on the “access virtual machines” and they would be
serving the mounted filesystem via NFS/CIFS.
Post by Kevin Olbrich
Post by Premysl Kouril
Second design is based on distributed block storage using CEPH. So we
would build distributed block storage on those commodity servers, and then,
via virtualization (like OpenStack Cinder) we would allocate the block
storage into the access VM. Inside the access VM we would deploy ZFS which
would aggregate block storage into a single filesystem. And this filesystem
would be served via NFS/CIFS from the very same VM.
Post by Kevin Olbrich
Post by Premysl Kouril
Any advices and insights highly appreciated
Cheers,
Prema
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Ashley Merrick
2018-11-12 14:40:05 UTC
Permalink
My 2 cents would be depends how H/A you need.

Going with the monster VM you have a single point of failure and a single
point of network congestion.

If you go the CephFS route you remove that single point of failure if you
mount to clients directly. And also can remove that single point of network
congestion.

Guess depends on the performance and uptime required , as I’d say that
could factory into your decisions.
Post by Premysl Kouril
Hi Kevin,
I should have also said, that we are internally inclined towards the
"monster VM" approach due to seemingly simpler architecture (data
distribution on block layer rather than on file system layer). So my
original question is more about comparing the two approaches (distribution
on block layer vs distribution on filesystem layer). "Monster VM" approach
being the one where we just keep mounting block volumes to a single VM
with normal non-distributed filesystem and then exporting via NFS/CIFS.
Regards,
Prema
Post by Kevin Olbrich
Hi Dan,
ZFS without sync would be very much identical to ext2/ext4 without
journals or XFS with barriers disabled.
The ARC cache in ZFS is awesome but disbaling sync on ZFS is a very high
risk (using ext4 with kvm-mode unsafe would be similar I think).
Also, ZFS only works as expected with scheduler set to noop as it is
optimized to consume whole, non-shared devices.
Just my 2 cents ;-)
Kevin
Am Mo., 12. Nov. 2018 um 15:08 Uhr schrieb Dan van der Ster <
Post by Dan van der Ster
We've done ZFS on RBD in a VM, exported via NFS, for a couple years.
It's very stable and if your use-case permits you can set zfs
sync=disabled to get very fast write performance that's tough to beat.
But if you're building something new today and have *only* the NAS
use-case then it would make better sense to try CephFS first and see
if it works for you.
-- Dan
Post by Kevin Olbrich
Hi!
ZFS won't play nice on ceph. Best would be to mount CephFS directly
with the ceph-fuse driver on the endpoint.
Post by Kevin Olbrich
If you definitely want to put a storage gateway between the data and
the compute nodes, then go with nfs-ganesha which can export CephFS
directly without local ("proxy") mount.
Post by Kevin Olbrich
I had such a setup with nfs and switched to mount CephFS directly. If
using NFS with the same data, you must make sure your HA works well to
avoid data corruption.
Post by Kevin Olbrich
With ceph-fuse you directly connect to the cluster, one component less
that breaks.
Post by Kevin Olbrich
Kevin
Am Mo., 12. Nov. 2018 um 12:44 Uhr schrieb Premysl Kouril <
Post by Premysl Kouril
Hi,
We are planning to build NAS solution which will be primarily used
via NFS and CIFS and workloads ranging from various archival application to
more “real-time processing”. The NAS will not be used as a block storage
for virtual machines, so the access really will always be file oriented.
Post by Kevin Olbrich
Post by Premysl Kouril
We are considering primarily two designs and I’d like to kindly ask
for any thoughts, views, insights, experiences.
Post by Kevin Olbrich
Post by Premysl Kouril
Both designs utilize “distributed storage software at some level”.
Both designs would be built from commodity servers and should scale as we
grow. Both designs involve virtualization for instantiating "access virtual
machines" which will be serving the NFS and CIFS protocol - so in this
sense the access layer is decoupled from the data layer itself.
Post by Kevin Olbrich
Post by Premysl Kouril
First design is based on a distributed filesystem like Gluster or
CephFS. We would deploy this software on those commodity servers and mount
the resultant filesystem on the “access virtual machines” and they would be
serving the mounted filesystem via NFS/CIFS.
Post by Kevin Olbrich
Post by Premysl Kouril
Second design is based on distributed block storage using CEPH. So we
would build distributed block storage on those commodity servers, and then,
via virtualization (like OpenStack Cinder) we would allocate the block
storage into the access VM. Inside the access VM we would deploy ZFS which
would aggregate block storage into a single filesystem. And this filesystem
would be served via NFS/CIFS from the very same VM.
Post by Kevin Olbrich
Post by Premysl Kouril
Any advices and insights highly appreciated
Cheers,
Prema
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Premysl Kouril
2018-11-12 14:51:40 UTC
Permalink
Some kind of single point will always be there I guess. Because even if we
go with the distributed filesystem, it will be mounted to the access VM and
this access VM will be providing NFS/CIFS protocol access. So this machine
is single point of failure (indeed we would be running two of them for
active-passive HA setup. In case of distributed filesystem approach the
failure of the access VM would mean re-mounting the filesystem on the
passive access VM. In case of "monster VM" approach, in case of the VM
failure it would mean reattaching all block volumes to a new VM.
Post by Ashley Merrick
My 2 cents would be depends how H/A you need.
Going with the monster VM you have a single point of failure and a single
point of network congestion.
If you go the CephFS route you remove that single point of failure if you
mount to clients directly. And also can remove that single point of network
congestion.
Guess depends on the performance and uptime required , as I’d say that
could factory into your decisions.
Post by Premysl Kouril
Hi Kevin,
I should have also said, that we are internally inclined towards the
"monster VM" approach due to seemingly simpler architecture (data
distribution on block layer rather than on file system layer). So my
original question is more about comparing the two approaches (distribution
on block layer vs distribution on filesystem layer). "Monster VM" approach
being the one where we just keep mounting block volumes to a single VM
with normal non-distributed filesystem and then exporting via NFS/CIFS.
Regards,
Prema
Post by Kevin Olbrich
Hi Dan,
ZFS without sync would be very much identical to ext2/ext4 without
journals or XFS with barriers disabled.
The ARC cache in ZFS is awesome but disbaling sync on ZFS is a very high
risk (using ext4 with kvm-mode unsafe would be similar I think).
Also, ZFS only works as expected with scheduler set to noop as it is
optimized to consume whole, non-shared devices.
Just my 2 cents ;-)
Kevin
Am Mo., 12. Nov. 2018 um 15:08 Uhr schrieb Dan van der Ster <
Post by Dan van der Ster
We've done ZFS on RBD in a VM, exported via NFS, for a couple years.
It's very stable and if your use-case permits you can set zfs
sync=disabled to get very fast write performance that's tough to beat.
But if you're building something new today and have *only* the NAS
use-case then it would make better sense to try CephFS first and see
if it works for you.
-- Dan
Post by Kevin Olbrich
Hi!
ZFS won't play nice on ceph. Best would be to mount CephFS directly
with the ceph-fuse driver on the endpoint.
Post by Kevin Olbrich
If you definitely want to put a storage gateway between the data and
the compute nodes, then go with nfs-ganesha which can export CephFS
directly without local ("proxy") mount.
Post by Kevin Olbrich
I had such a setup with nfs and switched to mount CephFS directly. If
using NFS with the same data, you must make sure your HA works well to
avoid data corruption.
Post by Kevin Olbrich
With ceph-fuse you directly connect to the cluster, one component
less that breaks.
Post by Kevin Olbrich
Kevin
Am Mo., 12. Nov. 2018 um 12:44 Uhr schrieb Premysl Kouril <
Post by Premysl Kouril
Hi,
We are planning to build NAS solution which will be primarily used
via NFS and CIFS and workloads ranging from various archival application to
more “real-time processing”. The NAS will not be used as a block storage
for virtual machines, so the access really will always be file oriented.
Post by Kevin Olbrich
Post by Premysl Kouril
We are considering primarily two designs and I’d like to kindly ask
for any thoughts, views, insights, experiences.
Post by Kevin Olbrich
Post by Premysl Kouril
Both designs utilize “distributed storage software at some level”.
Both designs would be built from commodity servers and should scale as we
grow. Both designs involve virtualization for instantiating "access virtual
machines" which will be serving the NFS and CIFS protocol - so in this
sense the access layer is decoupled from the data layer itself.
Post by Kevin Olbrich
Post by Premysl Kouril
First design is based on a distributed filesystem like Gluster or
CephFS. We would deploy this software on those commodity servers and mount
the resultant filesystem on the “access virtual machines” and they would be
serving the mounted filesystem via NFS/CIFS.
Post by Kevin Olbrich
Post by Premysl Kouril
Second design is based on distributed block storage using CEPH. So
we would build distributed block storage on those commodity servers, and
then, via virtualization (like OpenStack Cinder) we would allocate the
block storage into the access VM. Inside the access VM we would deploy ZFS
which would aggregate block storage into a single filesystem. And this
filesystem would be served via NFS/CIFS from the very same VM.
Post by Kevin Olbrich
Post by Premysl Kouril
Any advices and insights highly appreciated
Cheers,
Prema
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Ashley Merrick
2018-11-12 14:54:06 UTC
Permalink
Does your use case mean you need something like nfs/cifs and can’t use
CephFS mount directly?

Has been quite a few advances in that area with quotas and user management
in recent versions.

But obviously all depends on your use case at client end.
Post by Premysl Kouril
Some kind of single point will always be there I guess. Because even if we
go with the distributed filesystem, it will be mounted to the access VM and
this access VM will be providing NFS/CIFS protocol access. So this machine
is single point of failure (indeed we would be running two of them for
active-passive HA setup. In case of distributed filesystem approach the
failure of the access VM would mean re-mounting the filesystem on the
passive access VM. In case of "monster VM" approach, in case of the VM
failure it would mean reattaching all block volumes to a new VM.
Post by Ashley Merrick
My 2 cents would be depends how H/A you need.
Going with the monster VM you have a single point of failure and a single
point of network congestion.
If you go the CephFS route you remove that single point of failure if you
mount to clients directly. And also can remove that single point of network
congestion.
Guess depends on the performance and uptime required , as I’d say that
could factory into your decisions.
Post by Premysl Kouril
Hi Kevin,
I should have also said, that we are internally inclined towards the
"monster VM" approach due to seemingly simpler architecture (data
distribution on block layer rather than on file system layer). So my
original question is more about comparing the two approaches (distribution
on block layer vs distribution on filesystem layer). "Monster VM" approach
being the one where we just keep mounting block volumes to a single VM
with normal non-distributed filesystem and then exporting via NFS/CIFS.
Regards,
Prema
Post by Kevin Olbrich
Hi Dan,
ZFS without sync would be very much identical to ext2/ext4 without
journals or XFS with barriers disabled.
The ARC cache in ZFS is awesome but disbaling sync on ZFS is a very
high risk (using ext4 with kvm-mode unsafe would be similar I think).
Also, ZFS only works as expected with scheduler set to noop as it is
optimized to consume whole, non-shared devices.
Just my 2 cents ;-)
Kevin
Am Mo., 12. Nov. 2018 um 15:08 Uhr schrieb Dan van der Ster <
Post by Dan van der Ster
We've done ZFS on RBD in a VM, exported via NFS, for a couple years.
It's very stable and if your use-case permits you can set zfs
sync=disabled to get very fast write performance that's tough to beat.
But if you're building something new today and have *only* the NAS
use-case then it would make better sense to try CephFS first and see
if it works for you.
-- Dan
Post by Kevin Olbrich
Hi!
ZFS won't play nice on ceph. Best would be to mount CephFS directly
with the ceph-fuse driver on the endpoint.
Post by Kevin Olbrich
If you definitely want to put a storage gateway between the data and
the compute nodes, then go with nfs-ganesha which can export CephFS
directly without local ("proxy") mount.
Post by Kevin Olbrich
I had such a setup with nfs and switched to mount CephFS directly.
If using NFS with the same data, you must make sure your HA works well to
avoid data corruption.
Post by Kevin Olbrich
With ceph-fuse you directly connect to the cluster, one component
less that breaks.
Post by Kevin Olbrich
Kevin
Am Mo., 12. Nov. 2018 um 12:44 Uhr schrieb Premysl Kouril <
Post by Premysl Kouril
Hi,
We are planning to build NAS solution which will be primarily used
via NFS and CIFS and workloads ranging from various archival application to
more “real-time processing”. The NAS will not be used as a block storage
for virtual machines, so the access really will always be file oriented.
Post by Kevin Olbrich
Post by Premysl Kouril
We are considering primarily two designs and I’d like to kindly ask
for any thoughts, views, insights, experiences.
Post by Kevin Olbrich
Post by Premysl Kouril
Both designs utilize “distributed storage software at some level”.
Both designs would be built from commodity servers and should scale as we
grow. Both designs involve virtualization for instantiating "access virtual
machines" which will be serving the NFS and CIFS protocol - so in this
sense the access layer is decoupled from the data layer itself.
Post by Kevin Olbrich
Post by Premysl Kouril
First design is based on a distributed filesystem like Gluster or
CephFS. We would deploy this software on those commodity servers and mount
the resultant filesystem on the “access virtual machines” and they would be
serving the mounted filesystem via NFS/CIFS.
Post by Kevin Olbrich
Post by Premysl Kouril
Second design is based on distributed block storage using CEPH. So
we would build distributed block storage on those commodity servers, and
then, via virtualization (like OpenStack Cinder) we would allocate the
block storage into the access VM. Inside the access VM we would deploy ZFS
which would aggregate block storage into a single filesystem. And this
filesystem would be served via NFS/CIFS from the very same VM.
Post by Kevin Olbrich
Post by Premysl Kouril
Any advices and insights highly appreciated
Cheers,
Prema
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Premysl Kouril
2018-11-12 14:58:51 UTC
Permalink
Yes, the access VM layer is there because of multi-tenancy - we need to
provide parts of the storage into different private environments (can be
potentially on private IP addresses). And we need both - NFS as well as
CIFS.
Post by Ashley Merrick
Does your use case mean you need something like nfs/cifs and can’t use
CephFS mount directly?
Has been quite a few advances in that area with quotas and user management
in recent versions.
But obviously all depends on your use case at client end.
Post by Premysl Kouril
Some kind of single point will always be there I guess. Because even if
we go with the distributed filesystem, it will be mounted to the access VM
and this access VM will be providing NFS/CIFS protocol access. So this
machine is single point of failure (indeed we would be running two of them
for active-passive HA setup. In case of distributed filesystem approach the
failure of the access VM would mean re-mounting the filesystem on the
passive access VM. In case of "monster VM" approach, in case of the VM
failure it would mean reattaching all block volumes to a new VM.
Post by Ashley Merrick
My 2 cents would be depends how H/A you need.
Going with the monster VM you have a single point of failure and a
single point of network congestion.
If you go the CephFS route you remove that single point of failure if
you mount to clients directly. And also can remove that single point of
network congestion.
Guess depends on the performance and uptime required , as I’d say that
could factory into your decisions.
On Mon, 12 Nov 2018 at 10:36 PM, Premysl Kouril <
Post by Premysl Kouril
Hi Kevin,
I should have also said, that we are internally inclined towards the
"monster VM" approach due to seemingly simpler architecture (data
distribution on block layer rather than on file system layer). So my
original question is more about comparing the two approaches (distribution
on block layer vs distribution on filesystem layer). "Monster VM" approach
being the one where we just keep mounting block volumes to a single VM
with normal non-distributed filesystem and then exporting via NFS/CIFS.
Regards,
Prema
Post by Kevin Olbrich
Hi Dan,
ZFS without sync would be very much identical to ext2/ext4 without
journals or XFS with barriers disabled.
The ARC cache in ZFS is awesome but disbaling sync on ZFS is a very
high risk (using ext4 with kvm-mode unsafe would be similar I think).
Also, ZFS only works as expected with scheduler set to noop as it is
optimized to consume whole, non-shared devices.
Just my 2 cents ;-)
Kevin
Am Mo., 12. Nov. 2018 um 15:08 Uhr schrieb Dan van der Ster <
Post by Dan van der Ster
We've done ZFS on RBD in a VM, exported via NFS, for a couple years.
It's very stable and if your use-case permits you can set zfs
sync=disabled to get very fast write performance that's tough to beat.
But if you're building something new today and have *only* the NAS
use-case then it would make better sense to try CephFS first and see
if it works for you.
-- Dan
Post by Kevin Olbrich
Hi!
ZFS won't play nice on ceph. Best would be to mount CephFS directly
with the ceph-fuse driver on the endpoint.
Post by Kevin Olbrich
If you definitely want to put a storage gateway between the data
and the compute nodes, then go with nfs-ganesha which can export CephFS
directly without local ("proxy") mount.
Post by Kevin Olbrich
I had such a setup with nfs and switched to mount CephFS directly.
If using NFS with the same data, you must make sure your HA works well to
avoid data corruption.
Post by Kevin Olbrich
With ceph-fuse you directly connect to the cluster, one component
less that breaks.
Post by Kevin Olbrich
Kevin
Am Mo., 12. Nov. 2018 um 12:44 Uhr schrieb Premysl Kouril <
Post by Premysl Kouril
Hi,
We are planning to build NAS solution which will be primarily used
via NFS and CIFS and workloads ranging from various archival application to
more “real-time processing”. The NAS will not be used as a block storage
for virtual machines, so the access really will always be file oriented.
Post by Kevin Olbrich
Post by Premysl Kouril
We are considering primarily two designs and I’d like to kindly
ask for any thoughts, views, insights, experiences.
Post by Kevin Olbrich
Post by Premysl Kouril
Both designs utilize “distributed storage software at some level”.
Both designs would be built from commodity servers and should scale as we
grow. Both designs involve virtualization for instantiating "access virtual
machines" which will be serving the NFS and CIFS protocol - so in this
sense the access layer is decoupled from the data layer itself.
Post by Kevin Olbrich
Post by Premysl Kouril
First design is based on a distributed filesystem like Gluster or
CephFS. We would deploy this software on those commodity servers and mount
the resultant filesystem on the “access virtual machines” and they would be
serving the mounted filesystem via NFS/CIFS.
Post by Kevin Olbrich
Post by Premysl Kouril
Second design is based on distributed block storage using CEPH. So
we would build distributed block storage on those commodity servers, and
then, via virtualization (like OpenStack Cinder) we would allocate the
block storage into the access VM. Inside the access VM we would deploy ZFS
which would aggregate block storage into a single filesystem. And this
filesystem would be served via NFS/CIFS from the very same VM.
Post by Kevin Olbrich
Post by Premysl Kouril
Any advices and insights highly appreciated
Cheers,
Prema
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Dan van der Ster
2018-11-12 15:17:15 UTC
Permalink
Hi,
Is it identical?
In the places we use sync=disabled (e.g. analysis scratch areas),
we're totally content with losing last x seconds/minutes of writes,
and understood that on-disk consistency is not impacted.
Cheers,Dan
Post by Kevin Olbrich
Hi Dan,
ZFS without sync would be very much identical to ext2/ext4 without journals or XFS with barriers disabled.
The ARC cache in ZFS is awesome but disbaling sync on ZFS is a very high risk (using ext4 with kvm-mode unsafe would be similar I think).
Also, ZFS only works as expected with scheduler set to noop as it is optimized to consume whole, non-shared devices.
Just my 2 cents ;-)
Kevin
Post by Dan van der Ster
We've done ZFS on RBD in a VM, exported via NFS, for a couple years.
It's very stable and if your use-case permits you can set zfs
sync=disabled to get very fast write performance that's tough to beat.
But if you're building something new today and have *only* the NAS
use-case then it would make better sense to try CephFS first and see
if it works for you.
-- Dan
Hi!
ZFS won't play nice on ceph. Best would be to mount CephFS directly with the ceph-fuse driver on the endpoint.
If you definitely want to put a storage gateway between the data and the compute nodes, then go with nfs-ganesha which can export CephFS directly without local ("proxy") mount.
I had such a setup with nfs and switched to mount CephFS directly. If using NFS with the same data, you must make sure your HA works well to avoid data corruption.
With ceph-fuse you directly connect to the cluster, one component less that breaks.
Kevin
Hi,
We are planning to build NAS solution which will be primarily used via NFS and CIFS and workloads ranging from various archival application to more “real-time processing”. The NAS will not be used as a block storage for virtual machines, so the access really will always be file oriented.
We are considering primarily two designs and I’d like to kindly ask for any thoughts, views, insights, experiences.
Both designs utilize “distributed storage software at some level”. Both designs would be built from commodity servers and should scale as we grow. Both designs involve virtualization for instantiating "access virtual machines" which will be serving the NFS and CIFS protocol - so in this sense the access layer is decoupled from the data layer itself.
First design is based on a distributed filesystem like Gluster or CephFS. We would deploy this software on those commodity servers and mount the resultant filesystem on the “access virtual machines” and they would be serving the mounted filesystem via NFS/CIFS.
Second design is based on distributed block storage using CEPH. So we would build distributed block storage on those commodity servers, and then, via virtualization (like OpenStack Cinder) we would allocate the block storage into the access VM. Inside the access VM we would deploy ZFS which would aggregate block storage into a single filesystem. And this filesystem would be served via NFS/CIFS from the very same VM.
Any advices and insights highly appreciated
Cheers,
Prema
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Loading...