[ceph-users] Supermicro server 5019D8-TR12P for new Ceph cluster

Discussion:

Michal Zacek

2018-11-13 13:42:58 UTC

Hello,

what do you think about this Supermicro server:
http://www.supermicro.com/products/system/1U/5019/SSG-5019D8-TR12P.cfm ?
We are considering about eight or ten server each with twelve 10TB SATA
drives, one m.2 SSD and 64GB RAM. Public and cluster network will be
10Gbit/s. The question is if one**Intel XEON D-2146NT wit eight cores
(16 with HT) will be enough for 12 SAT disks. Cluster will be used for
storing pictures. File size from 1MB to 2TB ;-).

Thanks,
Michal

Ashley Merrick

2018-11-13 13:50:48 UTC

Permalink

Not sure about CPU, but I would definitely suggest more than 64GB of ram.

With the next release of Mimic the default memory will be set to 4GB per a
OSD (if I am correct), this only includes the bluestore layer, so id easily
expect to see you getting close to 64GB after OS cache's e.t.c, and the
last thing you want on a CEPH OSD box is a OOM.

Are you looking at near to cold storage for these photos? Or is it storage
for designers working out of programs which require low latency and quick
performance?

Post by Michal Zacek
Hello,
http://www.supermicro.com/products/system/1U/5019/SSG-5019D8-TR12P.cfm ?
We are considering about eight or ten server each with twelve 10TB SATA
drives, one m.2 SSD and 64GB RAM. Public and cluster network will be
10Gbit/s. The question is if one Intel XEON D-2146NT wit eight cores (16
with HT) will be enough for 12 SAT disks. Cluster will be used for storing
pictures. File size from 1MB to 2TB ;-).
Thanks,
Michal
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Michal Zacek

2018-11-13 14:11:51 UTC

Permalink

Hi,

The server support up to 128GB RAM, so upgrade RAM will not be problem.
The storage will be used for storing data from a microscopes. Users will
download data from the storage to local PC, make some changes and then
will upload data back to the storage. We want use the cluster for direct
computing in the future, but now we only need separate data from the
microscopes from normal office data. We are expecting up to 10TB
upload/download per day.

Michal

Post by Ashley Merrick
Not sure about CPU, but I would definitely suggest more than 64GB of ram.
With the next release of Mimic the default memory will be set to 4GB
per a OSD (if I am correct), this only includes the bluestore layer,
so id easily expect to see you getting close to 64GB after OS cache's
e.t.c, and the last thing you want on a CEPH OSD box is a OOM.
Are you looking at near to cold storage for these photos? Or is it
storage for designers working out of programs which require low
latency and quick performance?
Hello,
http://www.supermicro.com/products/system/1U/5019/SSG-5019D8-TR12P.cfm
? We are considering about eight or ten server each with twelve
10TB SATA drives, one m.2 SSD and 64GB RAM. Public and cluster
network will be 10Gbit/s. The question is if one**Intel XEON
D-2146NT wit eight cores (16 with HT) will be enough for 12 SAT
disks. Cluster will be used for storing pictures. File size from
1MB to 2TB ;-).
Thanks,
Michal
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Ashley Merrick

2018-11-13 14:15:03 UTC

Permalink

Iâd say them CPUâs should more than be fine for your use case and
requirements then.

You have more than one thread per an OSD which seems to be the ongoing
recommendation.

Post by Michal Zacek
Hi,
The server support up to 128GB RAM, so upgrade RAM will not be problem.
The storage will be used for storing data from a microscopes. Users will
download data from the storage to local PC, make some changes and then will
upload data back to the storage. We want use the cluster for direct
computing in the future, but now we only need separate data from the
microscopes from normal office data. We are expecting up to 10TB
upload/download per day.
Michal
Not sure about CPU, but I would definitely suggest more than 64GB of ram.
With the next release of Mimic the default memory will be set to 4GB per a
OSD (if I am correct), this only includes the bluestore layer, so id easily
expect to see you getting close to 64GB after OS cache's e.t.c, and the
last thing you want on a CEPH OSD box is a OOM.
Are you looking at near to cold storage for these photos? Or is it storage
for designers working out of programs which require low latency and quick
performance?

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Brady Deetz

2018-11-13 14:42:37 UTC

Permalink

I'd ensure the io performance you expect can be achieved. If your scopes
create tons of small files, you may have a problem. You mentioned 10TB/day.
But, what is the scope's expectation with regard to dumping the data to
network storage. For instance, does the scope function normally while it is
transferring data?

Post by Ashley Merrick
Iâd say them CPUâs should more than be fine for your use case and
requirements then.
You have more than one thread per an OSD which seems to be the ongoing
recommendation.

Post by Michal Zacek
Hi,
The server support up to 128GB RAM, so upgrade RAM will not be problem.
The storage will be used for storing data from a microscopes. Users will
download data from the storage to local PC, make some changes and then will
upload data back to the storage. We want use the cluster for direct
computing in the future, but now we only need separate data from the
microscopes from normal office data. We are expecting up to 10TB
upload/download per day.
Michal
Not sure about CPU, but I would definitely suggest more than 64GB of ram.
With the next release of Mimic the default memory will be set to 4GB per
a OSD (if I am correct), this only includes the bluestore layer, so id
easily expect to see you getting close to 64GB after OS cache's e.t.c, and
the last thing you want on a CEPH OSD box is a OOM.
Are you looking at near to cold storage for these photos? Or is it
storage for designers working out of programs which require low latency and
quick performance?

Post by Michal Zacek
Hello,
http://www.supermicro.com/products/system/1U/5019/SSG-5019D8-TR12P.cfm
? We are considering about eight or ten server each with twelve 10TB SATA
drives, one m.2 SSD and 64GB RAM. Public and cluster network will be
10Gbit/s. The question is if one Intel XEON D-2146NT wit eight cores
(16 with HT) will be enough for 12 SAT disks. Cluster will be used for
storing pictures. File size from 1MB to 2TB ;-).
Thanks,
Michal
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Martin Verges

2018-11-13 14:14:03 UTC

Permalink

Hello,

we believe 1 core or thread per OSD + 2-4 for OS and other services
are enough for most use cases, so yes. Same goes for 64 GB Ram, we
suggest ~4 G per OSD (12*4 = 48 GB) so 16 GB for the Linux is more
then enough. Buy good drives (ssd & hdd) to prevent performance
issues.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: ***@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx

Post by Michal Zacek
Hello,
http://www.supermicro.com/products/system/1U/5019/SSG-5019D8-TR12P.cfm ? We
are considering about eight or ten server each with twelve 10TB SATA drives,
one m.2 SSD and 64GB RAM. Public and cluster network will be 10Gbit/s. The
question is if one Intel XEON D-2146NT wit eight cores (16 with HT) will be
enough for 12 SAT disks. Cluster will be used for storing pictures. File
size from 1MB to 2TB ;-).
Thanks,
Michal
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com