[ceph-users] fibre channel as ceph storage interconnect

[Schlacta, Christ]

Is it possible? Can I use fibre channel to interconnect my ceph OSDs?
Intuition tells me it should be possible, yet experience (Mostly with
fibre channel) tells me no. I don't know enough about how ceph works
to know for sure. All my googling returns results about using ceph as
a BACKEND for exporting fibre channel LUNs, which is, sadly, not what
I'm looking for at the moment.

[Adrian Saul]

I could only see it being done using FCIP as the OSD processes use IP to communicate.

I guess it would depend on why you are looking to use something like FC instead of Ethernet or IB.

-----Original Message-----
Schlacta, Christ
Sent: Friday, 22 April 2016 1:12 PM
Subject: [ceph-users] fibre channel as ceph storage interconnect
Is it possible? Can I use fibre channel to interconnect my ceph OSDs?
Intuition tells me it should be possible, yet experience (Mostly with fibre
channel) tells me no. I don't know enough about how ceph works to know
for sure. All my googling returns results about using ceph as a BACKEND for
exporting fibre channel LUNs, which is, sadly, not what I'm looking for at the
moment.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Confidentiality: This email and any attachments are confidential and may be subject to copyright, legal or some other professional privilege. They are intended solely for the attention and use of the named addressee(s). They may only be copied, distributed or disclosed with the consent of the copyright owner. If you have received this email by mistake or by breach of the confidentiality clause, please notify the sender immediately by return email and delete or destroy all copies of the email. Any confidentiality, privilege or copyright is not waived or lost because this email has been sent to you by mistake.

[Schlacta, Christ]

My primary motivations are:
Most of my systems that I want to use with ceph already have fibre
Chantel cards and infrastructure, and more infrastructure is
incredibly cheap compared to infiniband or {1,4}0gbe cards and
infrastructure
Most of my systems are expansion slot constrained, and I'd be forced
to pick one or the other anyway.

In today’s world, OSDs communicate via IP and only IP*. Some FiberChannel
switches and HBAs support IP-over-FC, but it’s about 0.000002% of the FC
deployments.
Therefore, one could technically use FC, but it does’t appear to offer
enough benefit to OSD operations to justify the unique architecture.
What is your motivation to leverage FC behind OSDs?
-Paul
*Ceph on native Infiniband may be available some day, but it seems
impractical with the current releases. IP-over-IB is also known to work.
Is it possible? Can I use fibre channel to interconnect my ceph OSDs?
Intuition tells me it should be possible, yet experience (Mostly with
fibre channel) tells me no. I don't know enough about how ceph works
to know for sure. All my googling returns results about using ceph as
a BACKEND for exporting fibre channel LUNs, which is, sadly, not what
I'm looking for at the moment.

[Schlacta, Christ]

So it looks like because of reply to going to the user instead of the
list (Seriously, somebody needs to fix the list headers) by default,
the thread got kinda messed up, so I apologize if you're using a
threaded reader. That said, here goes.

from the responses I've gotten, it looks like there's no viable option
to use fibre channel as an interconnect between the nodes of the
cluster.
Would it be worth while development effort to establish a block
protocol between the nodes so that something like fibre channel could
be used to communicate internally? Unless I'm waaay wrong (And I'm
seldom *that* wrong), it would not be worth the effort. I won't even
feature request it. Looks like I'll have to look into infiniband or
CE, and possibly migrate away from Fibre Channel, even though it kinda
just works, and therefore I really like it :(

Post by Schlacta, Christ
Most of my systems that I want to use with ceph already have fibre
Chantel cards and infrastructure, and more infrastructure is
incredibly cheap compared to infiniband or {1,4}0gbe cards and
infrastructure
Most of my systems are expansion slot constrained, and I'd be forced
to pick one or the other anyway.

In today’s world, OSDs communicate via IP and only IP*. Some FiberChannel
switches and HBAs support IP-over-FC, but it’s about 0.000002% of the FC
deployments.
Therefore, one could technically use FC, but it does’t appear to offer
enough benefit to OSD operations to justify the unique architecture.
What is your motivation to leverage FC behind OSDs?
-Paul
*Ceph on native Infiniband may be available some day, but it seems
impractical with the current releases. IP-over-IB is also known to work.
Is it possible? Can I use fibre channel to interconnect my ceph OSDs?
Intuition tells me it should be possible, yet experience (Mostly with
fibre channel) tells me no. I don't know enough about how ceph works
to know for sure. All my googling returns results about using ceph as
a BACKEND for exporting fibre channel LUNs, which is, sadly, not what
I'm looking for at the moment.

[Paul Evans]

On Apr 21, 2016, at 11:10 PM, Schlacta, Christ <***@aarcane.org<mailto:***@aarcane.org>> wrote:

Would it be worth while development effort to establish a block
protocol between the nodes so that something like fibre channel could
be used to communicate internally?

With 25/100 Ethernet & IB becoming available now...and with some effort to integrate IB and ceph already completed, I can’t see FC for OSDs getting any traction.

Looks like I'll have to look into infiniband or
CE, and possibly migrate away from Fibre Channel, even though it kinda
just works, and therefore I really like it :(

If by CE you’re referring to Converged Enhanced Ethernet and it’s brethren (DCB/DCE), those technologies should be transparent to Ceph and provide some degree of improvement over standard Ethernet behaviors. YMMV.

As for FC ‘just works
’ +1 (but I really don’t want to inspire a flame war on the topic)

- Paul

[Adrian Saul]

from the responses I've gotten, it looks like there's no viable option to use
fibre channel as an interconnect between the nodes of the cluster.
Would it be worth while development effort to establish a block protocol
between the nodes so that something like fibre channel could be used to
communicate internally? Unless I'm waaay wrong (And I'm seldom *that*
wrong), it would not be worth the effort. I won't even feature request it.
Looks like I'll have to look into infiniband or CE, and possibly migrate away
from Fibre Channel, even though it kinda just works, and therefore I really
like it :(

I would think even conceptually it would be a mess - FC as a peer to peer network fabric might be useful (in many ways I like it a lot better than Ethernet), but you would have to develop an entire transport protocol over it (the normal SCSI model would be useless) for Ceph and then write that in to replace any of the network code in the existing Ceph code base.

A lot of work for something that is probably easier done swapping your FC HBAs for 10G NICs or IB HBAs.

Post by Schlacta, Christ
Most of my systems that I want to use with ceph already have fibre
Chantel cards and infrastructure, and more infrastructure is
incredibly cheap compared to infiniband or {1,4}0gbe cards and
infrastructure Most of my systems are expansion slot constrained, and
I'd be forced to pick one or the other anyway.

In today’s world, OSDs communicate via IP and only IP*. Some
FiberChannel switches and HBAs support IP-over-FC, but it’s about
0.000002% of the FC deployments.
Therefore, one could technically use FC, but it does’t appear to
offer enough benefit to OSD operations to justify the unique architecture.
What is your motivation to leverage FC behind OSDs?
-Paul
*Ceph on native Infiniband may be available some day, but it seems
impractical with the current releases. IP-over-IB is also known to work.
Is it possible? Can I use fibre channel to interconnect my ceph OSDs?
Intuition tells me it should be possible, yet experience (Mostly with
fibre channel) tells me no. I don't know enough about how ceph works
to know for sure. All my googling returns results about using ceph
as a BACKEND for exporting fibre channel LUNs, which is, sadly, not
what I'm looking for at the moment.

_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Confidentiality: This email and any attachments are confidential and may be subject to copyright, legal or some other professional privilege. They are intended solely for the attention and use of the named addressee(s). They may only be copied, distributed or disclosed with the consent of the copyright owner. If you have received this email by mistake or by breach of the confidentiality clause, please notify the sender immediately by return email and delete or destroy all copies of the email. Any confidentiality, privilege or copyright is not waived or lost because this email has been sent to you by mistake.

[LOPEZ Jean-Charles]

Hi,

your easiest way here if you want to use your FC hardware is to do IP over FC so that you can leverage the existing FC HBA in your servers but stick to IP as a communication layer. FC here would just be a low latency transport/encapsulation layer.

I’ve played with this gazillion years ago (early 2000’s) when I was with working with EMC and Brocade.

The question here is that I’m not sure who still supports IP over FC and it would be dependent on your FC HBAs, FC switches/directors.

For information, here is a link for AIX setup https://www.ibm.com/support/knowledgecenter/#!/ssw_aix_71/com.ibm.aix.networkcomm/fibrechan_intro.htm

Post by Schlacta, Christ
Is it possible? Can I use fibre channel to interconnect my ceph OSDs?
Intuition tells me it should be possible, yet experience (Mostly with
fibre channel) tells me no. I don't know enough about how ceph works
to know for sure. All my googling returns results about using ceph as
a BACKEND for exporting fibre channel LUNs, which is, sadly, not what
I'm looking for at the moment.
_______________________________________________
ceph-users mailing list
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

JC