[ceph-users] ceph-users Digest, Vol 70, Issue 23

[Lazuardi Nasution]

Hi Robin,

Do you mean that Cumulus quagga fork is FRRouting (https://frrouting.org/)?
As long as I know Cumulus using it now. What dummy interfaces do you mean?
Why did you use it instead of loopback address? Anyway, how can you isolate
between some kind of traffic to be not routable? On L2 implementation, I
separate two Ceph traffic (public and cluster) with other traffic by using
VLANs.

Best regards,

Date: Fri, 23 Nov 2018 00:29:17 +0000

Subject: Re: [ceph-users] Full L3 Ceph
Content-Type: text/plain; charset="us-ascii"

I'm looking example Ceph configuration and topology on full layer 3
networking deployment. Maybe all daemons can use loopback alias address

in

this case. But how to set cluster network and public network

configuration,

using supernet? I think using loopback alias address can prevent the
daemons down due to physical interfaces disconnection and can load

balance

traffic between physical interfaces without interfaces bonding, but with
ECMP.

I can say I've done something similar**, but I don't have access to that
environment or most*** of the configuration anymore.
One of the parts I do recall, was explicitly setting cluster_network
and public_network to empty strings, AND using public_addr+cluster_addr
instead, with routable addressing on dummy interfaces (NOT loopback).
- 99.9% IPv6 environment
- BGP everywhere
- The only IPv4 was on the outside of HAProxy for legacy IPv4 clients.
- Quanta switchgear running Cumulus Linux, 10Gbit ports
- Hosts running Cumulus quagga fork (REQUIRED)
- Host to 2xToR using IPv6 link-local addressing only
https://blog.ipspace.net/2015/02/bgp-configuration-made-simple-with.html
- Reliable ~19Gbit aggregate (2x10GBit)
- watch out for NIC overheating: no warning, just thermal throttle down
to ~2.5Gbit/port.
https://github.com/dreamhost/ceph-chef/tree/dokken
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[Robin H. Johnson]

Post by Lazuardi Nasution
Hi Robin,
Do you mean that Cumulus quagga fork is FRRouting (https://frrouting.org/)?
As long as I know Cumulus using it now.

I started this before Cumulus was fully shipping FRRouting; and used
their binaries.
Earlier versions of this:
https://docs.cumulusnetworks.com/display/ROH/Installing+the+Cumulus+Quagga+Package+on+a+Host+Server
Should be entirely possible w/ FRRouting now. VRFs as well for load
balancers is a huge help.

Post by Lazuardi Nasution
What dummy interfaces do you mean?

modprobe dummy

Post by Lazuardi Nasution
Why did you use it instead of loopback address?

Some applications (e.g. keepalived & BIRD) have hardcoded special
behavior for the 'lo' interface, with no easy way to work around that
behavior. Using Dummy prevents said special behavior. This isn't
directly relevant to Ceph itself, but matters on load balancers RGW
workloads for example.

Post by Lazuardi Nasution
Anyway, how can you isolate between some kind of traffic to be not routable?
On L2 implementation, I separate two Ceph traffic (public and cluster)
with other traffic by using VLANs.

In my past deployment, the cluster network was using IPv6 ULA addressing
to prevent leaks. QoS via DSCP as well to prioritize cluster traffic
over public traffic (Ceph already marks the heartbeat packets suitably).