Living in the underlay

Today we will focus on config, the funniest part of any IE track. In part 1 & 2 we cover fundamentals, now we understand how VxLAN works, how many addresses can we get and the different options of advertising MAC/IP information to peers. In this case we will start with Flood & Learn, I choose this one not only for being the first adopted but also to be the *poorly* documented on the web.

Note: If you want to find any VxLAN config info in NXOS I encourage you too look under N9K, since latest release of 7K will do the same and under 7K you will not find anything :)

Recommended reading: I've also mentioned this in the prep for DC IE track, but if you don't have the chance I really recommend Cisco Live Presentations, in this case BRKDCT-2404: VxLAN Deployment Models

Let's use this topology:

First we will define each component in the network:

H1 / H2 would be hosts in different VLAN, for this example VLAN 101 and 102.

L1 - L4 would be Nexus 5K running as VXLAN L2 GW.
L5 - L6 would be Nexus 7K running as VXLAN L3 GW

S1 / S2 will be the underlay L3 core between all the nodes, the L3 cloud also supporting multicast.

IGP + Multicast cloud

First things first, we need basic IGP reachability between our nodes plus multicast reachability. For multicast we are going to use static mapping for the RP and, in order to bring redundancy, we will deploy PhantomRP, of course you can use any other way that you want but, for the purpose of the example, static mapping is the simplest option. Here is the snippet of config in each device

L1 - L6

S1 / S2

Also the config for IGP is omitted here, since you can run whatever you want (also static routing right? yes! but lot of work), in our case we simply setup OSPF in area 0 and point to point interfaces in each link, the only consideration was the MTU, as you may recall from previous post of this series you will need to tune up MTU to be able to send a VXLAN packet inside, you can do the math by yourself (remember VXLAN adds an additional 8 byte field...) but lets assuming that anything beyond 1554 1600 would be ok.

Just a minor note on Naming conventions

Before taking any other step further we will need to clarify some naming conventions that cisco has impossed to us :)

If you recall correctly each host in the VXLAN network has a VTEP which establish the tunnel and is , in the end, the responsible of taking in/out encapsulated packet to our vxlan network... Well for Cisco is quite different, since they don't have any host here messing around with VXLAN for them VTEP is the device responsible for the encap/decap process, but if VTEP is the device how do I configure the interface on that device in which this magic occurs? and here we come with NVE (Network Virtual Interface) which is the logical interface where the previous mentioned magic of encap/decap actually takes place.

VXLAN L2 Gateway

Well here the fun begins, at L2 of course. I've stated the concept of VXLAN L2 GW and thats because it is so important to understand how it works and what it does, the L2 keyword is key to understand this.
Let's asume you are running a legacy network, with VLANs in place and you want to integrate to a new design with VXLAN. Of course that you want not just to allow inter-routing between them but also allow to place some hosts in your new network in the same segment as the legacy one. Nice scenario... but how we can accomplish this?

Essentially what we need is a device capable of taking L2 frames, let's say tagged since I state that we've VLANs in place, and put them in the same bridge domain with the VXLAN traffic. Using different words to say the same what we want to accomplish is to put VLAN traffic into the VXLAN VNI associated to that traffic and be able to flood interact between them.

Here I will post two suggested options for config, one at the interface level (i.e. config direct on interface that get the tagged frames) and one at the switch level (what will use in our 5K), Cisco refer to this config as VSI (VN-Segment service instance) and VLAN modes.

VSI CLI Mode (L1/L2)

For this config let's assume that L1/L2 leafs gets the frames on H1/H2 ports on a trunk interface and what we want to accomplish is the vlan-to-vni mapping.

VLAN CLI Mode

This mode is much more easier, and that is because what we do is just an association between a vlan and a vn-segment.

VXLAN L3 Gateway

So basically we cover full L2 reachability between VXLAN under same bridge domain, but what about inter VXLAN routing? Seems pretty obvious that we will need to add a gateway with interfaces in the interesting VXLAN segment to route among them and also like SVI we do have a BDI (Bridge Domain Interfaces). We will associate that BDI with our VNI and assign addressing to it in order to be the VXLAN gateway for that segment and also be able to route outside (also to other VXLAN segment if we have)


So if we dig into config we can see is pretty straight forward since the magic already happened, what that means is that we do the hardest part that is the association between BD-VNI or VLAN-VNI, so only thing left id to create the associated interface (SVI in VLAN mode or BDI in VSI mode) with the addressing and thats all:



Note #1: See, only "new" stuff here is that we need to run PIM sparse mode, can you guess why? ping me!
Note #2: Yes, I'm using a VRF since it's also best practice to run tenant traffic into a separate vrf, this is pretty common in real life deployments (not flood and learn, VRF isolation i mean :) )

Now lets consider some design caveats and the first thing in mind for everyone here is HA so...

Redundancy? VXLAN L3 GW + HSRP? What about VPC ?

Yes, Yes and also Yes. Of course you can run HSRP on top of BDI/SVI but the not so easy part here is VPC. As you know VPC provides MAC state sync between peering devices and if you've opted to HSRP redundant VTEPs share anycast VTEP IP address (underlay). This way VPC provides L2 + L3 redundancy in HW, to be able to do this some changes needed to be done to loopack interface used for VTEP sourcing and to VPC Domain config, as a starter point first thing to recall is that a secondary ip address should be shared between VPC peers in order to forward VXLAN packets that can be handled for any of the peering devices, does that remember you something?... Yes, we also need peer-gateway under VPC Domain. Based on what we just describe peer-gateway is mandatory and also requires a SVI configured with PIM across peer link, a list of requirements can be found here:

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_chapter_010.html#concept_C769B7878CE2458E98657905843DEEFA

But what you can never forget is:

• Unique primary IP for underlay loopback
• Same secondary
• PIM
• Consistent VNI to multicast group mapping
• peer-gateway and a "special SVI" (PIM enabled) // This is needed in case your leaf lost connectvity to Spines and needs to forward packet to peering device.

Those are key for me but in the doc you will find a lot of more information, also something to mention is that on Nexus 5K VXLAN VPC configuration requires the use of that special SVI for VXLAN traffic by issuing a special command:

Distributed Gateway

Well, this is a huge post... we will talk about distributed GW, anycast and that stuff in another one since if not this will take me a lifetime and I do want to explain it in detail :)

Stay in touch :)

As we mention, it uses MAC-in-UDP encapsulation to provide a means to extend Layer 2 segments across the data center network. The encapsulation scheme used places the original Layer 2 frame with a VXLAN header and  then placed in a UDP-IP packet. With this MAC-in-UDP encapsulation is easy to think in tunneling VXLAN across L3 networks, a great and easy way to see this is in the following packet format:

As seeing in the picture, VXLAN introduces an 8-byte VXLAN header that consists of a 24-bit VNID and a few reserved bits. The VXLAN header together with the original Ethernet frame goes in the UDP payload. The 24-bit VNID is used to identify L2 segments and to maintain L2 isolation between the segments. 

Also by seeing picture is now easier to get the idea on how tunneling can work with this frame, tunnels are formed between devices which want to exchange VXLAN data and for that only thing needed is destination IP (in Outer IP Header) also once reached we have to do the hard work that every tunnel does encap/decap, and for that we have to introduce other player that is the VTEP or VXLAN Tunnel Endpoint.

VXLAN Tunnel Endpoint

VTEPs are essential players in VXLAN work, as we mentioned earlier their role is similar to any tunnel endpoint (encap/decap) but we will explain in detail how this occur. I would use this simple scenario to do some explanations:

VTEP has presence in a Local LAN segment and has a defined mapping from that segment to a VNID. The encapsulation process consist on taking the L2 Frame sended by any of the End Systems on the local segment, add the VXLAN header with the corresponding VNID, add UDP header, and add Outer IP header (with destination IP for the remote VTEP where we want to send out our packet, if you are now thinking how this entries get populated and how arp is handled you have to wait for Part 2 :) ). Once packet arrives remote VTEP, decap process start by stripping off VXLAN Header and identifying, based on VNID, the local segment in which we have to deliver out our packet.

Being said this the following is pretty self-explanatory (if you follow me, if not here you have a nice picture):

In this figure, Host A is sending out a packet to host B, his associated VTEP, VTEP-1, has an entry for Destination MAC-B in his table and his referring to IP of remote VTEP VTEP-2, he also has the info of the VNID assignment for Host-A, VNID 10. Based on that VTEP-1 has all the info that he needs to encap the packet and deliver it out to VTEP-2. Once packet reach VTEP-2, gets decapsulated, based on VNID is associated to LAN segment where Host-B lives and packet is sent out to the destination (also on VTEP-B and switches if any there know how to reach MAC-B :) )

I think we are good for today, next post in this series will cover how BUM traffic is processed, control plane options and config scenarios. If you want to see any in particular, just let me know.

This is the real hard topic to cover, not in theory or technology but really in config section. For this I really tried to use Cisco Documentation but wasn't enough, so further reading follow me to this GREAT presentation: BRKDCT-2404 VXLAN Deployment Models - A Practical Perspective

Last but definitely not least is ACI, as you can imagine (I figured it out at least), Cisco will push you to use this new tech that they acquire a few years ago. Actually there is no good content related ACI from any partner or learning vendor, I took a quick look at Jason Lunde and other videos, but they where so so introductory that you really need further info. In my case I achieve that by setting up a POD several times and facing all kind of errors, I focused on what I though that it was the key objectives of having ACI deployed, and for me that was:

- Setting up VMM integration

- Deploying Contracts between EPG and Tenants

- Customizing several Fabric configurations (It's well known that in a CCIE Exam they can ask you to change whenever option is available at the interface, so be prepared)

- Deploy a L2 Out connection

- Deploy an L3 Out connection

- Share L3/L2 between tenants

For all this I really recommend to use and see Cisco Community, that was really helpful, Tomas De Leon (a TAC guy who wrote there) uploads very useful documents about how to configure APIC. And again, get a lot of hands on experience in this topic, ACI have a lot of errors and you have to get familiar with them, if you have no previous experience start with a video series to get familiar with concepts and after that book your labs, get an image or us dCloud but be sure to get hands on :)