Living in the underlay

So, what it is?

and... whats new?

ELAM Example

• If DC1 site fails but DCI and DC2 site alive, traffic will enter from DC2 side and traverse the DCI.
• If DCI fails, traffic will continue being served from DC1 for stretched VLANs subnets, this implies move by other method those servers to the surviving side or at least shut them down.
• If DC2 site fails but DCI and DC1 site alive, traffic will enter DC1 side and traverse DCI to reach DC2 side servers.
• Traffic should leave and enter from DC1 whenever possible and DC2 site should not be used unless strictly necessary (this was imposed by customer)

Now if we have a failure on DC1 side, conditional trigger will take place and start advertising from DC2.

Is this all what we need? Definitely No... There are still lot of things to resolve and we don't have an optimal design (we can discuss here, if we are meeting business requirements is there anything else to do?), but apart from that notice that stretching a VLAN is not a good choice, guess why? you're extending your fault domain and that doesn't simplify things it also make more complex the isolation and detection. so let's start wondering why we made such poor decisions and why we can't start talking about application level aware resiliency, making our life better by allowing us to use different subnets/networks at each site being able to handle traffic in/out more flexible by leveraging existing methods (long talk about BGP attributes and policy control enters here).


Some references:

 Cisco. (Agosto de 2010). Cisco IP Routing. http://www.cisco.com/en/US/tech/tk365/technologies_configuration_example09186a0080094309.shtml

BiDir PIM

Need of redundancy? Let's do it

• Create "significant" traffic: This means not to only stress performance but to use traffic patterns that are representative for you (i.e. workloads similar to the environment in which you are going to put the device under test)

• Don't forget to measure: An important and also tedious part of the testing is to take notes and write down the results, so always plan your performance scenarios with the idea behind of getting the data exported so you can easily write it down or graph results.

• Test boundaries: If a device is called to reach X performance, test it. Lets say here X is 100K IOPS @8k bs 70/30 (r/w), so you have to get a way to reach that performance in your infrastructure (generate that workloads). Also there are several considerations to take here, i.e. using one thread with that workload is not the same than running multiple which represent a much more real approach, we will deep into this in an specific post about AFA testing procedure.

• Tune the environment: This can also be called environment set-up, be sure to have your underlay infrastructure ready with best practices and no issues to be sure that the test you're running is not getting affected by any other factor than the test procedure itself.

• Automate as much as you can: Doing testing can be tough, imagine that you have to re-test changing few parameters, applying new versions, etc... impossible.. so get an automated approach to set up your test and even shoot them and plot the results in a fancy way.

• Understand what you're doing: Testing is not about running a workload and seeing if performance is good or not, or at least it shouldn't be. The whole purpose behind a testing procedure is to understand how the under-testing device react at stress conditions an under normal plus similar to real ones. Also to notice how does internally works and how behaves under changes... this guided me to the next bullet.

• Resiliency: So you have tested and all seems perfect, performance is outstanding and testing is going well... but have you tested how does this behaves under unexpected and planned changes? Resiliency is key to production environments since it not only gives you an overview of how high availability is performed (which can be important and most for production environments) and also on how doest the systems react to this changes (you can be easily surprised by well-known vendors running in panic mode after switchovers).

• Plan the tests accordingly: If you're running a PoC or a performance test you will do a lot of work for preparation and setup of the environment, this can involve doing changes in physical network to test HA, clusters, or other functionalities, you will lost lot of time by changing lot of times so is really important to order the test plan accordingly meaning to do the minimal changes necessary and in every change do the maximum amount of task prior to next change, this will save you lot of time.

So, what about technical requirements?

• Amount of IOPS: This is something that you can see a lot, and is completely wrong, since what is 1K IOPS? Which block size? which read/write ratio?
• Amount of IOPS based on a given IO distribution which includes block size and read/write ratio of each of those. Getting this numbers can be significant hard, most of current arrays have the information of all the workloads they had run, this is amazing for getting the IO distribution per block size plus rd/wr ratios but assumes that you are running similar workloads all day long (i.e. consistent distribution of a given set like 30K IOPS @4k 65/35, 15K IOPS @8k 60/40.. and so on, but what about the nightly jobs? when your performance gets affected with backup jobs?)
• Expected  and maximum latency: Based on application + OS/Guest OS needs 
• Expected compression ratio (for your data set!)
• HA and expected performance in contingency
• Network based (NFS), IP based (iSCSI) or FC? 
• Disk replacement policy and MTBF/MTTF

Ok, but where is the vendor Hype?

How can I test performance an avoid being seduced by pink elephants?

• Part 1: Let's overlay -- Basic info about VXLAN, addressing and headers.
• Part 2: It's all about knowledge -- Packet forwarding overview, VTEP control plane learning options
• Part 3: Hands On #1 -- Configuration on Cisco Nexus Devices, Flood and Learn.
• Part 4: Hands On #2 -- Configuration on Cisco Nexus Devices,  EVPN.

Get yourself use to read and read

Be in a group and collaborate as much as you can

Color scheme for highlighting

Be a chameleon

Suboptimal routing still works right?

Hope this help all the next aspirants, those new and those not so new. Also for those which have passed lab, congrats! and inputs are always welcome.

• Part 1: Let's overlay -- Basic info about VXLAN, addressing and headers.
• Part 2: It's all about knowledge -- Packet forwarding overview, VTEP control plane learning options
• Part 3: Hands On #1 -- Configuration on Cisco Nexus Devices, Flood and Learn.
• Part 4: Hands On #2 -- Configuration on Cisco Nexus Devices,  EVPN.
• Part 5: NSX Overview

VxLAN Flood and Learn

1. End System A (ES-A) sends out an ARP request for IP-B on its Layer 2 VXLAN network (note the Dst MAC Address).
2. VTEP-1 receives the ARP request. Since he doesn't have a mapping for IP-B yet, it encapsulates the ARP request in an IP multicast packet and forwards it to the VXLAN multicast group for that specific segment (VNI). The encapsulated multicast packet has the IP address of VTEP-1 as the source IP address and the VXLAN multicast group address as the destination IP address.
3. The IP multicast packet is distributed to all members in the tree, VTEP-2 and VTEP-3 receive the encapsulated multicast packet because they’ve joined that specific VXLAN multicast group, after that they decapsulate the packet and forward it locally to the local VXLAN network. In this process, if no prior communication was made between VTEP-1 to them, they insert into his local tablet the mapping between Mac Address of ES-A with IP of VTEP-1.
4. After the local transport of ARP, End System B (ES-B) gets the request forwarded by VTEP-2 and responds with its own MAC address (MAC‑B), and learns the IP-A-to-MAC-A mapping.
5. VTEP-2 receives the ARP reply of ES-B that has MAC-A as the destination MAC address, as per step 3 he knows about MAC-A-to- VTEP-1 mapping and therefore it can use the unicast tunnel to forward the ARP reply back to VTEP-1. The ARP reply is encapsulated in the UDP payload of a packet sourced from VTEP-2 and destined to VTEP-1.
6. VTEP-1 receives the encapsulated ARP reply from VTEP-2. It decapsulates and forwards the ARP reply back to ES-A, also it learns the IP address of VTEP-2 from the outer IP address header and inspects the original packet to learn MAC-B-to-VTEP-2 IP mapping.
7. Subsequent IP packets between ES-A and B are unicast forwarded, based on the mapping information on VTEP-1 and VTEP-2, using the VXLAN tunnel between them.
8. VTEP-1 can optionally perform proxy ARPs for subsequent ARP requests for IP-B to reduce the flooding over the transport network.

Head-end Replication

VxLAN MAC Distribution

VxLAN BGP EVPN Control plane

• Standardized solution: BGP plus VxLAN
• Real Control Plane learning

• VTEP/network virtualization edge (NVE) is the equivalent to PE node
• VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
• Broadcast, unknown unicast and multicast (BUM) data traffic is sent using a shared multicast tree.
• In order to reduce the need of full mesh between VTEPs we can rely on BGP route reflector (RR)
• Enhanced security by using well known Route filtering and constrained route distribution (control plane traffic for a given overlay is only distributed to the VTEPs that are in that overlay instance).
• Host (MAC) mobility mechanism to ensure that all the VTEPs in the overlay instance know the specific VTEP associated with the MAC

Packet forwarding in L2 VxLAN Segment

• Host-A sends traffic to his local VTEP V1 (post ARP resolution), DST MAC B.
• V1 will lookup in his table for an entry for MAC B.
• V1 has an entry for MAC B thru VTEP V2, it encapsulate the packets and unicast send to V2.
• V2 gets the packet, decapsulate and locally deliver to Host-B

Packet forwarding between different L2 VxLAN VNI

• Host-A sends traffic to DG (post ARP) which is configured on the locally attached VTEP V1.
• V1 make a FIB lookup based on DST IP
• V1 routes the packet to VTEP V2, but VXLAN packet is using core VNI 50000.
• V2 gets the packet, it decapsulates, made a FIB lookup determining that DST VNI is 30001, rewrites the packet and deliver locally.

• VXLAN uses a 24-bit segment ID known as the VXLAN network identifier (VNID), which enables up to 16 million VXLAN segments, this allow us for higher scalability and multi-tenancy
• VLAN based designs uses STP in the back to choose best path, VXLAN uses MAC-in-UDP encap and in consecuence he can take advantage of the underlying network (Layer 3) and can take complete advantage of Layer 3 routing, equal-cost multipath (ECMP) routing, and link aggregation protocols to use all available paths, this provides us better use of resources.

VXLAN Packet format

1. Read the blueprint at cisco cert page and topics included in written (https://learningnetwork.cisco.com/community/certifications/ccie_data_center/written_exam/exam-topics)
2. Start by identifying the topics that you (think) master and those you definitely don't.
3. Mix reading of topics you don't know anything of with those that you do, this is key to avoid being overwhelmed by new stuff (your brain will thank you)
4. Always take notes! For CCIE written exams and similar certs you have to note down those concepts and configuration maximums/limits that you will never remember in real life! 
5. To use as a guide, be sure that you have a section on your notes for each protocol that is in the blueprint, i.e. for FabricPath you should have at least this info:

"Running per Supervisor Engine, on a per-VDC basis:●   FabricPath IS-IS - SPF routing protocol process that forms the core of the FabricPath control plane
●   DRAP - Dynamic Resource Allocation Protocol, an extension to FabricPath IS-IS that ensures network‑wide unique and consistent Switch IDs and FTAG values
●   IGMP - Provides IGMP snooping support on FabricPath edge switches for building multicast forwarding database
●   U2RIB - Unicast Layer 2 RIB, containing the “best” unicast Layer 2 routing information
●   M2RIB - Multicast Layer 2 RIB, containing the “best” multicast Layer 2 routing information
●   L2FM - Layer 2 forwarding manager, managing the MAC address table
●   MFDM - Multicast forwarding distribution manager, providing shim between platform-independent control-plane processes and platform-specific processes on I/O modules

Global components that run on each of the I/O modules, processing forwarding information from each VDC and programming it into the I/O module hardware:
●   U2FIB - Unicast Layer 2 FIB, managing the hardware version of the unicast Layer 2 RIB
●   M2FIB - Multicast Layer 2 FIB, managing the hardware version of the multicast Layer 2 RIB
●   MTM - MAC table manager, managing the hardware version of the MAC address table"

• NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures, 2nd Edition
• I/O Consolidation in the Data Center
• Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems, Applications, Management, and File Systems
• Cisco Unified Computing System (UCS) (Data Center): A Complete Reference Guide to the Cisco Data Center Virtualization Server Architecture
• Policy Driven Data Center with ACI, The: Architecture, Concepts, and Methodology
• Cisco Live docs, don't you use that? You're missing a GREAT resource
• BRKDCT-2404 VXLAN Deployment Models - A Practical Perspective
• BRKDCT-2370 - Intermediate - End-to-End Application-Centric Infrastructure Automation with UCS Director
• BRKDCT-2049 - Overlay Transport Virtualization
• BRKDCT-3237 - Advanced - Versatile architecture using Nexus 7000 with a mix of F and M modules to deliver FEX, FabricPath, Multihop FCoE, MPLS and LISP all at the same time 
• BRKDCT-3145 - Advanced - Troubleshooting Cisco Nexus 5000 / 2000 Series Switches 
• BRKDCT-3378 - Advanced - Building simplified, automated and scalable DataCenter network with Overlays (VXLAN/FabricPath)
• Also lot of blogs... I will write down my RSS feeds soon :) there is really smart people near us :)