Living in the underlay

I had the opportunity to participate in the latest MPLS SDN NFV World @Paris 2018, and need to say I've saw a lot of SD-WAN gear around and had the opportunity to talk with some technical experts regarding their solutions.

The main goal of this post is to point some considerations I'd found during this week, if you feel that you company is the exception to this list please feel free to ping me and let's have a talk :)

Being that said this are the missing points I found and want to summarize in order to help those companies in their roadmap development:

- Big Data, really? - Lot of vendors claims that their SD-WAN solution has a built-in analytics and big data backend in order to get lot of information that will be useful for advanced reporting and metrics. That is not a lie for sure but the issue im still finding here is that no single vendor is using that data to do a feedback cycle to the system. What does this means? Imagine that you have a lot of information of the BW usage for customer links and (after a while) you can do basically two simple actions:

1. Detect real time high-usage/packet drop/service degradation/SLA not met/etc 
2. Predict or even prepare the network for future loads

I'm having lot of discussion with my students and colleagues regarding this topic and want to start by clarifying this question that someone made few time ago:

"Why do you teach Cobra SDK for CCIE DC curricula instead of pure raw REST API?"

The answer was quite simple "Is in the official curricula/blueprint and you need to master it"  but it has some drawbacks and I feel that some other things needs to be considered here:

• Embrace API: The freely way to operate is to use pure REST API and automate as you wish the service (at the end we want to cover some use case that serves some purpose and that can be considered a service for an specific end user - engineer, customer, etc- ). If you understand and comprehends how the API is structured and works it would be quite easy to automate that specific gear moreover you can even start thinking like that box/system ;) [1]

• SDK is a huge toolbox, use it wisely: The SDK just provides you a toolbox to quickly start coding without messing around with a **huge** API set, but even if it seems the easy way there are some caveats to consider: The SDK packages common operations into functional boxes (methods) for you to consume, however that doesn't mean that all your pretty weird use cases will be covered/reflected into the SDK [Even if they say that the SDK reflects the API, I've not met a single SDK from a network vendor that covers their full set of API operations into the SDK and lets not even start talking about documentation..].

• Don't reinvent the wheel: If you're planning to move towards a pure REST API model i'm happy for you (really, not sarcastic) but consider: are you going to encapsulate the specific device on your specific package/model? If the answer is yes for each specific device my first response would be "why not to use the SDK on the first time?" since you are deploying the same thing with a less code power than the vendor. if the answer is yes but not to a specific device but to a specific role in network we are start talking about good design choices ;) don't tie your code to a specific gear, make it independent and more on this...

• Don't repeat the past: If you're planning to use pure REST API to talk to a network device and you're thinking the legacy way you will end up in multiple API calls (being multiple equals to the lines of CLI commands that you need to enter in the device configuring by hand). So basically you will be doing old school network in a fancy way, why not starting asking devices about a desired state? (intent)

• Code a service not a function: Creating a [script/program/api] to do a specific task such as create vlan, trunk it, enable a protocol or even to do a correlate serie of tasks is not the same than creating a service, the aim of your code should be the service automation since network automation is well covered by many sources but service automation is specific to your business/use case (please have in mind that even if there are a lot of powerpoint $#$%$# around the probability that your use case is not a standard covered one is near to 99.99999%)

One of my colleagues just asked me about the recommended reading list for the Juniper DC track (in particular what I've used to clear JNCIP-DC few weeks ago), here is a complete list of free resources that you can access to prepare yourself for the exam, I will also recommend (if you don't have any real/lab experience with QFX/EX for vxlan setup and mostly with VCF) to do some labs with vQFX (you can try out EVE-NG, which is *highly recommended*)

Here it is:

Juniper Networks EVPN Implementation for Next-Generation Data Center Architectures - https://www.juniper.net/assets/us/en/local/pdf/whitepapers/2000606-en.pdf

Virtual Chassis Fabric Feature Guide - http://www.juniper.net/documentation/en_US/junos/information-products/pathway-pages/qfx-series/virtual-chassis-fabric.pdf

Comparing Layer 3 Gateway & Virtual Machine Traffic Optimization (VMTO) For EVPN/VXLAN And EVPN/MPLS - https://www.juniper.net/documentation/en_US/release-independent/solutions/information-products/pathway-pages/solutions/l3gw-vmto-evpn-vxlan-mpls.pdf

Clos IP Fabrics with QFX5100 Switches - https://www.juniper.net/assets/cn/zh/local/pdf/whitepapers/2000565-en.pdf

Virtual Chassis Fabric Best Practices Guide - http://www.juniper.net/documentation/en_US/release-independent/vcf/information-products/pathway-pages/vcf-best-practices-guide.pdf

EVPN Control Plane and VXLAN Data Plane Feature Guide for QFX Series Switches - https://www.juniper.net/documentation/en_US/junos/information-products/pathway-pages/junos-sdn/evpn-vxlan.pdf

Understanding Zero Touch Provisioning - https://www.juniper.net/documentation/en_US/junos/topics/concept/software-image-and-configuration-automatic-provisioning-understanding.html

Configuring Zero Touch Provisioning - https://www.juniper.net/documentation/en_US/junos12.3/topics/task/configuration/software-image-and-configuration-automatic-provisioning-confguring.html

Configuring Zero Touch Provisioning in Branch Networks - https://www.juniper.net/documentation/en_US/release-independent/nce/information-products/pathway-pages/nce/nce-151-zero-touch-provisioning.pdf


Also another great book which I just end reading is "Building Data Centers with VXLAN BGP EVPN", this book is Cisco NXOS oriented but provides an amazing background on how VXLAN BGP EVPN Fabrics works.

HTH,

I was using this quite lot past weeks and think that is a good resource to share to everyone playing around with Cisco ACI. When it comes to tshoot and to understand packet flow inside the Fabric ELAM is a great tool.

So, what it is?

ELAM stands for Embedded Logic Analyzer Module, It is a logic that is present in the ASICs that allows us to capture and view one or more packets, that match a defined rule, from all the packets that are traversing the ASIC. ELAM is not new at all, some of you can remember this from CAT6500, and thats ok, same logic also same from N7K (for the youngest?).

and... whats new?

Essentialy the concept is still the same, an we just need to focus on understand how is the architecture inside the ASICs on Leafs and Spines to fully apply this concept.

Cisco ASIC data path is divided into ingress and egress pipelines where two ELAMs are present (see figure) at the beginning of the lookup block.

ELAM Example

• If DC1 site fails but DCI and DC2 site alive, traffic will enter from DC2 side and traverse the DCI.
• If DCI fails, traffic will continue being served from DC1 for stretched VLANs subnets, this implies move by other method those servers to the surviving side or at least shut them down.
• If DC2 site fails but DCI and DC1 site alive, traffic will enter DC1 side and traverse DCI to reach DC2 side servers.
• Traffic should leave and enter from DC1 whenever possible and DC2 site should not be used unless strictly necessary (this was imposed by customer)

For those attending to my CCIE DC v2 bootcamp next week, here is the updated outline, I will be posting updated diagram in few (remember this course is not based in any rack rental so interface numbering is up to that :) )

Introduction

Exam Considerations / Oveview / Strategy


Section 1 – Cisco Data Center Layer 2/Layer 3 Technologies

1.1 – Configure VDC Resources
1.2 – Configure NXOS multicast
1.3- Understanding VxLAN
1.4 – Configure vPC & Deployment options
1.5 – Configure FEX & Deployment options
1.6- Configure VxLAN L2/L3 GW (EVPN | F&L)
1.7 – Configure NXOS Security
1.8 – Configure& Troubleshoot Spanning Tree Protocol
1.9 – Configure & Troubleshoot OTV


Section 2 – Cisco Data Center Network Services

2.1- ACI Service Graph
2.2 – RISE
2.3 – Unmanaged devices in ACI
2.4 –Configure Shared L3 Services


Section 3 – Data Center Storage Networking and Compute

3.1 – Configure FCoE
3.2 – Cisco UCS Connectivity
3.3 – UCS QoS
3.4 – Service Profiles
3.5 – Configure advanced policies
3.6 – Configure Cisco UCS Authentication
3.7 – Configure Call Home Monitoring
3.8 – Troubleshoot SAN Boot
3.9 – UCS Central Basics
3.10 – UCS Central Advanced configuration & tshoot


Section 4 – Data Center Automation and Orchestration

4.1 – Introduction to scripting in Python / cobra SDK
4.2 – Python Programming with ACI Advanced
4.3 – UCS Director Basics
4.4 – UCSD Advanced Workflows Design


Section 5 – ACI

5.1 – Understanding ACI Fabric Policies
5.2 –Understanding ACI Access policies
5.3 – ACI external L3 connectivity in shared resources
5.4 – ACI L2 bridge / L2out
5.5 – ACI VMM integration

Past week two weeks a colleague and also a student asked me about Phantom RP and how it works, all was related with a discussion we have around VXLAN Part 2 post and about supported Multicast configurations for VXLAN in NX-OS.

First of all, and in order to avoid further confusions around it, I would resume current supported methods for VXLAN underlay on Cisco NXOS/ASR devices:

Being clarified that, we can continue with the original purpose of this post.
So, based in our previous post we have configured our Nexus 5K / 7K underlay to run multicast in to support Flood and Learn configuration, by that time we choose Bidir PIM since is the only supported method in N5K. So let's get some background about bidir and how can we make it redundant (can we?)

BiDir PIM

PIM Bi Directional mode enable multicast group to route traffic over a single shared tree rooted at the RP, instead of using different unidirectional or sources tree. Since RP is the root  (his IP address :) ) is good to not to place it on a router but on an unused IP on the network reachable from PIM domain (this will be seen later in PhantomRP configuration).
Explicit join messages are used to establish group membership, Traffic from sources is unconditionally sent up the shared tree toward the RP and passed down the tree toward the receivers on each branch of the tree (note: traffic is not sent unidirectional to RP)

Bidir-PIM shares mechanisms of PIM-SM like unconditional forwarding of the source traffic toward the RP but without the registering process for sources (https://tools.ietf.org/html/rfc7761#section-4.2). Based on that forwarding can take place based on (*,G) entries, removing the need of any source specific state and, therefore, expanding scaling capabilities. This image extracted from Cisco white paper are good to see the differences in upstream process towards the RP in SM vs BiDir:


"PIM-SM cannot forward traffic in the upstream direction of a tree, because it only accepts traffic from one Reverse Path Forwarding (RPF) interface. This interface (for the shared tree) points toward the RP, therefore allowing only downstream traffic flow. In this case, upstream traffic is first encapsulated into unicast register messages, which are passed from the designated router (DR) of the source toward the RP. In a second step, the RP joins an SPT that is rooted at the source. Therefore, in PIM-SM, traffic from sources traveling toward the RP does not flow upstream in the shared tree, but downstream along the SPT of the source until it reaches the RP. From the RP, traffic flows along the shared tree toward all receivers."

Need of redundancy? Let's do it

We mention that our shared tree is rooted at RP address, so in order to give him redudancy we need a way to duplicate this or use a virtual IP. For bidir pim no traffic is targeted at RP (no control plane functions) so our solution is easier, instead of actually assign same IP in a sort of anycast we can just advertise it thru our IGP, the only issue foreseen is that the actual shared tree should be only one at a given time (we dont want that our RPF interface changes everytime) so in oirder to avoid that we can leverage the path decision to a more specific match in the RIB (by advertising same subnet with largest mask by some of the redundant points).
Well, that was so much talk I think that a code/config snippet worths more than a millon words:


Primary


Secondary (hmm.. if you don't see any difference here is a hint: look at the mask)




Now it's done, you can run your set of favourite verification commands to see if this is working:



Also you can shutdown the active interface (lo1) and see how does this change and our redundancy is working.

For CCIE / CCDE students:
- What is the convergence time of RP in case  of a failure on primary?
- Can we give sub-second convergence?
- In flood and learn configuration for VxLAN what would you recommend ASM or bidir PIM?
- In case of choosing ASM how is your redundancy going to be solved?
- Why are we using "ip ospf network point to point" ?

More on Multicast ASM/SSM/Bidir comparisson: http://lostintransit.se/2015/08/09/many-to-many-multicast-pim-bidir/