Living in the underlay

Tuesday, February 28, 2017

CCDE, be a chameleon

9:33 AM

After passing DE want to take some time and write this words to all futures aspirants and also for those who failed and also pass to see their point of views regarding CCDE. I was thinking about strategy in lab and mind strategy as well and I realize that being a chameleon is one of the important aspects that you will need to address in order to be successful in this path.

Disclaimer: This post is intended to be a wrap up of my experience to succeed in lab, this study methodology and all suggested here is based on my background and available times on week for studying, feel free to use it and adjust it to you own pace.

Being said that...

Get yourself use to read and read

Asumming that you have passed written and you feel like all the theory needed is covered I can assure you, that's not true. Even if you have passed the written you still need to read and re read technology and also be able to understand pros and cons of each design. A general rule that I can say is that to any desig you will need to be able to find the pros and the cons, if you can't you're just being shortsigthed and you're missing to cover all points in a given design. For this aspect is also key to get a group study, so let's talk about this one too.

Be in a group and collaborate as much as you can

Some guys claim that studying alone is the way that best suits them, that can be great if you're a giving a IE level exam or any written, but for DE is completely the opposite. I do really like to do this comparison and say that if you study alone you're just a vlan with a polarized HSRP gateway, despite of the nerd comparison what I do really mean is that you can't argue anyone! and that is not good at all, since I don't expect guys that argue for everything but I do expect people that say their point of view and discuss also why they have thought that caveat or pro in a proposed design. And remember discussions are a good thing, just like other good stuff don't abuse to them and use it carefully.

The methodology that we use in our group was to check one scenario every week, we set up call on weekend and on week we study by ourselves. You can use the scenarios in Orhan and Martin Duggan book, but since we run out of scenarios what we made is split ourselves in group of two and make modifications to create a specific scenario that cover some specific design concern, I do recommend you that in this "split" you choose to create scenarios based on your areas of expertise (in my case it was SP and DC :) )

My study methodology prior to lab was aprox 6hs per day (Monday-Friday), and 4hs sessions on Saturday. This by two months... and assuming that you have cleared written (i cleared mine long ago) and you have good design expertise (if not i will recommend you another 2 months of study), in my case I have near 13 years of expertise working on huge companies (like design scenarios hehe) and last 6 years on a Service Provider (who divest itself into two, nice scenario for CsC we made in real life!)

For lab strategy I can summarize my key points, some of them you will find it along a lot of posts, but this was really useful for me:

Color scheme for highlighting

I do really quick reading and that has a caveat, you can omit stuff, so be sure that you read all of the sentences and also take a color scheme to highlight information. I choose a real simple one: Green is good design option taken, Red is bad design or caveat on design, Yellow is constraint or requirement. I want to use pink for IGP info but i realize that info is so mixed that in second scenario I back to the roots and only use those three.

Be a chameleon

This is the mind status that you have to reach in the exam, you will have to be a chameleon to read all requirements and constraints and be ready to transform yourself in the designer of ABC Company to take the best choice, But inmmediately , in next scenario you will transform yourself into a Service provider architect being part of team who is evaluating X or Y technology. All this happens really fast and you have plenty of info around you to support you in this transformation, what you need is the ability to quickly focus in the job role that they have assigned to you, gather the info and take the best shot.

Suboptimal routing still works right?

I do really love to do comparisons of real life and technology and in this case I can think in suboptimal routing as a wrong (or not so good) branch in exam. If you have readed all Cisco Live material of DE you may be aware that in scenarios there is always several branches based on your selections, so even if you're not in the best branch you can get points (STOP! if you're not reading Cisco Live info for DE take a moment to review that, is key to understand exam flow). So in the time frame of an scenario the branch will become your life, so being able to manage bad decissions is important, because you will realize that you don't choose the best path (trust me, it happens) but they will provide you options to understand why you make that choice and you will have to choose the not so good and not so bad option to be able to continue, so don't get worried suboptimal routing still works :) and we know next time you will not face same mistake :)

Hope this help all the next aspirants, those new and those not so new. Also for those which have passed lab, congrats! and inputs are always welcome.

Friday, February 24, 2017

CCDE... happy ending

10:25 AM

Well guys, you may note that I was out of blogging for a while, and there was a reason behind that... CCDE was so time consuming, I spend at least 4 to 6 hours last two months with a my awesome study group debating scenarios, design choices and pros/cons of each of them... it was real hard, it requires ****a lot**** of reading but definitely worth it!! Past Feb 22nd I pass the lab, it was at my first attempt and I feel very lucky about that but it was really about training, reading, understanding and debate with my colleagues what make me succeed, I will write down a post with strategy and training used as well, but at this moment just want to take me some time to write down this line and congratulate all my friends and colleagues who have taken the CCDE lab, passed or not, was a real good experience and an amazing learning!

Keep you updated guys :)
CCDE 2017::27

Friday, January 27, 2017

VXLAN Deep Dive Part II: It's all about knowledge

3:37 PM

Before starting with second part of this post and in order to calm down anxiety will briefly describe what this series is going to cover:

Part 1: Let's overlay -- Basic info about VXLAN, addressing and headers.
Part 2: It's all about knowledge -- Packet forwarding overview, VTEP control plane learning options
Part 3: Hands On #1 -- Configuration on Cisco Nexus Devices, Flood and Learn.
Part 4: Hands On #2 -- Configuration on Cisco Nexus Devices, EVPN.
Part 5: NSX Overview

So if you're interested in any other topic that you think is not going to be covered kindly ping me and will add.

After all that prelude I think we can start. As we see in Part I, we cover the header added into the original frame in order to be forwarded into an L3 network and also we end the post by giving an overview of packet forwarding. In order to reference later here is a pic of a VXLAN packet:

Figure 1: VXLAN Packet header

In later post we just reach a point in where host (hypervisor or device with VTEP, I will use any of these indistinctly) get a packet (VXLAN) which is not local and need to be delivered. Let's think like any L2 forwarding plane, we need to know where to route/send out this packet, this process is made by a lookup made by the host based on DST MAC Address of Original L2 frame (see picture above) and based on that we should get a ~~destination port~~ (hehe no L2 switching ) destination VTEP Address. This post would cover different methods of learning and populating this internal table, and as usual for forwarding it's all about knowledge.

VxLAN Flood and Learn

This scenario was the first introduced, it relies in head end replication, meaning that end host in case of not having any entry for the destination MAC address will send out an ARP to other devices / VTEPs in the VXLAN network. This is done by sending the request to the VXLAN multicast group for this Bridge domain, remote VTEPs will get the packet and answer accordingly direct to the originating VTEP (Here we can be aware of two requirements for running this: multicast core, IGP or unicast reachability between VTEP Addresses)

Figure 2: VXLAN Peer Discoveries and Tenant Address Learning

I will base the explanation using this amazing pic that I just stole from cisco web page :)

End System A (ES-A) sends out an ARP request for IP-B on its Layer 2 VXLAN network (note the Dst MAC Address).
VTEP-1 receives the ARP request. Since he doesn't have a mapping for IP-B yet, it encapsulates the ARP request in an IP multicast packet and forwards it to the VXLAN multicast group for that specific segment (VNI). The encapsulated multicast packet has the IP address of VTEP-1 as the source IP address and the VXLAN multicast group address as the destination IP address.
The IP multicast packet is distributed to all members in the tree, VTEP-2 and VTEP-3 receive the encapsulated multicast packet because they’ve joined that specific VXLAN multicast group, after that they decapsulate the packet and forward it locally to the local VXLAN network. In this process, if no prior communication was made between VTEP-1 to them, they insert into his local tablet the mapping between Mac Address of ES-A with IP of VTEP-1.
After the local transport of ARP, End System B (ES-B) gets the request forwarded by VTEP-2 and responds with its own MAC address (MAC‑B), and learns the IP-A-to-MAC-A mapping.
VTEP-2 receives the ARP reply of ES-B that has MAC-A as the destination MAC address, as per step 3 he knows about MAC-A-to- VTEP-1 mapping and therefore it can use the unicast tunnel to forward the ARP reply back to VTEP-1. The ARP reply is encapsulated in the UDP payload of a packet sourced from VTEP-2 and destined to VTEP-1.
VTEP-1 receives the encapsulated ARP reply from VTEP-2. It decapsulates and forwards the ARP reply back to ES-A, also it learns the IP address of VTEP-2 from the outer IP address header and inspects the original packet to learn MAC-B-to-VTEP-2 IP mapping.
Subsequent IP packets between ES-A and B are unicast forwarded, based on the mapping information on VTEP-1 and VTEP-2, using the VXLAN tunnel between them.
VTEP-1 can optionally perform proxy ARPs for subsequent ARP requests for IP-B to reduce the flooding over the transport network.

Head-end Replication

When you are working with VXLAN and reading literature also is common to hear or read the concept of head-end replication, what this essentially means is that the local VTEP has the overhead of replicate the broadcast traffic out to the other VTEPs, in the original release of VXLAN which uses multicast as underlying layer to reach VTEPs this only means encapsulate packet and sent out to multicast group, but also there is the possibility of have unicast peering (full-mesh) with all the VTEPs and in this scenario the head-end replication has a notorious impact.

Figure 3: Head-end replication example in unicast VTEP reachability

VxLAN MAC Distribution

Another well know method is VXLAN MAC Distribution, head-end replication is still used to deliver broadcast and multicast frames to remote VTEPs, but.. what about unknown unicast? You shouldn't have any (wish, read further). In this scenario MAC learning is not based on data plane activity and instead of that we have a central control unity (Nexus 1000V VSM, NSX controller, etc) which is used to keep track of all MAC addresses in the domain and send this information to the VTEPs on the system. Why do I say that this is a wish? Basically things are there to be broken, just like anY mapping table (CAM i.e.) entries have an aging associated to it, so if in first scenario VTEP-2 announces MAC-B entry through it and VTEP-1 gets populated with that all traffic will flow accordingly and VTEP-1, if doesn't have an entry for MAC-B, will query controller to get this info. Here two branches appears, a) controller has an entry and reply back to VTEP-1, entry gets installed and unicast traffic flow; b) controller doesn't have an entry for MAC-B and reply with an invalid entry so VTEP-1 must use head-end replication to reach learn where to send his packet (*this may vary depending on VTEPs OS/SW implementation).

Also there is another case in which VTEP-1 has a valid entry but it lost connectivity to controller and that entry gets old (and removed from table), in this case controller can't be queried and head-end replication will be used again.

VxLAN BGP EVPN Control plane

Quick disclaimer: Before starting with this I will say that you will find a lot of literature for this approach, also a lot of information regarding configuration to make this possible. This is the desired scenario for any real / production environment, Flood and Learn was showed just to understand what we got in the beginning and how we came up with a real control plane solution (and in a standard fashion way!).

EVPN overlay specifies adaptations to the BGP MPLS-based EVPN solution to enable it to be applied as a network virtualization overlay with VXLAN encapsulation, essentially this bring us great benefits (I will add more later):

Standardized solution: BGP plus VxLAN
Real Control Plane learning

For this approach what we made is (for MPLS EVPN knowers):

VTEP/network virtualization edge (NVE) is the equivalent to PE node
VTEPs use control plane learning/distribution via BGP for remote MAC addresses instead of data plane learning.
Broadcast, unknown unicast and multicast (BUM) data traffic is sent using a shared multicast tree.
In order to reduce the need of full mesh between VTEPs we can rely on BGP route reflector (RR)
Enhanced security by using well known Route filtering and constrained route distribution (control plane traffic for a given overlay is only distributed to the VTEPs that are in that overlay instance).
Host (MAC) mobility mechanism to ensure that all the VTEPs in the overlay instance know the specific VTEP associated with the MAC

MP BGP could be used for L2 VXLAN and also for L3 VXLAN (instead of Mac addresses learning think of IP association to VTEPs, do you remember LISP?). It's not my goal to enumerate all the benefits of running BGP EVPN control plane for VXLAN, apart of greater scalability, well known and proven protocols, etc. instead of that I will focus in the life of a packet in this new scenario and hopefully in next post we can cover all the variations for this (anycast GW, asymmetric. symmetric IRB, etc)

Packet forwarding in L2 VxLAN Segment

In this scenario we are covering L2 VxLAN communication, Host-A and Host-B belong to same VNI: 30000.

Host-A sends traffic to his local VTEP V1 (post ARP resolution), DST MAC B.
V1 will lookup in his table for an entry for MAC B.
V1 has an entry for MAC B thru VTEP V2, it encapsulate the packets and unicast send to V2.
V2 gets the packet, decapsulate and locally deliver to Host-B

End of happy tale, right? What about L3 traffic between VxLAN (see that we didn't cover this in flood and learn, since in that approach traffic should reach a device with the two VxLAN segments involved and logically route)

Packet forwarding between different L2 VxLAN VNI

In this scenario Host-A (VNI 30000) sends packet to Host-F (VNI 30001), core network is using VNI 50000, based on that the process is similar to:

Host-A sends traffic to DG (post ARP) which is configured on the locally attached VTEP V1.
V1 make a FIB lookup based on DST IP
V1 routes the packet to VTEP V2, but VXLAN packet is using core VNI 50000.
V2 gets the packet, it decapsulates, made a FIB lookup determining that DST VNI is 30001, rewrites the packet and deliver locally.

Ok, so now what about the ugly tale? As you can see I made two examples saying "this happens post ARP resolution", but how do we process ARP?

There is an ARP suppression mechanism, essentially, the IP-MACs learnt locally via ARP as well as those learnt over BGP-EVPN are stored in a local ARP suppression cache. ARP request sent from the end host is trapped at the source ToR and a lookup is performed in the ARP suppression cache with the destination IP as the key. If there is a HIT, then the ToR proxies on behalf of the destination with the destination MAC.

In case the lookup results in a MISS, when the destination is unknown or a silent end host, the ToR re-injects the ARP request received from the requesting end host and broadcasts it within the layer-2 VNI. This entails sending the ARP request out locally over the server facing ports as well as sending a VXLAN encapsulated packet with the layer-2 VNI over the IP core. This follows same process that we saw but only difference is that at reply the ToR will store the MAC binding in his ARP supression cache for further usage

Tuesday, January 17, 2017

Let's Overlay: VXLAN Deep Dive - Part I

9:37 AM

I was getting a lot of technical questions regarding VXLAN and overlays, how did they work, how you can configure them, etc. So I always think that is better if we can share all of that to all of you instead of reply each of you separately.

Let's start with a quick definition on overlays. Overlays, as the name mentions, allow us to reach different points in network without the need of caring about the under layer, you can say "OK, so it's all about tunnels", well I will not lie to you, there is a lot more behind that (address replication, control plane, ARP resolution, etc) but you can start with that vague idea (in coming posts I promise that you will get a better picture of it).

Being said that, one of the most used overlay technologies used and spreaded is VXLAN, which is designed to provide the same Layer 2 network services as VLAN does, but with greater extensibility and flexibility. How do we achieve this? Key aspects to understand are:

VXLAN uses a 24-bit segment ID known as the VXLAN network identifier (VNID), which enables up to 16 million VXLAN segments, this allow us for higher scalability and multi-tenancy
VLAN based designs uses STP in the back to choose best path, VXLAN uses MAC-in-UDP encap and in consecuence he can take advantage of the underlying network (Layer 3) and can take complete advantage of Layer 3 routing, equal-cost multipath (ECMP) routing, and link aggregation protocols to use all available paths, this provides us better use of resources.

VXLAN Packet format

As we mention, it uses MAC-in-UDP encapsulation to provide a means to extend Layer 2 segments across the data center network. The encapsulation scheme used places the original Layer 2 frame with a VXLAN header and then placed in a UDP-IP packet. With this MAC-in-UDP encapsulation is easy to think in tunneling VXLAN across L3 networks, a great and easy way to see this is in the following packet format:

As seeing in the picture, VXLAN introduces an 8-byte VXLAN header that consists of a 24-bit VNID and a few reserved bits. The VXLAN header together with the original Ethernet frame goes in the UDP payload. The 24-bit VNID is used to identify L2 segments and to maintain L2 isolation between the segments.

Also by seeing picture is now easier to get the idea on how tunneling can work with this frame, tunnels are formed between devices which want to exchange VXLAN data and for that only thing needed is destination IP (in Outer IP Header) also once reached we have to do the hard work that every tunnel does encap/decap, and for that we have to introduce other player that is the VTEP or VXLAN Tunnel Endpoint.

VXLAN Tunnel Endpoint

VTEPs are essential players in VXLAN work, as we mentioned earlier their role is similar to any tunnel endpoint (encap/decap) but we will explain in detail how this occur. I would use this simple scenario to do some explanations:

VTEP has presence in a Local LAN segment and has a defined mapping from that segment to a VNID. The encapsulation process consist on taking the L2 Frame sended by any of the End Systems on the local segment, add the VXLAN header with the corresponding VNID, add UDP header, and add Outer IP header (with destination IP for the remote VTEP where we want to send out our packet, if you are now thinking how this entries get populated and how arp is handled you have to wait for Part 2 :) ). Once packet arrives remote VTEP, decap process start by stripping off VXLAN Header and identifying, based on VNID, the local segment in which we have to deliver out our packet.

Being said this the following is pretty self-explanatory (if you follow me, if not here you have a nice picture):

In this figure, Host A is sending out a packet to host B, his associated VTEP, VTEP-1, has an entry for Destination MAC-B in his table and his referring to IP of remote VTEP VTEP-2, he also has the info of the VNID assignment for Host-A, VNID 10. Based on that VTEP-1 has all the info that he needs to encap the packet and deliver it out to VTEP-2. Once packet reach VTEP-2, gets decapsulated, based on VNID is associated to LAN segment where Host-B lives and packet is sent out to the destination (also on VTEP-B and switches if any there know how to reach MAC-B :) )

I think we are good for today, next post in this series will cover how BUM traffic is processed, control plane options and config scenarios. If you want to see any in particular, just let me know.

Friday, January 13, 2017

About Writtens and CCIE DC written prep

10:05 AM

I'm constantly been asked of several things about certs, and particularly about written exams. In this post I will try to summarize two things, one is my personal belief and the second one is the study method / reading list that I followed up.

So, lets start by saying a bad word... dumps. Lot (and i mean a LOT) of people asks me "which dumps have you used to clear XYZ?", "I'm waiting till dumps get more accurate" and so on... My answer is always the same: "you can do whatever you want, but think if time spent by reading questions worth it", how much time do you spend by reading a series of 2^20 Qs? And by memorizing them? And it gets better, do you enjoy that process? If you do, just skip this post since you're not going to need the book list and start thinking if you can memorize cards in a casino, maybe you get a bigger reward ;) if you don't... don't feel silly, we are at least two guys (And believe me I know a lot of people who enjoy the process of reading books instead of memorizing Qs)
Also, another thing that i really want to point out about this is that I know that Cisco is working really hard to overcome dumpers by releasing new Qs everyday (maybe that is too much, but they are working on it, believe me)

Well, after all the introduction setted up (feel free to comment if you agree or not), I will post my study methodology for writtens and my book list for DC (v1, sorry guys.. I will post the books readed for v2 but for written i took v1 in beggining of 2016).

Study methodology

Read the blueprint at cisco cert page and topics included in written (https://learningnetwork.cisco.com/community/certifications/ccie_data_center/written_exam/exam-topics)
Start by identifying the topics that you (think) master and those you definitely don't.
Mix reading of topics you don't know anything of with those that you do, this is key to avoid being overwhelmed by new stuff (your brain will thank you)
Always take notes! For CCIE written exams and similar certs you have to note down those concepts and configuration maximums/limits that you will never remember in real life!
To use as a guide, be sure that you have a section on your notes for each protocol that is in the blueprint, i.e. for FabricPath you should have at least this info:

"Running per Supervisor Engine, on a per-VDC basis:●   FabricPath IS-IS - SPF routing protocol process that forms the core of the FabricPath control plane
●   DRAP - Dynamic Resource Allocation Protocol, an extension to FabricPath IS-IS that ensures network‑wide unique and consistent Switch IDs and FTAG values
●   IGMP - Provides IGMP snooping support on FabricPath edge switches for building multicast forwarding database
●   U2RIB - Unicast Layer 2 RIB, containing the “best” unicast Layer 2 routing information
●   M2RIB - Multicast Layer 2 RIB, containing the “best” multicast Layer 2 routing information
●   L2FM - Layer 2 forwarding manager, managing the MAC address table
●   MFDM - Multicast forwarding distribution manager, providing shim between platform-independent control-plane processes and platform-specific processes on I/O modules

Global components that run on each of the I/O modules, processing forwarding information from each VDC and programming it into the I/O module hardware:
●   U2FIB - Unicast Layer 2 FIB, managing the hardware version of the unicast Layer 2 RIB
●   M2FIB - Multicast Layer 2 FIB, managing the hardware version of the multicast Layer 2 RIB
●   MTM - MAC table manager, managing the hardware version of the MAC address table"

Well I think that finally a book list is expected:

NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures, 2nd Edition
I/O Consolidation in the Data Center
Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems, Applications, Management, and File Systems
Cisco Unified Computing System (UCS) (Data Center): A Complete Reference Guide to the Cisco Data Center Virtualization Server Architecture
Policy Driven Data Center with ACI, The: Architecture, Concepts, and Methodology
Cisco Live docs, don't you use that? You're missing a GREAT resource

BRKDCT-2404 VXLAN Deployment Models - A Practical Perspective
BRKDCT-2370 - Intermediate - End-to-End Application-Centric Infrastructure Automation with UCS Director
BRKDCT-2049 - Overlay Transport Virtualization
BRKDCT-3237 - Advanced - Versatile architecture using Nexus 7000 with a mix of F and M modules to deliver FEX, FabricPath, Multihop FCoE, MPLS and LISP all at the same time
BRKDCT-3145 - Advanced - Troubleshooting Cisco Nexus 5000 / 2000 Series Switches
BRKDCT-3378 - Advanced - Building simplified, automated and scalable DataCenter network with Overlays (VXLAN/FabricPath)

Also lot of blogs... I will write down my RSS feeds soon :) there is really smart people near us :)

Thursday, January 12, 2017

Next Goals 2017

11:00 AM

As many of you already know me I'm not a man who can be relaxed after reaching a goal, I really enjoy chasing new stuff and for this year I 've set up several goals:

CCDE (Passed my written past year, going for lab soon...)
CCIE RnS or SP, I have a story behind this, early last year I've started my RnS trip but people of my team telled me that I've to chase DC and they convinced me, after achieving that goal now is time to back to basics SP and RnS can do that for me.
VCDX-NV, working actively to get it, lot of time consuming but really like it, hope to find time to end with this
VCIX-DCV
JNCIE, I owe juniper the chance to delight me with a lab and hopefully i can find a timeslot at the end of this year to go out for lab (I'm already a JNCIP-ENT just have to take the courage and set up a lab for hands on practice)

As always, this is my wishlist for this 2017, lets see at the end of this year how much do I achieve (if any, hope at least one :) ), in the meanwhile I will keep posting about DC since as you may figured it out I love that track, and also about tech and the preparation for all of this goals, hope you join me in this trip :)

Wednesday, January 11, 2017

IE expert level certification training

12:07 PM

As you may also know, I've recently passed my CCIE DC v2 and i really want to thanks again CCIE-HOME team. If you have read my first post about this track, i do not work for any of them, so comments are on my own.

Being said that I want to start a discussion, and I'm open to comments, about level of bootcamps / training provided for this tracks. Specifically for DC when I start my journey to v2 I feel "alone". No info for any training partners, no bootcamps offering, no twitter posts of people passing...

Also and added to that, I'm based on Argentina, here and I will not miss if I say in whole Latin America, the level of training is so poor... we're used to reach INE or other companies outside of our region to get trained. Based on that I was able to reach CCIE HOME and saw that they we're starting this track and need people interested in join and test the level of their new bootcamp.

After that, and now being passed this track and being also certified as VCIX-NV, JNCIP-ENT and several other certs I've started the path to build my own training company, based only on IE level for all the major vendors (Cisco, Juniper, VMware), being in technology for so many years made me reach a real network of expert professionals in different subjects. Im also instructor for majority of this tracks and being able to know a lot of people trained and capable of deliver a real IE level training, so I've decided to start with a new offering for bootcamps, based in Latin America and for Latin America mostly (I really want to give a Spanish based offering, I know that we need English to learn this track that's for sure but for training we have very well trained people that can deliver the training in our native tongue :) ). I will keep you posted about this, I'm actually training a few guys in VMware and Cisco DC track, after that I will focus 100% on this and hope you joined me :)