Living in the underlay

Past week two weeks a colleague and also a student asked me about Phantom RP and how it works, all was related with a discussion we have around VXLAN Part 2 post and about supported Multicast configurations for VXLAN in NX-OS.

First of all, and in order to avoid further confusions around it, I would resume current supported methods for VXLAN underlay on Cisco NXOS/ASR devices:

Being clarified that, we can continue with the original purpose of this post.
So, based in our previous post we have configured our Nexus 5K / 7K underlay to run multicast in to support Flood and Learn configuration, by that time we choose Bidir PIM since is the only supported method in N5K. So let's get some background about bidir and how can we make it redundant (can we?)

BiDir PIM

PIM Bi Directional mode enable multicast group to route traffic over a single shared tree rooted at the RP, instead of using different unidirectional or sources tree. Since RP is the root  (his IP address :) ) is good to not to place it on a router but on an unused IP on the network reachable from PIM domain (this will be seen later in PhantomRP configuration).
Explicit join messages are used to establish group membership, Traffic from sources is unconditionally sent up the shared tree toward the RP and passed down the tree toward the receivers on each branch of the tree (note: traffic is not sent unidirectional to RP)

Bidir-PIM shares mechanisms of PIM-SM like unconditional forwarding of the source traffic toward the RP but without the registering process for sources (https://tools.ietf.org/html/rfc7761#section-4.2). Based on that forwarding can take place based on (*,G) entries, removing the need of any source specific state and, therefore, expanding scaling capabilities. This image extracted from Cisco white paper are good to see the differences in upstream process towards the RP in SM vs BiDir:


"PIM-SM cannot forward traffic in the upstream direction of a tree, because it only accepts traffic from one Reverse Path Forwarding (RPF) interface. This interface (for the shared tree) points toward the RP, therefore allowing only downstream traffic flow. In this case, upstream traffic is first encapsulated into unicast register messages, which are passed from the designated router (DR) of the source toward the RP. In a second step, the RP joins an SPT that is rooted at the source. Therefore, in PIM-SM, traffic from sources traveling toward the RP does not flow upstream in the shared tree, but downstream along the SPT of the source until it reaches the RP. From the RP, traffic flows along the shared tree toward all receivers."

Need of redundancy? Let's do it

We mention that our shared tree is rooted at RP address, so in order to give him redudancy we need a way to duplicate this or use a virtual IP. For bidir pim no traffic is targeted at RP (no control plane functions) so our solution is easier, instead of actually assign same IP in a sort of anycast we can just advertise it thru our IGP, the only issue foreseen is that the actual shared tree should be only one at a given time (we dont want that our RPF interface changes everytime) so in oirder to avoid that we can leverage the path decision to a more specific match in the RIB (by advertising same subnet with largest mask by some of the redundant points).
Well, that was so much talk I think that a code/config snippet worths more than a millon words:


Primary


Secondary (hmm.. if you don't see any difference here is a hint: look at the mask)




Now it's done, you can run your set of favourite verification commands to see if this is working:



Also you can shutdown the active interface (lo1) and see how does this change and our redundancy is working.

For CCIE / CCDE students:
- What is the convergence time of RP in case  of a failure on primary?
- Can we give sub-second convergence?
- In flood and learn configuration for VxLAN what would you recommend ASM or bidir PIM?
- In case of choosing ASM how is your redundancy going to be solved?
- Why are we using "ip ospf network point to point" ?

More on Multicast ASM/SSM/Bidir comparisson: http://lostintransit.se/2015/08/09/many-to-many-multicast-pim-bidir/

Finally, after a tough work done by me and the CCIE HOME team, we ended creating ie-bootcamps, the first expert level training company based in Latin America that will deliver courses and bootcamps for the most challenging tracks. We plan to cover America completely, and as a start point we introduce the CCIE DC v2 Lab Bootcamp that would be held at Buenos Aires, Argentina in May 22-26th. I will be delivering the course, so you are all invited :)

More info at: http://ie-bootcamps.com/course/ccie-dc-v2-0-lab-bootcamp/
Or reach me directly

-----

Finalmente tras un largo trabajo de parte del equipo de CCIE HOME y mio, hemos dado a luz a la primer empresa encargada de dar capacitaciones de nivel experto en habla hispana: ie-bootcamps.
Como primer medida, hemos arrancado con la coordinacion del bootcamp de CCIE DC v2.0 a darse en Buenos Aires, Argentina el 22 al 26 de Mayo.

Past month I was involved in a project to test the performance of several All Flash Array boxes, let's call them AFAs, since it is expected to start delivering new class of services (and SLA?) to customers.

Being in a R&D team gives you the opportunity to know the whole part of the story, you start with sales pitch, then the pre-sales *not-so-technical and at the end you break a box in a PoC and you end with real engineers which explains you the details of his architecture and why they don't support what you just test (but it was on the sales pitch, right?)

What I do want to remark here is the relation between vendor hype and business needs. AFA can brings you an outstanding performance in amount of IOPS, latency, compression, and so on... but the point is, what do you really need?. In any architecture design you are supposed to deliver a solution that mets technical requirements + business requirements/needs.  In AFA this could be quite tricky since if you don't have a clear understanding on your business needs you're going to be pushed to an unfair or un-precise comparison.

So, what about technical requirements?

I've faced two kinds of scenarios for the AFA deployment. The easier was the deploy of a new infrastructure aimed to fulfill specific requirements for an application suite, this is always the best case scenario for an architect since you can gather technical requirements easily including not only actual requirements bur also a forecast for upcoming demands. As you may know there are not all pink elephants, and the other well seen scenario is to move a current deployment to an AFA solution.
In the latter case you have to take a huge amount of considerations in order to do the planning accordingly, I can summarize the following:

• Amount of IOPS: This is something that you can see a lot, and is completely wrong, since what is 1K IOPS? Which block size? which read/write ratio?
• Amount of IOPS based on a given IO distribution which includes block size and read/write ratio of each of those. Getting this numbers can be significant hard, most of current arrays have the information of all the workloads they had run, this is amazing for getting the IO distribution per block size plus rd/wr ratios but assumes that you are running similar workloads all day long (i.e. consistent distribution of a given set like 30K IOPS @4k 65/35, 15K IOPS @8k 60/40.. and so on, but what about the nightly jobs? when your performance gets affected with backup jobs?)
• Expected  and maximum latency: Based on application + OS/Guest OS needs 
• Expected compression ratio (for your data set!)
• HA and expected performance in contingency
• Network based (NFS), IP based (iSCSI) or FC? 
• Disk replacement policy and MTBF/MTTF

Ok, but where is the vendor Hype?

Well, quick answer is everywhere you get a sales/pre-sales engineer talking, but to be targeted to the topic I've found solutions that claims in a  single Box 100K IOPS @32K block size.... easy math here is to ask them how many IOPS did they support in the case of 4/8K if they reach that at 32, also how many bandwidth did they expect?

How can I test performance an avoid being seduced by pink elephants?

When we begin the test process we aim at a huge amount of VMs with RDM to the array but later we figured it out that there is no way to do such, with few VMs with lot of disk in each you can easily setup a quick test.
For the testing procedure and consideration IDC has written a good recap in theirs "All - Flash Array Performance Testing Framework" and also there is a Tool made by EMC engineers to test arrays called "AFA PoC Toolkit" which setups a few VMs under vmware host, connect the host to the LUNs of the array then makes RDM and set up VDBench on the VMs.
I do recommend using this approach by changing the parameters on VDB files to meet your requirements. Also there is one caveat with that Toolkit and is that in only runs with EMC XtremIO, I've made several changes to be able to run it against PureStorage boxes and i'm doing the changes to be able to test against SolidFire too. In a later post we will discuss design considerations for an AFA platform and the testing tools plus results for each vendor (We're testing PureStorage, EMC XtremeIO, NetApp SolidFire)

After passing DE want to take some time and write this words to all futures aspirants and also for those who failed and also pass to see their point of views regarding CCDE. I was thinking about strategy in lab and mind strategy as well and I realize that being a chameleon is one of the important aspects that you will need to address in order to be successful in this path.

Get yourself use to read and read

Be in a group and collaborate as much as you can

Color scheme for highlighting

Be a chameleon

Suboptimal routing still works right?

Well guys, you may note that I was out of blogging for a while, and there was a reason behind that... CCDE was so time consuming, I spend at least 4 to 6 hours last two months with a my awesome study group debating scenarios, design choices and pros/cons of each of them... it was real hard, it requires ****a lot**** of reading but definitely worth it!! Past Feb 22nd I pass the lab, it was at my first attempt and I feel very lucky about that but it was really about training, reading, understanding and debate with my colleagues what make me succeed, I will write down a post with strategy and training used as well, but at this moment just want to take me some time to write down this line and congratulate all my friends and colleagues who have taken the CCDE lab, passed or not, was a real good experience and an amazing learning!



Keep you updated guys :)
CCDE 2017::27

As many of you already know me I'm not a man who can be relaxed after reaching a goal, I really enjoy chasing new stuff and for this year I 've set up several goals:

• CCDE (Passed my written past year, going for lab soon...)
• CCIE RnS or SP, I have a story behind this, early last year I've started my RnS trip but people of my team telled me that I've to chase DC and they convinced me, after achieving that goal now is time to back to basics SP and RnS can do that for me.
• VCDX-NV, working actively to get it, lot of time consuming but really like it, hope to find time to end with this
• VCIX-DCV
• JNCIE, I owe juniper the chance to delight me with a lab and hopefully i can find a timeslot at the end of this year to go out for lab (I'm already a JNCIP-ENT just have to take the courage and set up a lab for hands on practice)