So, what it is?
ELAM stands for Embedded Logic Analyzer Module, It is a logic that is present in the ASICs that allows us to capture and view one or more packets, that match a defined rule, from all the packets that are traversing the ASIC. ELAM is not new at all, some of you can remember this from CAT6500, and thats ok, same logic also same from N7K (for the youngest?).and... whats new?
Essentialy the concept is still the same, an we just need to focus on understand how is the architecture inside the ASICs on Leafs and Spines to fully apply this concept.Cisco ASIC data path is divided into ingress and egress pipelines where two ELAMs are present (see figure) at the beginning of the lookup block.
As we can see in the picture Before we can use ELAM to capture a packet, we must be sure that the packet is sent from the BCM ASIC to the Northstar ASIC. ELAM operates only in the Northstar (for leafs, on Spine takes place on Alpine), so any packets that are locally switched in the BCM ASIC will not trigger the ELAM, this is important since in some scenarios the packet will not reach Northstar and will not trigger an ELAM event (we can cover this in a future post about PL-to-PL traffic on ACI fabric :) )
So, assuming that our traffic will be processed by Northstar we need to configure our ELAM instance, first of all is good to know which kind of rules can we configure based on the pipeline, this is also referred as "select lines" and the following are available:
Input Select Lines Supported
3 - Outerl2-outerl3-outerl4
4 - Innerl2-innerl3-inner l4
5 - Outerl2-innerl2
6 - Outerl3-innerl3
7 - Outerl4-innerl4
Output Select Lines Supported
0 - Pktrw
5 - Sideband
With this in mind we can configure our ELAM instance, first of all is always good to have an image to understand the whole process of what we need to do:
Where on INIT we choose the ASIC and pipeline in which the capture should take place, CONFIG refers to the proper configuration of the rulo to match the packets, ARM is like arming the bomb :) but in this case we arm our packet capture to be triggered once the rule defined on CONFIG section has a match, after this READ the captured data and RESET to start over :)
Now lets dig into the packet capture, we will refer to this topology for the capture.
ELAM Example
This image is extracted from a Cisco Live presentation of ELAM but we will focus on LEAF4 only, traffic will traverse from VM1 to the EP at the right going toward Northstar (at 1) and this example is also useful to show how this behaves on Alpine.
We will arm the ELAM on Leaf 4 to capture a packet coming from EP1 (the one at the left side, directly connected to Leaf1). In this example we show use of in-select 3, which means the fields we can match on or outer L2, L3, or L4. We show also the out-select of 0.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Leaf4# vsh_lc | |
| module-1# debug platform internal ns elam asic 0 | |
| module-1(NS-elam)# trigger init ingress in-select ? | |
| 3 Outerl2-outerl3-outerl4 | |
| 4 Innerl2-innerl3-innerl4 | |
| 5 Outerl2-innerl2 | |
| 6 Outerl3-innerl3 | |
| 7 Outerl4-innerl4 | |
| module-1(NS-elam)# trigger init ingress in-select 3 out-select 0 |
This will work for basic ELAM packet capture.As we mention we need to configure (CONFIG section of the ELAM) 1 aspect of the trigger to match on. For this example we will use the SMAC of the locally attached endpoint:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| module-1(NS-elam-insel3)# set ? | |
| outer Mask and Match By Outer Packet Fields | |
| module-1(NS-elam-insel3)# set outer ? | |
| ipv4 IPv4 Fields | |
| l2 All Layer 2 Fields | |
| l4 L4 Fields | |
| module-1(NS-elam-insel3)# set outer l2 ? | |
| cfi | |
| cntag_vld | |
| cos | |
| dst_mac | |
| hg2 | |
| qtag_vld | |
| snap_vld | |
| src_mac | |
| vlan | |
| vntag_dvif | |
| vntag_looped | |
| vntag_pointer | |
| vntag_svif | |
| vntag_vld | |
| module-1(NS-elam-insel3)# set outer l2 src_mac 0050.5665.34bd | |
| !! | |
| !!The start command will arm the ELAM and it will start looking for the first packet that matches the trigger. | |
| !! | |
| module-1(NS-elam-insel3)# start | |
In order to see the ELAM state the status command can be used, esentially three different status can be found:
- Triggered: indicates that a packet has been detected as matching the trigger, and that packet is available for analysis.
- Armed: it means that that no packet has been detected as matching the trigger yet, and ELAM is actively looking at packets for a match to the trigger.
- Initialized: the ELAM is available for triggers to be configured, or to be armed with the start command. It is not currently attempting to capture a matched packet.
Once ELAM is triggered, the packet can be viewed for analysis with the report command. The report will show the relevant header fields in the packet (note that will not show the complete payload of the packet), once this is done we can restart the process with the reset command.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| module-1(NS-elam-insel3)# report | |
| ... | |
| module-1(NS-elam-insel3)# report | egrep ce_|ar_|drop|hg2_src | |
| GBL_C++: [INFO] hg2_srcpid: 0A | |
| GBL_C++: [INFO] ce_da: FFFFFFFFFFFF | |
| GBL_C++: [INFO] ce_sa: 0050566534BD | |
| GBL_C++: [INFO] ce_etype: 0806 | |
| GBL_C++: [INFO] ar_sha: 0050566534BD | |
| GBL_C++: [INFO] ar_spa: 0A108030 | |
| GBL_C++: [INFO] ar_tha: 000000000000 | |
| GBL_C++: [INFO] ar_tpa: 0A108001 | |
| GBL_C++: [INFO] ar_spare: 0000000000000000000000000000 | |
| GBL_C++: [MSG] - pktrw is complete | |
| GBL_C++: [INFO] drop: 0 | |
| GBL_C++: [INFO] hg2_srcpid: 0A | |
| GBL_C++: [INFO] hg2_vid_lo: 63 | |
| GBL_C++: [INFO] vlan0: 063 | |
| GBL_C++: [INFO] adj_index: 000C | |
| GBL_C++: [INFO] ol_encap_idx: 2FF6 | |
| GBL_C++: [INFO] ol_ttl: 08 | |
| GBL_C++: [INFO] ol_segid: 2A8001 | |
| GBL_C++: [INFO] sclass: C005 | |
| GBL_C++: [INFO] sup_redirect: 0 | |
| GBL_C++: [INFO] mcast: 0 | |
| !! | |
| !! Once done, we need to reset the ELAM to start the process again | |
| !! | |
| module-1(NS-elam-insel3)# reset |
This is pretty much all for a good start on ELAM usage for ACI, more info is available at N9K config guide and a good resource as well is the Cisco Live Session BRKACI-2102, from which I already took some images for this post.
Hope you enjoy and next time maybe I found some time to start the amazing post of PL-to-PL traffic on ACI Fabric.
