If you’re running a VMware vSphere cluster on a two-tier (or greater) Cisco network, you might be in a situation like I was. You see, we built in redundancy when we planned our core and access switches, but the design had one significant flaw (see the simplified diagram to the right). Pretend all of those lines are redundant paths. Looks good so far, right? If CoreA goes down, ESX(i) can still send traffic up through AccessB to CoreB. The reverse applies if -B is down, and likewise for either of the Access- switches.
The catch comes for VMs on ESX(i) when one of the Core- switches goes down. ESX(i) balances VMs across the ports in the Virtual Machine port group(s). If a port goes down, it will smartly move the VM(s) to another port that is up. If an “upstream” hop like CoreB goes down, though, ESX(i) doesn’t know about that event, so it keeps its VMs in place, oblivious to the fact that the VMs on AccessB ports are as good as dead to the world. [Enter Link-State Tracking]
Link-state tracking (LST) is a feature in Cisco IOS 12.2(54)SG and later (and possibly a few minor revisions sooner) that enables Cisco switches and routers to manage the link state of ports based on the status of other port. In our case, LST can be configured on AccessB to watch the uplink port(s) to CoreB and act if something they go down. See the example config below:
switch# config t
switch (config)# link state track 1
switch (config)# int GigabitEthernet1/48
switch (config-if)# link state group 1 upstream
switch (config-if)# int GigabitEthernet1/1
switch (config-if)# link state group 1 downstream
switch (config-if)# int GigabitEthernet1/2
switch (config-if)# link state group 1 downstream
In this config, we have specified that the last port on our 48-port Cisco Catalyst switch (i.e. a 4900 series) is what links (“upstream”) to our core switch, CoreB. Then we add two other ports which our ESX(i) server is using for VMs as “downstream” ports. Once this configuration is in place, if GigabitEthernet1/48 goes down (unplugged, issues on CoreB, etc), AccessB will put GigabitEthernet1/1 and 1/2 into an “ErrorDisabled” state (down), so our ESX(i) server will know that it needs to choose new paths for the VMs that are traversing AccessB.
Of course, another solution to this topology would be to physically reconfigure it as a mesh with CoreA-to-AccessB and CoreB-to-AccessA links, but then you encounter spanning-tree and other factors at multiple levels. Even if that is your end game, link-state tracking is a great intermediate step in the mean time.
For more info on whether beaconing or link state tracking in your best fit, check out VMware’s blog:
Beaconing Demystified: Using Beaconing to Detect Link Failures