My Default Route has gone! Integrating Embedded Event Manager (EEM) into SolarWinds

So….what can we do to detect these issues before the IT director comes to you with an angry face? Well, if you have SolarWinds® a few options come to mind:

• Use SNMP in SolarWinds Network Performance Monitor (NPM) to monitor the routing table and examine the existence of the 0.0.0.0/0 route
• Use SolarWinds Network Configuration Manager (SCM) to schedule show ip route 0.0.0.0/0 commands to check if it exists
• Use SolarWinds Voice & Network Quality Monitor (VNQM) and create an IP SLA operation to 8.8.8.8 (or any other external route)

Yep, all of the above would work well, but there is one that really stands out: receive a Syslog message straight away when we lose the default route. But there is a problem here guys, out of the box, no device will notify you when a route is lost,  they will notify you when a routing neighbour goes down, when an interface goes down, but won’t do that on default routing changes.

But what would you say if I tell you that you can set up a customised Syslog message that is fired every time the default route is lost ( you can even define the content of the message and the priority)?  – Quoting Sheldon Cooper:  No, I’m not crazy! My mom had me tested!

The solution is called EVENT MANAGEMENT

Event management is a powerful and flexible feature available in some network devices that provides a programmatic method to control and perform on-board automation. It gives you the ability to adapt the behaviour of your network devices to align them with your business needs. There are several vendors that have an event management feature such as Cisco Embedded Event Manger (EEM) or Juniper Event Manager.

CAVEAT: The purpose of this blog is not to discuss the full potential of Event management, but rather to demonstrate how to make use of it within the SolarWinds Orion platform. If you want to know more about this cool feature, please visit the links above.

Let’s get our hands dirty
What we are going to do is to configure our devices to notify me when the default route is gone using EEM. I’m going to use a Cisco device for demonstration purposes, however, as we pointed out before, there are other platforms that support this feature.

Imagine the following topology where we have two devices with internet access distributing the default route into the network:

We are using OSPF with a single area in order to redistribute the default route from the WAN routers into our network (PROSWRTRTR01).

The following is the output of the Cisco command show ip route:

Gateway of last resort is 10.0.13.3 to network 0.0.0.0
[code]O*E2 0.0.0.0/0 [110/1] via 10.0.13.3, 00:00:19, FastEthernet1/1
[110/1] via 10.0.12.2, 00:00:38, FastEthernet1/0
10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks
C 10.0.0.1/32 is directly connected, Loopback0
O 10.0.0.2/32 [110/2] via 10.0.12.2, 00:00:38, FastEthernet1/0
O 10.0.0.3/32 [110/2] via 10.0.13.3, 00:00:19, FastEthernet1/1
C 10.0.12.0/24 is directly connected, FastEthernet1/0
L 10.0.12.1/32 is directly connected, FastEthernet1/0
C 10.0.13.0/24 is directly connected, FastEthernet1/1
L 10.0.13.1/32 is directly connected, FastEthernet1/1
C 10.0.14.0/24 is directly connected, GigabitEthernet0/0
L 10.0.14.1/32 is directly connected, GigabitEthernet0/0
172.16.0.0/32 is subnetted, 1 subnets
O IA 172.16.0.1 [110/2] via 10.0.13.3, 00:00:19, FastEthernet1/1
192.168.25.0/32 is subnetted, 1 subnets
O IA 192.168.25.1 [110/2] via 10.0.14.4, 00:00:04, GigabitEthernet0/0
192.168.100.0/32 is subnetted, 1 subnets
O IA 192.168.100.1 [110/2] via 10.0.14.4, 00:00:04, GigabitEthernet0/0[/code]

Everything looking good so far.

Configuration time
First of all I need to monitor default route status on of our Cisco router. Using the track feature will do here:
[code]PROSWRTRTR01 (config)# track 1 ip route 0.0.0.0 0.0.0.0 reachability
PROSWRTRTR01 (config-track)#exit[/code]
Nice… that was easy! Now let’s go ahead and configure Cisco EEM in order to trigger a Syslog message when we lose the default route:
[code]PROSWRTRTR01 (config)#event manager applet DEFAULTROUTELOST
PROSWRTRTR01 (config-track)# event track 1 state down
PROSWRTRTR01 (config-track)#action 1.0 syslog priority errors msg “Default route 0.0.0.0/0 is lost”[/code]

Isn’t it great when configuring a cool feature is so easy? With just four commands we have already configured it.

NOTE: please make sure that your device is configured to send Syslog messages with your SolarWinds server’s IP address as the destination.

Now it’s testing time. On the scenario previously illustrated these two devices have internet access and distribute the default route into the network:

What happens when the default route is lost from one of the devices? Let’s say that PROSWRTWAN01 stops sharing the default route within the OSPF area.
[code]<router>#show ip route
Gateway of last resort is 10.0.13.3 to network 0.0.0.0
O*E2 0.0.0.0/0 [110/1] via 10.0.13.3, 00:02:40, FastEthernet1/1
10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks
C 10.0.0.1/32 is directly connected, Loopback0
(continues…)[/code]
As you can see, we can still reach the Internet as we get the default route from the other device. Let’s break PROSWRTWAN02 too then:
[code]*Aug 3 21:08:53.947: %TRACKING-5-STATE: 1 ip route 0.0.0.0/0 reachability Up->Do wn
*Aug 3 21:08:54.035: %HA_EM-3-LOG: DEFAULTROUTELOST: Default route 0.0.0.0/0 is lost[/code]
This syslog message is the one we have configured and is telling us that the default route is gone as there is not any routing neighbour advertising a route to this subnet (in this case, the default route).
Let’s double check the routing table:
[code]Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks
C 10.0.0.1/32 is directly connected, Loopback0[/code]So…. thus far we have configured our Cisco router(s) to send us Syslog messages if we lose the default route. So we can now go back to the SolarWinds web console and review the information the Syslog dashboard is showing. At this point I’m sure you guys are busy bees and have lots of stuff to do, and most likely don’t have time (or the will) to review that dashboard periodically looking for default route events. No worries, we can configure SolarWinds to forward this syslog message to our mailbox. Steps:

• RDP the SolarWinds server
• Open Syslog Viewer
• Open Rules/Filters
• Add a new Rule
• Change Name to : Default route is lost
• Go to Message tab
  • Syslog Message Pattern: *Default route 0.0.0.0/0 is lost*

• Go to Alert Actions tab
  • Add new action: email

• Complete email recipient and Reply address
• Complete SMTP server

NOTE: there are options that you can implement such as limiting the IP address range, or filtering by severity level, etc… For this particular scenario, the steps above will work for us.

To Recap
In this blog article, we have reviewed how we can use the Event Management feature of your network devices in order to get an immediate notification when any of our network devices loses the default route.This is just an example though, the possibilities for Cisco EEM are endless. As an illustration: we all know Syslog is great but has many downsides: such as verbosity and volume (too many non-important messages being generated) or inconsistency (different devices will send different types of Syslog). This is something that we can solve by using Cisco EEM:

1. We can modify the output of any existing Syslog message,
2. We can modify the severity level (why is interface down only a notification level??!!)
3. We can get alerted immediately instead of having to wait for the next SNMP poll. (you are polling your devices aren’t you?!)
4. We can get notifications for new types of events (ie, default route gone) with customised output. We can even attach the output of a show command in a Syslog message!