PerfStack (Performance Analysis) is a feature introduced into the SolarWinds® Orion platform 2017.1 and despite how fantastic this feature is, we’ve noticed as we work with our customers, the few that are aware of it or not taking full advantage of it.

Just a quick introduction; PerfStack is and easy to build dashboard that allows you to drag and drop current and historical data metrics of your monitored infrastructure, across all Orion modules. The way the data is presented, with metrics stacked on top of each other, provides an interface that makes is easier and more efficient to investigate incidents, identify root causes of intermittent issues and present in views.

In this post I wanted to show you one of the ways to utilise PerfStack is to build a dynamic link, based on a URL built-on a specific set of metrics covering distinct monitored elements and a historical timeseries. This URL can then be used within Alert notifications, within other areas of the web interface to allow users to access this rich interface quickly and easily.

As an example, an alert for packet loss on an interface is a well known issue that might be experienced on a network from time to time. The root cause of packet loss can vary from a lose cable connection to a duplex mismatch but, in all likelihood, it will be some performance or configuration issue. Let us utilise PerfStack within an alert notification so which ever interface triggers this alert the Perfstack URL link can be embedded providing this powerful analysis method.

To build a PerfStack dashboard got Performance Analysis under Home = My Dashboards.

Note: You can navigate directly to http(s)://OrionServer/ui/perfstack/

We’ll be listing the metrics I’m interested in that will provide insight to packet loss.

• Availability
• Received Discards
• Transmit Discards
• Received Errors
• Transmit Errors
• Average Receive bps
• Total Packets Received
• Total Packets Transmitted
• Received Percent Utilization
• Transmit Percent Utilization

From the parent node, I’m also adding:

• Average CPU Load
• Average Percent Memory Used

It will look something like this.

• The time period PerfStack is covering
• The metrics that are displayed
• The object on which the metrics are reporting on
• Metrics are separated or grouped in charts using the semicolon. A comma, within semicolons, groups those metrics in a single chart.

[code]http(s)://OrionServer/ui/perfstack/?presetTime=last12Hours&charts=0_Orion.NPM.Interfaces_98-
Orion.NPM.InterfaceAvailability.Availability;0_Orion.NPM.Interfaces_98-
Orion.NPM.InterfaceErrors.InDiscards,0_Orion.NPM.Interfaces_98-
Orion.NPM.InterfaceErrors.OutDiscards,0_Orion.NPM.Interfaces_98-
Orion.NPM.InterfaceErrors.InErrors,0_Orion.NPM.Interfaces_98-
Orion.NPM.InterfaceErrors.OutErrors,0_Orion.NPM.Interfaces_98-
Orion.NPM.InterfaceTraffic.InTotalPkts,0_Orion.NPM.Interfaces_98-
Orion.NPM.InterfaceTraffic.OutTotalPkts;0_Orion.NPM.Interfaces_98-
Orion.NPM.InterfaceTraffic.InPercentUtil,0_Orion.NPM.Interfaces_98-
Orion.NPM.InterfaceTraffic.OutPercentUtil;0_Orion.Nodes_23-
Orion.CPULoad.AvgLoad,0_Orion.Nodes_23-Orion.CPULoad.AvgPercentMemoryUsed;[/code]

By understanding the URL structure, we can include this URL as a link in email alert. If you have ever looked at the URL format when drilling into a detail page, the numbers within the blue wording above will make sense. Just to be clear, when you navigate in Orion to say the Node Detail page, you will see the URL format includes the internal database ID of that node. This is the same value represented above, which in our example is NodeID 23;

[code]https://pros-orion-05/Server/0/Orion/NetPerfMon/NodeDetails.aspx?NetObject=N:25[/code]

In this format, this link is static and points to a specific set of elements. We need this link to be dynamic by automatically inserting relevant elements as alerts are triggered. In this scenario, this can be done by replacing Interface ID, and Node ID, with the appropriate variables.

[code]http(s)://OrionServer
/ui/perfstack/?presetTime=last12Hours&charts=0_Orion.NPM.Interfaces_${N=SwisEntity;M=InterfaceID}-
Orion.NPM.InterfaceAvailability.Availability;0_Orion.NPM.Interfaces_${N=SwisEntity;M=InterfaceID}-
Orion.NPM.InterfaceErrors.InDiscards,0_Orion.NPM.Interfaces_${N=SwisEntity;M=InterfaceID}-
Orion.NPM.InterfaceErrors.OutDiscards,0_Orion.NPM.Interfaces_${N=SwisEntity;M=InterfaceID}-
Orion.NPM.InterfaceErrors.InErrors,0_Orion.NPM.Interfaces_${N=SwisEntity;M=InterfaceID}-
Orion.NPM.InterfaceErrors.OutErrors,0_Orion.NPM.Interfaces_${N=SwisEntity;M=InterfaceID}-
Orion.NPM.InterfaceTraffic.InTotalPkts,0_Orion.NPM.Interfaces_${N=SwisEntity;M=InterfaceID}-
Orion.NPM.InterfaceTraffic.OutTotalPkts;0_Orion.NPM.Interfaces_${N=SwisEntity;M=InterfaceID}-
Orion.NPM.InterfaceTraffic.InPercentUtil,0_Orion.NPM.Interfaces_${N=SwisEntity;M=InterfaceID}-
Orion.NPM.InterfaceTraffic.OutPercentUtil;0_Orion.Nodes_${N=SwisEntity;M=NodeID}-
Orion.CPULoad.AvgLoad,0_Orion.Nodes_${N=SwisEntity;M=NodeID}-
Orion.CPULoad.AvgPercentMemoryUsed;[/code]

This is an example of an email alert with PerfStack URL embedded into the HTML code of the email.

This works across all areas of Orion, so we can take this approach for any other common issues which may occur. For example, for a High CPU Utilisation alert on a server hosting a Job Queue Service. The incorporated PerfStack link can include;

• Average Response Time
• Average CPU Load
• Average Percent Memory Used
• Peak CPU Load (for every core)
• Job Queue Stats (for an application)
• Server Configuration Changes (Statistical)
• Alerts (Statistical)
• Events (Statistical)

[code]http(s)://OrionServer/ui/perfstack/?presetTime=last24Hours&charts=0_Orion.Nodes_149-
Orion.ResponseTime.AvgResponseTime;0_Orion.Nodes_149-Orion.CPULoad.AvgLoad,0_Orion.Nodes_149-
Orion.CPULoad.AvgPercentMemoryUsed;0_Orion.Nodes_149-
Orion.CPUMultiLoad.MaxLoad;0_Orion.APM.GenericApplication_246-
Orion.APM.PortEvidenceChart.37.27587,0_Orion.APM.GenericApplication_246-
Orion.APM.PortEvidenceChart.37.27586,0_Orion.APM.GenericApplication_246-
Orion.APM.PortEvidenceChart.37.27588;0_Orion.Nodes_149-
Orion.PerfStack.ScmChanges;0_Orion.Nodes_149-Orion.PerfStack.Alerts;0_Orion.Nodes_149-
Orion.PerfStack.Events;[/code]

These PerfStack Dashboards can be shared with other parties, if they have access on the Orion Platform, and the necessary permissions on the elements being reported on.

What metrics are available to you is dependent on the installed SolarWinds Orion modules. For example, SAM is required for metrics like processes/services, online resources in a cluster, requests/sec (e.g. Sharepoint), etc.. and, SRM for LUN metrics like IOPs, Throughput, Cache Hit Ratio, etc…

Don’t forget, if you are creating a Perfstack project to analyse an issue, just simply copy and paste that link to a colleague if you wish to share your data and findings.

This is just the beginning. As these examples demonstrate, with PerfStack, you can incorporate ‘end-to-end’ metrics, from across your monitored infrastructure, that represent a critical service in a single link, which is embedded into an alert for your support engineers to have instantaneous insight onto an incident.

We have all had an application or service that runs away with itself and consumes all of the compute resource on a server, where you know that a simple restart of the service or killing of the process will restore operation. For some, this is the pre-cursor IT fix before rebooting the machine, as that invariably gets the right result. If you have a slow running machine or application, and you happened to be on the desktop of that machine, you are going to go look at Task Manager to see what is consuming all of the compute resource.

If you are using a monitoring solution, such as SolarWinds Server and Application Monitor (SAM), you are no doubt monitoring those key servers. Now the big question comes where you know for absolute certainty (notice how this is in bold?!) that if your server consumes continuously more than say 95%, that stopping the process with the highest resource usage will resolve the issue. This is a big if and this post should if you have not already picked up, will give you a method within Orion itself to automate that operation, but you must be at that 100% to apply the capability outlined in this post.

If you are at this place the ability for Orion to identify the issue and fix the issue and all you need to see is the alert notification to confirm it did this, you would be kinda happy right?

OK, I have given enough of a warning to configure this function in your Orion, so lets get to it.

First of all you need to be monitoring your servers within Orion SAM, which will provide CPU and memory utilisation for the whole system.

Now let’s look at these actions in detail:

Action 1: Execute program : SolarWinds.APM.RealTimeProcessPoller.exe

This action in type of “Execute external program” and it executes SolarWinds® out-of-the-box tool; “SolarWinds.APM.RealTimeProcessPoller.exe” in order to poll in realtime for the top 5 processes sorted by “Physical Memory” and insert this information to “Alert Notes” area for the triggered alert object. This is the actual command to be run by the action:

[code]APM\SolarWinds.APM.RealTimeProcessPoller.exe -n=${NodeID} -alert=${N=Alerting;M=AlertDefID} -count=5 -sort=PhysicalMemory -timeout=120[/code]

The available parameters that can be provided for this Solarwinds tool and details can be found in related SolarWinds article.

Action 2: Execute program : Send an Email

This e-mail notification action will be executed 2 minutes after the first action in order to give enough time to the first action to pull top processes and update Alert Notes.

Email Subject: Memory load on ${Caption} is currently ${PercentMemoryUsed}

Email Message:

The Memory on ${Caption} is currently running at ${PercentMemoryUsed}. The top 5 processes running at the time of this poll are listed below:

[code]${N=Alerting;M=Notes}<br /> <code>For more information click the link below.</code><br /> <code>${NodeDetailsURL}[/code]

These parameters will allow us to send the Email notification to related IT contact to inform high memory usage issue on a server with the list of the top 5 processes consuming the most memory. You may wish to stop here, with this information in the external notification and visible in the Alert Details page in Orion, this may be sufficient for you to manually take action.

If you want to automate, then buckle up..

Action 3: Execute program : KillWindowsProcess.ps1

This action in type of “Execute external program” and it executes custom PowerShell script to get the top process information from alert notes and kill that process on the server that triggered the alert. This is the actual command to be run by the action:

[code]C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe "C:\Webinar_Scripts\KillWindowsProcess.ps1" "’${N=SwisEntity;M=Caption}’" "’${N=Alerting;M=AlertObjectID}’"[/code]

As shown in this command, we defined custom script “KillWindowsProcess.ps1” in folder “C:\Webinar_Scripts” of Primary SolarWinds server. This script gets two parameters; first parameter is server name and the second parameter is alert object ID.

Action should be configured to be executed with a domain account that can connect and kill process on destination server:

“KillWindowsProcess.ps1” script connects to SolarWinds API to pull top process information from alert note and then checks the found process if it is in protected process list. It is recommended to define a protected process list to prevent killing critical processes. We do NOT want to render a server inoperable!. If the process is not in the protected list, then the script tries to kill that process on the  destination server. The script generates a log file in “C:\Webinar_Scripts\logs” folder for each day, which can be reviewed to identify and fault diagnose the actions performed. Here is the content for the script:

[code]#Logging Details
$WORKING_FOLDER = "C:\Webinar_Scripts\"
$LOGFILE = $WORKING_FOLDER + "logs\action_trigger_log_" + $(get-date -f yyyy-MM-dd) + ".txt"

# List of protected services that should not be terminated
$PROTECTED_PROCESSES = @(‘sqlserver.exe’,’WmiPrvSE.exe’, ‘svchost.exe’)

# Variables
$targetProcessName = ""
$targetHostName = $args[0]
$AlertObjectID = $args[1]

$time = get-date
$message = "$Time – Started action for Hostname: $targetHostName and Alert Object ID: $AlertObjectID"
$message >> $LOGFILE

# SolarWinds Connection Properties
$hostname = "SOLARWINDS_HOSTNAME"
$username = "SOLARWINDS_API_USER"
$password = "SOLARWINDS_API_PASSWORD"

$swis= Connect-Swis -Hostname $hostname -username $username -password $password

# Get Alert Note for the provided Alert Object
$alertQuery = "SELECT TOP 1 AlertNote FROM Orion.AlertObjects WHERE AlertObjectID=" + $AlertObjectID

$alertData = Get-SwisData $swis $alertQuery

$noteLine = $alertData.Split([Environment]::NewLine,[System.StringSplitOptions]::RemoveEmptyEntries)

$noteLineCount = $noteLine.Count

# Parse Alert Note to get Top Process Information
if ($noteLineCount -gt 1)
{
$headRow = $noteLine[0]
$topRow = $noteLine[1]

$headItems = $headRow.Split("`t",[System.StringSplitOptions]::RemoveEmptyEntries)

$headerItem = $headItems[0]

if($headerItem -eq ‘Name’)
{
$topProcItems = $topRow.Split("`t",[System.StringSplitOptions]::RemoveEmptyEntries)
$targetProcessName = $topProcItems[0]

}

}

if (($targetProcessName -ne "") -and ($targetProcessName -ne $null))
{
$time = get-date
$message = "$Time – Identified top process is : $targetProcessName for Alert Object ID: $AlertObjectID"
$message >> $LOGFILE

$IsProcessProtected = 0

forEach($p in $PROTECTED_PROCESSES)
{
if($targetProcessName -eq $p)
{
$IsProcessProtected = 1
}
}
if($IsProcessProtected -eq 0)
{

$Processes = Get-WmiObject -Class Win32_Process -ComputerName $targetHostName -Filter "name=’$targetProcessName’"

$foundProcessCount = $Processes.Count
$foundProcessName = $Processes.Name

if (($foundProcessCount -gt 0) -or ($foundProcessName -eq $targetProcessName))
{
foreach ($process in $Processes)
{
$returnval = $process.terminate()
$processid = $process.handle

if($returnval.returnvalue -eq 0)
{
#write-host "The process $targetProcessName `($processid`) terminated successfully"
$time = get-date
$message = "$Time – The process $targetProcessName `($processid`) terminated successfully"
$message >> $LOGFILE
}
else
{
#write-host "The process $targetProcessName `($processid`) termination has some problems"
$time = get-date
$message = "$Time – The process $targetProcessName `($processid`) termination has some problems"
$message >> $LOGFILE
}
}
}
else
{
$time = get-date
$message = "$Time – The process $targetProcessName can not be found on Hostname: $targetHostName "
$message >> $LOGFILE
}
}
else
{
$time = get-date
$message = "$Time – The process $targetProcessName is in the protected list, it will NOT be terminated"
$message >> $LOGFILE
}
}
else
{
#write-host "Unable to find target process for Alert Object ID: $AlertObjectID"
$time = get-date
$message = "$Time – Unable to find target process for Alert Object ID: $AlertObjectID"
$message >> $LOGFILE
}

$time = get-date
$message = "$Time – Completed action for Hostname: $targetHostName and Alert Object ID: $AlertObjectID"
$message >> $LOGFILE[/code]

This is an example of using the Alerting capabilities to take remediation action to resolve a problem, which I hope you find useful, but please use with care!

SolarWinds IP Address Manager (IPAM) module is a great tool to directly manage and monitor with rich visual presentation, the usage of IP addresses in a network. SolarWinds® provides multiple options to populate subnet information in our network to IPAM. Subnet information can be added manually, a spreadsheet can be imported, DHCP integration can be used to continually pull directly from DHCP servers and for the more adventurous or needy out there, via the Orion API.

As we can run different SolarWinds modules in a single platform; when we populate our subnets to IPAM and start monitoring IP addresses in our network, that IPAM scope also includes management IP addresses of our nodes that are currently monitored or will be monitored using Network Performance Monitor Server & Application Monitor modules. This is the starting point for our blog post; we can use subnet data in IPAM to enrich node information.

Let’s look at a standard subnet definition in SolarWinds; we must have Subnet Address and CIDR information in our subnet definition but at the same time, we can also customise other fields; description, location, subnet name, etc.

These fields can be used to enrich node custom properties and this approach can be automated using “Set Custom Property” action in alerts to update properties for newly populated nodes.

In order to demonstrate that feature, we have defined an alert to populate “Site” custom property of nodes based on “Location” attribute of IPAM subnets.

Requirements:

• IPAM subnets were defined with “Location” attribute value:

Alert Definition:

We defined a new alert as shown below, it is important to use less frequent evaluation time as we don’t need to check new devices every minute. Severity of the alert should also be selected as “Notice” or “Informational” as we are defining this alert for automation rather than the normal monitoring data analysis.

In “Trigger Condition” tab, we should select “Custom SQL Alert” in order to define SQL condition to correlated IPAM subnet dat with node information:

SQL Condition:

[code]INNER JOIN
(
SELECT DISTINCT ipaddress,
subnetid
FROM ipam_node) iplist ON nodes.ip_address=iplist.ipaddress INNER JOIN
(
SELECT groupid AS subnetid,
friendlyname AS subnetname,
location
FROM ipam_group
WHERE grouptype=8) subnets ON iplist.subnetid=subnets.subnetid WHERE nodes.site IS NULL[/code]

“Reset Condition” and “Time of Day” tabs can be skipped with default settings, as these are not relevant for our function:

In “Trigger Action” tab, we should add the “Change Custom Property” action type and we will utilise the advance ability for using a custom SQL query for “Site” custom property value:

“Reset Actions” tab can be skipped without adding any action and alert should be saved using “Submit” button on the last “Summary” tab. Once submitted, the alert will do it’s thing, pulling data from IPAM and inserting into the Custom Property data fields where we should see updated “Site” custom property values for nodes.

This is an example of using the Alerting capabilities to update data, which I hope you find useful. Please feel free to download the created Alert definition for importing into you installation. You will need to create the Custom Property structure or update the alert if your ‘site’ field is named differently.

Capacity Forecast is one of the features available in the SolarWinds® Orion platform, that allows you to monitor the trends of some of the main KPIs in your network. However, SolarWinds Orion will not analyse the capacity forecast information for every single metric that is being gathered in the platform.

Available out of the box, only CPU and memory at the node level, interface traffic at the interface level, and disk space used at the volume level are compatible with the capacity forecast feature.

The Problem

There have been situations where I would have loved to get the capacity forecast information for other metrics, especially when analysing future capacity requirements for my applications, but as mentioned before, SolarWinds Orion does not give us that info for every single metric.

This might look like a problem, however, SolarWinds Orion stores the historical data for the whole retention period, i.e. the data is available for us, we just need to create the report to show the data in the format we want.

The Report

The following custom Orion report displays the capacity forecast analysis of all the performance counters monitors in SolarWinds Server & Application Monitor, showing the average value of that monitor per month for the last 6 months, the average increase/decrease per day, the warning and critical thresholds (if defined) and the amount days left until the metric reaches the threshold.

This report may be very useful for lots of customers, however, in big installations of SolarWinds SAM, there might be thousands of performance counters monitors, and some of those monitors maybe not as important as some others.

It would be nice if we could filter the performance counters displayed on this report in some way and show only the important ones. We can do this by creating SQL filters with the SQL query used to create this report.

There are three filters that we will use to include the monitors required. In order to edit the filters, we just need to edit the SQL query configured on the data selection of the report.

The three filters used here are and note how I have utilised SQL variables to separate the filter values, which I prefer just to make it easier to define, as it can be edited outside of the SELECT statement:

• @Component: filters by component name
• @Application: filters by application name
• @node: filters by node name

These three filters can be used at the same time if required. As an example, the following configuration would show all the components where the component name contains ‘job engine’ and application name contains Orion and node name contains ‘SW’.

In case you don’t want to use one or more of these filters, just leave the variable with no value (as default).

Real Case

The first time I thought about creating this report was when a SolarWinds customer asked me to create a capacity report that showed the trend of web users on their IIS servers. They wanted to know if their current web servers were currently saturated, and, if they were not, they wanted to know when they will require a new webserver to be added to the pool in order not to impact on the web performance. Typically, a line or bar chart would be used, as this gives a nice clear trend display, but where many lines would be shown, it would make it very difficult to read information out or would require multiple charts. Therefore, a different approach was necessary, hence this report.

In this scenario, we can use this report using the application name filter @application = ‘IIS Web Users’. The ‘IIS Web Users’ application is a custom template I’m using instead of the Appinsight for IIS template due to license restrictions (the NFR licenses are quite small). Otherwise, if you guys are using Appinsight for IIS, I would use the filter @application = ‘Microsoft IIS’ and @component = ‘Current’.

NOTE: the default retention period of SAM is 90 days, which means you will be restricted to this time period of analysis. If you wish to extend it to a period your report requires to cover, this setting can be edited on SAM Settings -> Data and Database Settings. It is important to note, this setting has a significant impact on Database size, as more data is being recorded and therefore please ensure your SQL Server can both store, but more importantly compute resource to allow it to process the higher data volume.

So, if you wish to use a report like this one, please feel free to download the report available below, and please let us know your feedback.

I hope this article has been informative for you and thank you for your time reading it.

I know that there is the option to automatic geolocate devices based on the field ‘location’ available in SNMP, with the following KB detailing how to use this, but:

• who is using that field properly?
• what happens with the rest of the devices? Say devices monitored using WMI, Agent or even ICMP

There is also a way to import coordinates within the GUI of Orion, but this is a manual process, which would need to be performed regularly in order to keep objects on the map up to date.

That’s why in Prosperon Networks we have created a SQL script that will set the coordinates of the devices based on the value of a custom property. This script can be scheduled to be run every day, or any other frequency that you might consider in order to place any new device on the map.

There are just a few prerequisites in order to use this script properly:

• Create a custom property named Site
• Assign the correct Site to each device
• Modify the SQL script adding your locations and the coordinates of those locations in Site: Add more locations just by adding more options in each case.
• Create a job in SQL to run this script every day

BONUS INSIGHT

We can use IP addresses instead of the custom property site if you wish. The script would look like
…

Case

When IP_address like ‘192.168.% then 50.813104

When IP_address like ‘172.16.1.% then 41.429853

One of the common pains in the monitoring world is alerting, and more specifically how to avoid being flooded with false alerts.

What is an Alert

Before we start reviewing how to minimise false alerts, we should define what a false alert is, and in order to do so, first of all, we need to define what an alert is.

An alert is an event that occurs within a monitoring solution where it identifies a potential status or health issue within the ICT infrastructure that can potentially generate issues or problems. Hence a false alert is when an alert is generated but there is no potential issue assigned to that event.

What kind of event can generate a potential issue? Well, this is a tricky question, because what a potential issue looks like will differ among different people and different infrastructure. For some people having a device with CPU 100% is a potential performance issue, whereas for others this is an acceptable condition. There is, and never can be, a one size fits all when it comes to the conditions that require an alert to be generated.

How to Minimise False Alerts in SolarWinds

Can we eliminate all false alerts? I’m afraid this is a very difficult task, however, we can try to minimise them using some techniques and features that we have in SolarWinds® Orion alerts.

We have the following tips for you:

• Monitor what you really need
• Create only meaningful alerts
• Use time delays before triggering
• Use object level thresholds
• Filter common offenders
• Take advantage of Complex conditions
• Custom SQL/SWQL definition alerts

Monitor What You Really Need
It is very impressive when you talk to your colleagues and you tell them how you manage to gather 300+ different metrics from your switch, but, in reality, do you really need everything. Do you just need to know the traffic utilization of the interfaces, CPU/Memory and the VLANs configured?

Monitor only what you really need will avoid temptations of creating alerts for metrics that you don’t really need.

Create Only Meaningful Alerts
Even though you have picked carefully the metrics you really need, it doesn’t mean that all the metrics need an alert assigned.

For example, it is very useful to know how many people are connected to one of your websites. This allows you to correlate this value with other metrics such as database activity, website response time, etc…, but you probably don’t need to receive an alert when the number of users connected is higher than XX. Include this information in the alert output when the performance of the website is poor, but don’t trigger on this single threshold.

Use Time Delays Before Triggering
One simple but powerful trick to reduce the alerts is waiting some time before triggering the alert. This way we make sure the condition that we have defined is true for a continuous period of time, avoiding alerts generated by spikes.

For example, where you are monitoring metrics such as CPU load or interface traffic, it is better to wait for one, two, or even more polling intervals before triggering the alert. Otherwise, just a spike of CPU can potentially trigger an alert when that situation is not a potential issue on your network. In the diagram above, the default High CPU alert would have triggered, when in fact the red cross is a more appropriate point to fire an alert.

On the alert Trigger page, utilise the following setting, converting the polling frequency into a value here, for example, 600 seconds would guarantee at least 2 polls of data are evaluated on a node statistic value polled every 10 minutes.

Use Object Level Thresholds
I think that configuring an alert that is triggered when any volume percentage of space used is more than 95% is a good idea. Full disks are a clear issue on your network because it can stop applications, not allow backups to be stored, etc.

However, what happens with disks that are 10TB+ of size? 5% would mean at least 500GB free space available, and I’m sure you don’t want to receive an alert when a disk still has more than 500GB capacity.

Using custom thresholds would allow you to receive only alerts when the problem is real. In SolarWinds there are two ways to use custom thresholds:

• Built-in custom thresholds: available on some metrics such as CPU, memory, packet loss, interface traffic, errors and discards, custom pollers, SAM components, VM metrics,…

• Using custom properties: for any other metric that doesn’t have a built-in custom threshold, you can create custom properties to be used as custom thresholds. Examples: percentage disk space used, temperature, connections,…

In those situations, the best way to move forward is to eliminate the option of that device triggering the alert. There are two main options here:

• • Mute alerts on that device: this device will stop triggering any alert. Bear in mind that any alert means any alert.

• Editing trigger conditions
• Create a Custom Property to which you can use as a filter to exclude from common alerts
• Priority, for example, could work well for this, where this is an integer and the definition states ‘is not equal to’ 3

Take Advantage of Complex Conditions
In some situations, one way to configure the ultimate alert that you need is by using ‘complex conditions ’. This is an option available in SolarWinds that allows you to create more complex trigger conditions. This feature allows us to:

• Trigger when at least more than one device/object is triggering the alert: for example, trigger the alert when there are at least two devices from a cluster down
• Additional sections: allowing you to mix conditions from different types of objects. For example, alert me when the VM latency is high but the latency of the VM datastore is not high.

I personally use this feature when I use alerts to trigger automation processes for self-healing purposes, as this feature allows you to be very (very) specific on when to trigger the alert.

However, today’s topic is not how to use SolarWinds alerts to automatically self-heal your network, we will leave this discussion for another day.

Custom SQL/SWQL Definition Alerts
When all the options above are not helping you to create the alert you need, then you should try using SQL/SWQL alerts.

This is the most flexible option mentioned on this blog, but also the most complicated, as it requires SQL/SWQL query skills and knowledge of the SolarWinds database schema. However, this way to create the alert will allow you to define (almost) any type of alert.

This type of alert is commonly used when you need to achieve the following:

• Alerts involving historical data: common conditions only alert using the last value polled (unless you use the option of a wait before trigger), but what if you need to trigger an alert based on the average value for the last xx hours?

• Complex filters: when an alert becomes extremely long or complex using any of the other methods, using SQL/SWQL will potentially make it easy to configure. You are in control of the logic

To create a searchable query for this we’ll want to start by adding a Custom Query widget to our Home Summary page (or any other page should you prefer).

1. On the left, mouse over the pencil and then click Customize Page
2. Click Add Widgets
3. Search for Custom Query and drag the custom query into the view

Don’t you just love the new way of building web pages in Orion?? As with all widgets and layout design for Orion pages place it somewhere that is relevant for the importance i.e. at the top if you are going to use regularly or on a sub tab or lower down the page if not. You should now see a new widget on the screen, go ahead and click edit.

Give the Custom Query a name, something like ‘Search all IP addresses on nodes’.

Copy and paste the following into the Custom SWQL Query block:
[code]SELECT n.Caption AS [Node Name], i.Caption AS [Interface Name], ip.IPAddress AS [IP Address]
FROM Orion.Nodes n
INNER JOIN Orion.NPM.Interfaces i ON n.NodeID = i.NodeID
INNER JOIN Orion.NodeIPAddresses ip ON i.InterfaceIndex = ip.InterfaceIndex AND n.NodeID = ip.NodeID
WHERE ip.IPAddress IN (SELECT ip.IPAddress FROM Orion.NPM.Interfaces i INNER JOIN Orion.NodeIPAddresses ON ip.NodeID AND ip.Interfaceindex = i.interfaceindex)[/code]

The great and I mean GREAT thing about this Custom Query resource, is the ability to include a search function, which we can use to map into filter clauses in the SWQL. To enable search for the query tick ‘Enable Search’ and copy and paste the following into the text block. Note the only change is in the last 3 lines, which will take the value from the text field to use in the filter:
[code]SELECT n.Caption AS [Node Name], i.Caption AS [Interface Name], ip.IPAddress AS [IP Address]
FROM Orion.Nodes n
INNER JOIN Orion.NPM.Interfaces i ON n.NodeID = i.NodeID
INNER JOIN Orion.NodeIPAddresses ip ON i.InterfaceIndex = ip.InterfaceIndex AND n.NodeID = ip.NodeID
WHERE ip.IPAddress LIKE ‘%${SEARCH_STRING}%’ OR N.CAPTION LIKE ‘%${SEARCH_STRING}%’ OR i.Caption LIKE ‘%${SEARCH_STRING}%'[/code]

Click Submit. You should now have a searchable widget that will allow you to search for Node Name, Interface Name or IP Address!

SolarWinds® Orion is a magnificent tool for troubleshooting performance issues in your network, however when it comes to analysing future capacity or trends, even though it has some good insights, it doesn’t give you the fully comprehensive view that we would like to get, at least the way it is configured out of the box.

Nonetheless, SolarWinds stores historical data for each of the metrics that we monitor for analysis and forecasting purposes, as well as the option to use SQL or SWQ queries directly against the database. Hence it shouldn’t be too difficult to create trends and analyse data if we have a little bit of knowledge in SQL (or SWQL).

I’m not saying that SolarWinds Orion doesn’t have any way to get capacity forecast data if you don’t create custom dashboards using SQL queries. Actually, there are some nice resources available out of the box that will give you valuable capacity forecast data and tell you when your devices will run out of CPU capacity, memory, disk or interface bandwidth.

Let’s get the data

To extract the data from the SolarWinds database, the first thing we need to do is to find the tables that contain the data we want. SolarWinds database is quite intuitive in this matter, and the name of the tables from SQL or SWQL are clear enough to help us to locate the data we need. However, due to the quantity and inherent complexity of the database, there are some content structures which are more difficult to find.

As an example, the following table contains some of the most common metrics that we monitor from our network devices and the SWQL tables where you can find this data:

Let’s test this with a couple of examples that have had really good feedback when we have configured them for our customers as part of our professional services. The first example displays a chart with the CPU time consumed per hour for the last 30 days (or another time period); and the second example displays the amount of time WAN interfaces are over a specific threshold (in the example, over 75% utilization).

CPU Time
Question: over the last few months, are the devices on your network consuming more CPU or less CPU time than their normal baseline behaviour?

I guess that your first thought would be they use more CPU, because we always think that this is the case, but is it true? How can we answer this question?

Actually, if I tell you that your servers are consuming 1000 min of CPU time per hour currently, you wouldn’t be able to answer me if this is a high or a small number, because, as with lots of other types of metrics, you need something to compare these values with, a baseline to tell you what is normal and what is not. Therefore the real question here is how we can create a baseline and calculate the trend of CPU time used in our network.

The following resource sums the CPU time consumed each hour by all the devices monitored in SolarWinds, and displays it in a chart. As we are displaying hourly data, the time frame has to be equal or lower than the hourly retention period (by default 30 days, but we recommend to extend it to 62 days, two full months).

On this example we see that the trend is going down, therefore we are using less CPU time than before, which makes sense, as this chart was generated during Christmas time, where there are more people on holidays and the servers are not as busy as usual.

We can also create some variants of these resource displaying last 12 months (we should group data daily instead of hourly) or display CPU time from critical devices only (using custom properties or name conventions).

If you want to configure this resource in your SolarWinds platform, please follow these steps:

• Go to a summary view
• Add new widget custom chart
• Change the Datasource at the top to Advanced Database Query, and then set the query type to SWQL
• Add the following SWQL Query to the white text field

[code]Select DATETRUNC('hour',datetime) as [Date],
round(sum((avgload * weight)/6000),2) as [CPU Time],
'in Minutes' as [Minutes]
from orion.CPULoad
where datetime >= ${FromTime} and datetime < ${ToTime}
and weight <= 3600
group by  DATETRUNC('hour',datetime)[/code]

• Time period: choose anything equal to or lower than the hourly retention period, for example, Last 30 days
• Data Series: CPU Time
  • More -> Time Column -> Date
  • Calculated series: Show Trend
• Units Displayed:
  • Custom-> min
• Group chart data by: Minutes
• The legend shows: Minutes
• Sample Interval: Every hour

WAN Interface Time Over 75%
Question: are your WAN connections saturated?You might think that, as you have some alerts configured in SolarWinds that emails you when a WAN interface is saturated, that is enough. Further questions exist; are you really going to upgrade a circuit because, once, you received an email saying that the bandwidth utilisation of that link was high? Should you wait until this happens twice? Three-time? In a week? A month? The big issue here is alert analysis is only part of the trend analysis you need to be doing and that does not allow you to answer questions on the utilisation of a key metric such as this WAN connection example.

The following chart shows you the amount of time per day that WAN interfaces are over a specific threshold (75% in this example). In this case there is not a specific limit in terms of time frame for the chart, however the longer the retention periods are, the more accurate date this chart will display.

NOTE: in order to display WAN interface data, first of all, we need to find a way to identify the interfaces are connected to the WAN network and which ones are not. There are several ways to accomplish this in SolarWinds, however, we would always recommend the use of custom properties. For this chart, we are using a custom property called WANInterface, type Boolean, that defines normal interfaces (WANInterface = false) from WAN interfaces (WANInterface = true).

Based on this chart, it is fair to say that the last few days we had a little bit more of traffic (buying Christmas presents during working hours?), except for the last few days, the trend does not show any increase on WAN traffic. Therefore in this situation, I wouldn’t recommend upgrading your WAN links and I would want to correlate this to further event and Netflow analysis to understand what was generating more traffic during these 4 days.

If you want to configure this resource in your SolarWinds platform, please follow these steps:

• Go to a summary view
• Add new widget custom chart
• Change the Datasource at the top to Advanced Database Query, and then set the query type to SWQL
• Add the following SWQL Query to the white text field

[code]Select sum(it.weight)/60 as [Time],
datetrunc(‘day’,it.datetime) as [Date],
‘Time’ as label
from orion.npm.InterfaceTraffic it
where (it.inpercentutil >75 or it.outpercentutil > 75)
and it.datetime >= ${FromTime} and it.datetime < ${ToTime}
and it.interface.customproperties.WANInterface = 1
group by datetrunc(‘day’,it.datetime)[/code]

• Time period: choose anything equal to or lower than the hourly retention period, for example, Last 60 days
• Data Series: Time
  • More -> Time Column -> Date
  • Calculated series: Show Trend
• Units Displayed:
  • Custom-> min
• Line: Stacked Column
• Group chart data by: Label
• Legend shows: Label
• Sample Interval: Every day

Other Trends
The charts we have created for CPU or interface traffic can be replicated for any of the metrics available in SolarWinds such as disk utilization, firewall connections, VPN connections, availability, etc. Just think about which metric you would like to forecast. I hope this article has been informative for you and thank you for your time reading it.

The following week, at another customer, the IT director’s boss came down to the networks department asking when the network is less busy. This company was thinking about doing some stress tests on the network, and he was asking when the best time of the week to perform these tests is. This created a bit of a discussion as some guys immediately replied that the weekend is when the network is less busy, but some other mentioned all those backup jobs configured on Saturday and Sunday. In the end, there was no consensus and the IT director left a little bit frustrated with the situation.

These are just two of the multiple scenarios where IT engineers need to know what is ‘normal’ in the network infrastructure and what is not. In order to find anomalies in your infrastructure, it is essential to have an understanding of what the baselines are.

Baseline your network

The question now is how we can baseline our network traffic, as collecting data is the essential ingredient here. If we have SolarWinds (for this example we will use SolarWinds Network Performance Monitor, however depending on the metric you want to baseline other modules will be required. The tool will store in the database every time we poll data from our network during the retention period. In plain English, this means that SolarWinds will keep historical data of the metrics we get from the devices we are monitoring.

In this example, if we need to baseline interface traffic, I would recommend getting the average of traffic per hour and per weekday. This makes sense to me as, normally, IT engineers create admin tasks with daily or weekly frequency. For example, we take VM backups daily at 23:00 or running configs of our network devices on Sunday at 02:00. If you guys create tasks with a different frequency (say monthly), I would recommend baselining your network based on that frequency (for example based on the day of the month).

Steps to display your average weekday traffic

Having the option to see what is normal and what is not in my network sounds great to me, but the question remains the same: how can we display the data we want on the format we need?

SolarWinds Orion gives us a lot of options to display the metrics we are polling from our network devices, and in situations where we need something a little bit more customized (like the current one) we can use SWQL/SQL queries to display the information the way we need.

If you want to display the average traffic per hour and per weekday on a SolarWinds view, please follow these steps:

NOTE: if you want to create a report instead of a SolarWinds view, the steps are quite similar

• Go to the dashboards you want to edit in SolarWinds
• Customize page

• Add new widget
• Create a custom chart
• Datasource -> Advanced Database Query -> SWQL
• Enter the following query:

• Time Period: Custom
  • Named time period: Average
  • Relative time period -> Last 1 DaysData Series: Total Bytes
• Data Series: Total Bytes
  • More -> Time Column -> Date
• Units displayed: Bytes

• Group chart data by: DayWeek
• Legend shows: DayWeek
• Sample Interval: Every hour

If you have followed the steps above, the chart should look like something similar to this one:

Based on the chart above, we can see that every day we have a spike of traffic around 21:00, and a similar amount of traffic on Sundays and Mondays around 01:00. This makes sense as every day at 21:00 is when we back up our VMs to our internal backup server and on Sundays we backup the VMs to the cloud. However, it is not clear why we have the same spike on Mondays, probably a misconfiguration that I will need to review internally.

What if we want to profile the traffic of one single interface? That is also possible using the following script, with the same steps described before, just add the widget on the Interface Details View (view displayed when you click on any monitored interface).

Bear in mind that now we are displaying bits per second, and not total bytes, therefore in ‘Units displayed’ select Bit/s.

Other Baselines

To be fair, any metric that is monitored in SolarWinds can be baselined as shown above. Think about what is important in your network and create the chart that will display the data you need. Metrics like the following are good candidates to baseline:

• CPU load
• Volume space used
• Availability
• Virtual Machine latency
• Volume IOPs

Thank you for your time reading this article, and please let us know which metrics you would like to baseline in your network.

This is not a history lesson, but an introduction to the fact that since then SolarWinds has created the Orion platform and turned this into a monitoring and management behemoth. A platform that is modular in design and has grown and grown, moving away from purely a network focused solution to a solution that monitors everything that has any sort of service connectivity in an organisation. Add in there a mix of management functions as well and it is no wonder why SolarWinds has over 275,000 customers around the world.

Orion which now contains a total of 12 full modules and 3 integrations to other SolarWinds applications, with the 12th being the latest new addition to the fold: Server Configuration Monitor.

This is a ground-up Orion module, which has been built to provide a method to identify and create a record of changes applied to servers. This is important because human errors and changes to systems are consistently in the top 3 reasons for loss of services within IT. Keeping on top of changes, knowing when things change, what has changed and what it changed from.

In a time-honoured traditional the module is itself modular in the sense that it has out of the box functionality and the ability to be extended via the built-in customisation. Referred to as profiles, in the first release of SCM, there are 7 provided profiles; folder content, Windows firewall, IIS, Orion SDK, Registry, Hardware, and Software. Certainly a fundamental for system changes is the adjustment to hardware and software, but with IIS, this is looking inside an application configuration for changes.

Creating your own profile allows you to define your own system or application change detectors, with capabilities to monitor file and registry changes. All it takes is identifying the files and registry settings that have an impact on the configuration of a system or application and very quickly you can build a set of entries to monitor them.

The baseline feature built into SCM allows not just for changes since the data was last collected to be performed but allowing a defined point in time to be used as the comparison point. This may be something straightforward as creating a known secure point in configuration or that your organisation is starting a change freeze period and therefore any changes identified from this point forward can be quickly identified, reported and understood.

The above screenshot clearly shows a server with 2 configuration differences to the baseline defined on it. Let’s go and look at what has changed here. The information is available within the familiar Node Details page for the node device as a sub-tab menu page.

Drilling into the change detected reveals the true detail of the change, so in this example, we can see that the server had more memory added to it.

One last feature to show you and I highly recommend you look at a previous video of ours where we show off the marvellousness of the PerfStack area of Orion. Here we are using Perfstack to populate Orion NPM metric data with SCM identified configuration changes. Notice the blue towers in the top chart, these are the individual changes, which can be highlighted to give the details on the left sidebar, how cool is that?!

Continuously monitoring for changes and being able to correlate this with other Orion monitoring data provides such insight into root causes of issues affecting the health, performance, and security of your servers and the services they deliver to your business.