Recent Improvements to SQL Sentry

In the fast-paced world of database management, staying on top of performance monitoring and optimisation is crucial. Database administrators and developers are continually seeking ways to improve efficiency, streamline operations, and ensure the optimal functioning of their SQL Server environments. DBA’s are, quite rightly, very precious about their platforms, and inefficient use of these leads to poor service performance and higher financial cost impacts.

One indispensable tool in this pursuit is SQL Sentry, a comprehensive monitoring solution from SolarWinds® that has recently undergone significant improvements to further empower users. In this blog post, we’ll delve into the recent enhancements in SQL Sentry 2023.3 that will greatly improve the way you manage and optimise SQL Server databases.

Viewing Overall Health

One of the notable enhancements in recent versions of SQL Sentry is the introduction of a new view that provides a holistic snapshot of the overall health of your monitored targets. This feature is a game-changer for database administrators, offering a consolidated dashboard that displays critical information about the state of your SQL Server instances and databases.

In the Health Overview dashboard we have an easy-to-digest status box displaying the current health of your environment over the last 24 hours. Of that time period, three figures are given: the highest recorded health score; the lowest recorded health score; and the total number of monitored targets within the selection. Next to that overview, a bar graph gives us the overall health score of your environment across the last 30 days, allowing you to quickly compare your current health score with more historical data.

We’re all very happy to see the burden of judging health alleviated with this feature. No more hesitant responses to stakeholders with caveats and overly technical language – we can now easily present the overview and say with confidence whether an object is healthy, at risk, unhealthy, or critical.

1.2 Top SQL Wait Statistics Charts

Another significant improvement in SQL Sentry is the ability to gain a unified perspective on collected SQL statements. SQL queries and statements lie at the heart of database performance, and having a cohesive analysis tool is invaluable. With the top wait charts enhancement, SQL Sentry offers a more efficient way to analyse SQL statements across different databases and SQL Server instances that are causing the most waits, consuming the most resources, taking the most time, and putting the most load on your SQL Server.

Starting with Version 2023.3, wait information can be divided up and presented by the following categories:

• Queries
• Waits by Application (the source of the query being processed)
• Waits by Database (which database on the server is being used)
• Waits by Host (the source machine of the query being processed)
• Waits by Login (which user login is processing the query)

We find the waits by login extremely valuable. While identifying waits by other categories gives great insight, logins will often immediately give us the answer as to where issues lie. How often have we administrators been able to pinpoint a problem the moment a user or service account is named? As you can see below, hovering over the chart tells us immediately the logins behind the bar chart segments.

Of course, spotting the user is only part one. What exactly is that login querying? What are the queries, and how is that wait distributed? Here is where the queries filter really gets to shine.

At the bottom of Top SQL page, we have the Totals table. This displays the text data and associated information such as database, duration, count, and CPU for the collected SQL statements (including procedure stats, query stats, and completed queries).

1.3 Conclusion

What You Will Learn

What you will learn on this webinar:

• Optimise Database Health & Configure Performance: Discover how to efficiently optimise database health and configure performance using SolarWinds SQL Sentry.
• Administrate Databases Using Advanced Automation: Learn how to administrate databases and automate essential tasks using automation and a library of scripts.
• Minimise Downtime & Resolve Database Issues Faster: Explore how SolarWinds can help mitigate database performance issues and reduce costly downtime.
• Live Product Demonstrations: SolarWinds Engineers will demonstrate the core functionality of SolarWinds SQL Sentry.

What you will learn on this webinar:

• Understand What Database KPIs & Metrics Are Important: Discover what data is needed to identify underlying database issues affecting application performance.
• Meet Database Monitoring & Administration Requirements: Learn how to identify database problems and how to troubleshoot advanced database performance issues.
• Reduce Data Loss & Resolve Issues Faster With SolarWinds: Explore how SolarWinds Database Performance Analyser can help meet your Junior DBA needs.
• Live Product Demonstrations: SolarWinds Engineers will demonstrate the core functionality of SolarWinds Database Performance Analyzer.

What you will learn on this webinar:

• Understand What Database KPIs & Metrics Are Important: Discover what data is needed to identify underlying database issues affecting application performance.
• Meet Database Monitoring & Management Requirements: Learn how to identify database anomalies, troubleshoot database health, and respond to issues.
• Resolve Database Performance Issues With SolarWinds: Explore how SolarWinds Server & Application Monitor and Database Performance Analyser can help you.
• Live Product Demonstrations: SolarWinds Engineers will demonstrate the core functionality of SolarWinds Server & Application Monitor and SolarWinds Database Performance Analyzer.

SolarWinds® Orion® has a cool feature that allows us to create an interactive searchable table, which we are using here as we need one place to go to see all software installed across multiple machines that are within SolarWinds. This can be completed by using the Custom Query widget and creating a SWQL script to pull the exact data we want from different tables and presenting it in a way that is pleasing to the eye and clear to understand.

Requirements: To be able to successfully complete this guide, below is required to be able to replicate this view in your own environment.

• SolarWinds Server & Application Monitor (SAM) installed
• Servers monitored with the ‘Asset Inventory’ under list resources selected

• Account with Customise View rights for creating the Orion widget
• Optional: SWQL Studio installed – https://github.com/solarwinds/OrionSDK/releases
  • Orion Administrator level account

Process:
We are using SWQL Studio here as it allows us to build the query efficiently, view the data we require, so we can take this in to the web UI widget configuration. The data can be pulled from multiple tables within the database using an easy to understand query language based on SQL, called SWQL (SolarWinds Query Language).

Open SWQL Studio and connect to the SolarWinds Application Server with your account that has admin level permissions.

Find the table down the left hand side that has most if not all the data you require to present your view. The table we will be using to complete this task is ‘Orion.AssetInventory.Software’. Right-click and click ‘Generate Select Statement’, which will immediately give us a query that shows all available columns in this Orion SWQL Entity and is a great starting place to build our final query.

On the top bar hit the play button this will show you the top 1000 records in this table and what detail is under which column

Now we need to decide what columns of data on this view we want to show and what we are missing. The only column here which isn’t useful is the NodeID and what we really need is the Name of the Node so it is clear for the person seeing this table which node it is referring to.

We are going to go with a best practice approach to SWQL/SQL queries, by utilising the ‘alias’ syntax, which also means we don’t have to type Orion.AssetInventory.Software everytime. In the query after Orion.AssetInventory.Software give it an alias using a letter/s which here I am keeping it simple and use ‘S’. I have also reorganised the columns to show in the order I prefer and removed the ‘NodeID’ column.

(Tip: make sure there is a comma after every column except the last column otherwise you will receive an error).

Another benefit of using aliases is that to add another column, start typing s. after a comma and SWQL Studio will display columns not listed in the original select statement are displayed for automatic completion. This also includes linked SWQL entities  (related tables) which we can use to present the data we want to see in a clean efficient manner.

Within the Orion.AssetInventory.Polling entity there is a column named Caption so we can use this by typing S.Polling.Node.Caption then hit execute query.

We should also add the abbreviation for the entity to all columns used and I have also reordered putting the caption as the first column.

Lets Move to the Orion Web Interface

We could end there by copying the text into a custom query on the web console and show it in its raw form and it would look ok. But we really want to improve the look and feel of the widget to fit in with the rest of SolarWinds Web Console.

We can make it more interactive by showing if the node is up and being able to go to that node from the query.

Firstly we need to name the column we want to make it easier to reference, to do this after the column we add “AS [Node Name]”. We need to use square brackets when using a word that is used as a column header or if we want to add spaces into the column header like [Node Name] or [Server ID] for example, I have also done this for the INSTALL DATE column.

[code]’/Orion/images/StatusIcons /small-‘ + ToString(s.Polling.Node.StatusIcon) AS [_IconFor_Node Name][/code]

[code]’/Orion/images/StatusIcons /small-‘[/code] Is the URL location on the server of the images folder. This will be found in the Inetpub folder where your website is stored, with the + used to join two parts together. The second section after the +

[code]ToString (s.Polling.Node.StatusIcon)[/code] Is a dynamic value taking the output of s.Polling.Node.StatusIcon and forcing it to be text so it can be used to complete the URL string.

The AS is naming the column but [_IconFor_Node Name] is advising the custom query to use this as a web link for a specific column as before the square brackets is due to the column name having a space within it. This will show as a column within SWQL Studio with the correct URL in it the fields but when copied to the web console the column is used as a reference and not shown in the table.

[code]SELECT
s.Polling.Node.Caption as [Node Name],
‘/Orion/images/StatusIcons/small-‘ + ToString(s.Polling.Node.StatusIcon) AS [_IconFor_Node Name],
s.Polling.Node.DetailsUrl AS [_LinkFor_Node Name],
s.Name,
s.InstallDate as [Install Date],
s.Publisher,
s.Version
FROM Orion.AssetInventory.Software s[/code]

Notice how this not only provides a link to the Node detail page, but also includes the hover over function, which means we may not even need to click into the detail page to get what information we needed.

We can also enable the search function of this widget to give the ability of filtering and decide what columns to search through by adding the where clause on the end. This is simply a case of using the variable for the text input ${SEARCH_STRING} and including in the WHERE clause:

[code]SELECT
s.Polling.Node.Caption as [Node Name],
‘/Orion/images/StatusIcons/small-‘ + ToString(s.Polling.Node.StatusIcon) AS [_IconFor_Node Name],
s.Polling.Node.DetailsUrl AS [_LinkFor_Node Name],
s.Name,
s.InstallDate as [Install Date],
s.Publisher,
s.Version
FROM Orion.AssetInventory.Software s
WHERE s.Polling.Node.Caption LIKE ‘%${SEARCH_STRING}%’ OR s.Name LIKE ‘%${SEARCH_STRING}%'[/code]

In this webinar, you will discover how to Monitor Application Health & Performance across the Application Stack with SolarWinds®. This webinar examines how to gain visibility across your application environment, from servers, virtual machines, storage and database layers. Discover how to troubleshoot and speed up resolution times by correlating application performance metrics from multiple sources, and how to accelerate application performance by removing infrastructure bottlenecks.

What you will learn on this webinar:

• Gain Deep Visibility Across All Application Layers: Learn how to visualise the relationship between applications and underlying IT infrastructure layers with intuitive SolarWinds application dependency maps that deliver end-to-end visibility.
• Correlate Cross-Stack Application Performance Metrics: Discover how to accelerate application resolution time by comparing network, systems, virtualisation, storage and database data side-by-side on a common timeline.
• Quickly Eliminate Infrastructure Related Application Issues: Explore the relationship between underlining infrastructure layers and application health, and how to optimise and fine tune infrastructure layers to accelerate application performance.
• Live Product Demonstrations & Q&A Session: Our SolarWinds Engineers will demonstrate the core functionality of the SolarWinds System Management portfolio.

We have all had an application or service that runs away with itself and consumes all of the compute resource on a server, where you know that a simple restart of the service or killing of the process will restore operation. For some, this is the pre-cursor IT fix before rebooting the machine, as that invariably gets the right result. If you have a slow running machine or application, and you happened to be on the desktop of that machine, you are going to go look at Task Manager to see what is consuming all of the compute resource.

If you are using a monitoring solution, such as SolarWinds Server and Application Monitor (SAM), you are no doubt monitoring those key servers. Now the big question comes where you know for absolute certainty (notice how this is in bold?!) that if your server consumes continuously more than say 95%, that stopping the process with the highest resource usage will resolve the issue. This is a big if and this post should if you have not already picked up, will give you a method within Orion itself to automate that operation, but you must be at that 100% to apply the capability outlined in this post.

If you are at this place the ability for Orion to identify the issue and fix the issue and all you need to see is the alert notification to confirm it did this, you would be kinda happy right?

OK, I have given enough of a warning to configure this function in your Orion, so lets get to it.

First of all you need to be monitoring your servers within Orion SAM, which will provide CPU and memory utilisation for the whole system.

Now let’s look at these actions in detail:

Action 1: Execute program : SolarWinds.APM.RealTimeProcessPoller.exe

This action in type of “Execute external program” and it executes SolarWinds® out-of-the-box tool; “SolarWinds.APM.RealTimeProcessPoller.exe” in order to poll in realtime for the top 5 processes sorted by “Physical Memory” and insert this information to “Alert Notes” area for the triggered alert object. This is the actual command to be run by the action:

[code]APM\SolarWinds.APM.RealTimeProcessPoller.exe -n=${NodeID} -alert=${N=Alerting;M=AlertDefID} -count=5 -sort=PhysicalMemory -timeout=120[/code]

The available parameters that can be provided for this Solarwinds tool and details can be found in related SolarWinds article.

Action 2: Execute program : Send an Email

This e-mail notification action will be executed 2 minutes after the first action in order to give enough time to the first action to pull top processes and update Alert Notes.

Email Subject: Memory load on ${Caption} is currently ${PercentMemoryUsed}

Email Message:

The Memory on ${Caption} is currently running at ${PercentMemoryUsed}. The top 5 processes running at the time of this poll are listed below:

[code]${N=Alerting;M=Notes}<br /> <code>For more information click the link below.</code><br /> <code>${NodeDetailsURL}[/code]

These parameters will allow us to send the Email notification to related IT contact to inform high memory usage issue on a server with the list of the top 5 processes consuming the most memory. You may wish to stop here, with this information in the external notification and visible in the Alert Details page in Orion, this may be sufficient for you to manually take action.

If you want to automate, then buckle up..

Action 3: Execute program : KillWindowsProcess.ps1

This action in type of “Execute external program” and it executes custom PowerShell script to get the top process information from alert notes and kill that process on the server that triggered the alert. This is the actual command to be run by the action:

[code]C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe "C:\Webinar_Scripts\KillWindowsProcess.ps1" "’${N=SwisEntity;M=Caption}’" "’${N=Alerting;M=AlertObjectID}’"[/code]

As shown in this command, we defined custom script “KillWindowsProcess.ps1” in folder “C:\Webinar_Scripts” of Primary SolarWinds server. This script gets two parameters; first parameter is server name and the second parameter is alert object ID.

Action should be configured to be executed with a domain account that can connect and kill process on destination server:

“KillWindowsProcess.ps1” script connects to SolarWinds API to pull top process information from alert note and then checks the found process if it is in protected process list. It is recommended to define a protected process list to prevent killing critical processes. We do NOT want to render a server inoperable!. If the process is not in the protected list, then the script tries to kill that process on the  destination server. The script generates a log file in “C:\Webinar_Scripts\logs” folder for each day, which can be reviewed to identify and fault diagnose the actions performed. Here is the content for the script:

[code]#Logging Details
$WORKING_FOLDER = "C:\Webinar_Scripts\"
$LOGFILE = $WORKING_FOLDER + "logs\action_trigger_log_" + $(get-date -f yyyy-MM-dd) + ".txt"

# List of protected services that should not be terminated
$PROTECTED_PROCESSES = @(‘sqlserver.exe’,’WmiPrvSE.exe’, ‘svchost.exe’)

# Variables
$targetProcessName = ""
$targetHostName = $args[0]
$AlertObjectID = $args[1]

$time = get-date
$message = "$Time – Started action for Hostname: $targetHostName and Alert Object ID: $AlertObjectID"
$message >> $LOGFILE

# SolarWinds Connection Properties
$hostname = "SOLARWINDS_HOSTNAME"
$username = "SOLARWINDS_API_USER"
$password = "SOLARWINDS_API_PASSWORD"

$swis= Connect-Swis -Hostname $hostname -username $username -password $password

# Get Alert Note for the provided Alert Object
$alertQuery = "SELECT TOP 1 AlertNote FROM Orion.AlertObjects WHERE AlertObjectID=" + $AlertObjectID

$alertData = Get-SwisData $swis $alertQuery

$noteLine = $alertData.Split([Environment]::NewLine,[System.StringSplitOptions]::RemoveEmptyEntries)

$noteLineCount = $noteLine.Count

# Parse Alert Note to get Top Process Information
if ($noteLineCount -gt 1)
{
$headRow = $noteLine[0]
$topRow = $noteLine[1]

$headItems = $headRow.Split("`t",[System.StringSplitOptions]::RemoveEmptyEntries)

$headerItem = $headItems[0]

if($headerItem -eq ‘Name’)
{
$topProcItems = $topRow.Split("`t",[System.StringSplitOptions]::RemoveEmptyEntries)
$targetProcessName = $topProcItems[0]

}

}

if (($targetProcessName -ne "") -and ($targetProcessName -ne $null))
{
$time = get-date
$message = "$Time – Identified top process is : $targetProcessName for Alert Object ID: $AlertObjectID"
$message >> $LOGFILE

$IsProcessProtected = 0

forEach($p in $PROTECTED_PROCESSES)
{
if($targetProcessName -eq $p)
{
$IsProcessProtected = 1
}
}
if($IsProcessProtected -eq 0)
{

$Processes = Get-WmiObject -Class Win32_Process -ComputerName $targetHostName -Filter "name=’$targetProcessName’"

$foundProcessCount = $Processes.Count
$foundProcessName = $Processes.Name

if (($foundProcessCount -gt 0) -or ($foundProcessName -eq $targetProcessName))
{
foreach ($process in $Processes)
{
$returnval = $process.terminate()
$processid = $process.handle

if($returnval.returnvalue -eq 0)
{
#write-host "The process $targetProcessName `($processid`) terminated successfully"
$time = get-date
$message = "$Time – The process $targetProcessName `($processid`) terminated successfully"
$message >> $LOGFILE
}
else
{
#write-host "The process $targetProcessName `($processid`) termination has some problems"
$time = get-date
$message = "$Time – The process $targetProcessName `($processid`) termination has some problems"
$message >> $LOGFILE
}
}
}
else
{
$time = get-date
$message = "$Time – The process $targetProcessName can not be found on Hostname: $targetHostName "
$message >> $LOGFILE
}
}
else
{
$time = get-date
$message = "$Time – The process $targetProcessName is in the protected list, it will NOT be terminated"
$message >> $LOGFILE
}
}
else
{
#write-host "Unable to find target process for Alert Object ID: $AlertObjectID"
$time = get-date
$message = "$Time – Unable to find target process for Alert Object ID: $AlertObjectID"
$message >> $LOGFILE
}

$time = get-date
$message = "$Time – Completed action for Hostname: $targetHostName and Alert Object ID: $AlertObjectID"
$message >> $LOGFILE[/code]

This is an example of using the Alerting capabilities to take remediation action to resolve a problem, which I hope you find useful, but please use with care!

Capacity Forecast is one of the features available in the SolarWinds® Orion platform, that allows you to monitor the trends of some of the main KPIs in your network. However, SolarWinds Orion will not analyse the capacity forecast information for every single metric that is being gathered in the platform.

Available out of the box, only CPU and memory at the node level, interface traffic at the interface level, and disk space used at the volume level are compatible with the capacity forecast feature.

The Problem

There have been situations where I would have loved to get the capacity forecast information for other metrics, especially when analysing future capacity requirements for my applications, but as mentioned before, SolarWinds Orion does not give us that info for every single metric.

This might look like a problem, however, SolarWinds Orion stores the historical data for the whole retention period, i.e. the data is available for us, we just need to create the report to show the data in the format we want.

The Report

The following custom Orion report displays the capacity forecast analysis of all the performance counters monitors in SolarWinds Server & Application Monitor, showing the average value of that monitor per month for the last 6 months, the average increase/decrease per day, the warning and critical thresholds (if defined) and the amount days left until the metric reaches the threshold.

This report may be very useful for lots of customers, however, in big installations of SolarWinds SAM, there might be thousands of performance counters monitors, and some of those monitors maybe not as important as some others.

It would be nice if we could filter the performance counters displayed on this report in some way and show only the important ones. We can do this by creating SQL filters with the SQL query used to create this report.

There are three filters that we will use to include the monitors required. In order to edit the filters, we just need to edit the SQL query configured on the data selection of the report.

The three filters used here are and note how I have utilised SQL variables to separate the filter values, which I prefer just to make it easier to define, as it can be edited outside of the SELECT statement:

• @Component: filters by component name
• @Application: filters by application name
• @node: filters by node name

These three filters can be used at the same time if required. As an example, the following configuration would show all the components where the component name contains ‘job engine’ and application name contains Orion and node name contains ‘SW’.

In case you don’t want to use one or more of these filters, just leave the variable with no value (as default).

Real Case

The first time I thought about creating this report was when a SolarWinds customer asked me to create a capacity report that showed the trend of web users on their IIS servers. They wanted to know if their current web servers were currently saturated, and, if they were not, they wanted to know when they will require a new webserver to be added to the pool in order not to impact on the web performance. Typically, a line or bar chart would be used, as this gives a nice clear trend display, but where many lines would be shown, it would make it very difficult to read information out or would require multiple charts. Therefore, a different approach was necessary, hence this report.

In this scenario, we can use this report using the application name filter @application = ‘IIS Web Users’. The ‘IIS Web Users’ application is a custom template I’m using instead of the Appinsight for IIS template due to license restrictions (the NFR licenses are quite small). Otherwise, if you guys are using Appinsight for IIS, I would use the filter @application = ‘Microsoft IIS’ and @component = ‘Current’.

NOTE: the default retention period of SAM is 90 days, which means you will be restricted to this time period of analysis. If you wish to extend it to a period your report requires to cover, this setting can be edited on SAM Settings -> Data and Database Settings. It is important to note, this setting has a significant impact on Database size, as more data is being recorded and therefore please ensure your SQL Server can both store, but more importantly compute resource to allow it to process the higher data volume.

So, if you wish to use a report like this one, please feel free to download the report available below, and please let us know your feedback.

I hope this article has been informative for you and thank you for your time reading it.

This is not a history lesson, but an introduction to the fact that since then SolarWinds has created the Orion platform and turned this into a monitoring and management behemoth. A platform that is modular in design and has grown and grown, moving away from purely a network focused solution to a solution that monitors everything that has any sort of service connectivity in an organisation. Add in there a mix of management functions as well and it is no wonder why SolarWinds has over 275,000 customers around the world.

Orion which now contains a total of 12 full modules and 3 integrations to other SolarWinds applications, with the 12th being the latest new addition to the fold: Server Configuration Monitor.

This is a ground-up Orion module, which has been built to provide a method to identify and create a record of changes applied to servers. This is important because human errors and changes to systems are consistently in the top 3 reasons for loss of services within IT. Keeping on top of changes, knowing when things change, what has changed and what it changed from.

In a time-honoured traditional the module is itself modular in the sense that it has out of the box functionality and the ability to be extended via the built-in customisation. Referred to as profiles, in the first release of SCM, there are 7 provided profiles; folder content, Windows firewall, IIS, Orion SDK, Registry, Hardware, and Software. Certainly a fundamental for system changes is the adjustment to hardware and software, but with IIS, this is looking inside an application configuration for changes.

Creating your own profile allows you to define your own system or application change detectors, with capabilities to monitor file and registry changes. All it takes is identifying the files and registry settings that have an impact on the configuration of a system or application and very quickly you can build a set of entries to monitor them.

The baseline feature built into SCM allows not just for changes since the data was last collected to be performed but allowing a defined point in time to be used as the comparison point. This may be something straightforward as creating a known secure point in configuration or that your organisation is starting a change freeze period and therefore any changes identified from this point forward can be quickly identified, reported and understood.

The above screenshot clearly shows a server with 2 configuration differences to the baseline defined on it. Let’s go and look at what has changed here. The information is available within the familiar Node Details page for the node device as a sub-tab menu page.

Drilling into the change detected reveals the true detail of the change, so in this example, we can see that the server had more memory added to it.

One last feature to show you and I highly recommend you look at a previous video of ours where we show off the marvellousness of the PerfStack area of Orion. Here we are using Perfstack to populate Orion NPM metric data with SCM identified configuration changes. Notice the blue towers in the top chart, these are the individual changes, which can be highlighted to give the details on the left sidebar, how cool is that?!

Continuously monitoring for changes and being able to correlate this with other Orion monitoring data provides such insight into root causes of issues affecting the health, performance, and security of your servers and the services they deliver to your business.