Important Announcement: Applying SolarWinds Orion Remediation To Vulnerability

Following the identification of security vulnerabilities Sunburst and SuperNova, SolarWinds® have released an update patch in order to address these vulnerabilities. If you are on one of the following three SolarWinds Orion versions, you are recommended to immediately upgrade the platform to a patched version appropriate for your installed version. The following table provides the guidance

Due to the nature of the Sunburst attack, where the code was hidden within genuine application files and function, it is difficult to determine if your environment has been truly compromised. This may be determined through an intensive review of your SIEM tooling for events that could indicate that unexpected activity has occurred. Such a review should be focused on the use of privileged accounts, unaccounted change, and access to the known central command domains for signs of misuse.

Identification of the SuperNova malware is easier, as this requires a file to be placed by a 3rd party actor into your genuine Orion file structure. If this exists, again cleansing and forensics reviews of compromise are advised.

SolarWinds have updated their Incident Advisory page and also added an FAQ page to assist with any questions you may have.

Decisions to Make

To bring your platform to a level where the vulnerability is removed, based on your installed version being one of those affected, this will require an upgrade activity. There are two principle avenues to take for performing the upgrade:

• • In place upgrade
    • Apply the HotFix only – suitable if you are on 2020.2.1 or 2019.4 HF5
    • Follow the standard upgrade procedures for a full upgrade
  • Upgrade Migration
    • Provision new Orion application servers and install the upgrade on these
  • Rebuild
    • Start with fresh application servers and blank database

The migration option will mitigate any potential risk of compromise, by reinstalling Orion on fresh servers, which are inherently clean. The process for this option is not much different to the upgrade, other than the provision of new servers.

We feel that a manual review of the Orion SQL databases can be performed to mitigate risk, so this option can be avoided.

• Review any database Schema changes made (Database monitoring tools can provide this)
• Manual review of areas of the DB that could be used to re-execute malicious code;
  • Alerting Engine, Stored Procedures, SAM script based monitors, NCM scripts

Upgrade Advice

If you have previously taken our and SolarWinds advice and disabled Internet access to your Orion server, to perform the upgrades there are two options: centralised upgrade via the web interface of Orion; or the offline installer.

Clearly for the first option to be viable, it is necessary to allow Internet access for which the following URL’s need to be allowed as white list entries:

https://downloads.solarwinds.com
https://api.solarwinds.com 
https://installer.solarwinds.com
https://licenseserver.solarwinds.com
https://licensestatusserver.solarwinds.com

The centralised upgrade via the Orion web UI is the quickest and easiest method to perform a full upgrade, as Orion will effectively self-manage the upgrade and automatically upgrade any scalability engine servers you may have i.e. Additional Polling Engines and Additional Web Servers.

If your installation is fully isolated the offline installer can be downloaded. This is a single file which can be used on all of the role servers. It is necessary to upgrade the Primary Orion server first, then each of the additional role servers, on which the installation can be executed concurrently.

Overview of actions

• Prepare the environment
  • Provision new servers if selecting the Upgrade Migration approach
  • Update security devices to allow the above URL’s if you wish to perform a Centralised upgrade via the Orion web UI
  • Perform a backup of the Orion SQL databases
    • Orion Core (default name SolarWindsOrion)
    • Orion Logs (default name SolarWindsOrionLog)
    • Orion NetFlow (if you own this licensed module, default name SolarWindsFlowStorage)
  • Perform a snapshot or machine image backup of the application server/s
• Perform the pre-flight checks in the Orion web UI
  • Log in with Admin account
  • Navigate to Settings > My Orion Deployment > Updates and Evaluations
  • Resolve any actions that are listed here before you begin the actual upgrade
• Within this same screen, it is also advised to pre-stage the files, so the upgrade when the request is made will apply immediately
  • This is a step that requires the controlled Internet access
• For the Offline installation it is highly recommended to perform this by logging in to the Orion servers using a Local Administrator account, and not a domain based account
• If you are performing a migration, then it is advised to use temporary IP address (assuming your live platform is still active) to build the new servers and then change the IP to the production IP’s when the current server is shutdown
  • If you are keeping new IP addresses, then your security policies for firewalling/ACL rules will need to be updated to reflect the new polling sources. Also Netflow, Syslog and SNMP Trap destination configurations will need to be updated to point to the new IP addresses

If performing the offline installation method, the files can be downloaded from the SolarWinds Customer Portal for which two options exist:

• • If you are running 2020.2.1, then you can apply the smaller HotFix 2 patch
  • If you are running 2019.4 HF5 or 2020.2 then the full version upgrade can be applied

Rather than performing an upgrade, if you are running 2019.2 or below, the specific Security Patches can be applied for your version.

Additional Resources

Security Advisory from SolarWinds
SolarWinds FAQ
Prosperon Blog Post On Incident
FireEye Blog Post
Orion Upgrade Guide
Orion Migration Guide
Upgrade made easy (YouTube)

If you have any further questions, need any specific guidance or have any issues, please contact our support on support@prosperon.co.uk or 01903 340993. If you are raising a case with SolarWinds directly and need assistance with this, please provide us with the SolarWinds Case reference.