NetX is aware of a zero-day vulnerability in a logging library (Log4j) which was bundled in versions 8.12 and later of NetX software. This exploit potentially allows for remote execution of code. See: https://logging.apache.org/log4j/2.x/security.html
Considering the ongoing nature of this exploit, NetX will post updates to this page to keep you informed on our remediation efforts.
Customers running versions of NetX prior to version 9.0 should contact their NetX account manager for further assistance.
If you have not already done so, please follow the remediation steps listed below.
Step 1: Back up all NetX Components and Data
Before performing any maintenance, NetX always recommends backing up your environments. But in this case, NetX also recommends that you take this opportunity to have your IT staff verify that all backup regimes are operating properly.
Step 2: Patch NetX
Download NetX version 9.16.11 and upgrade NetX
or:
Upgrade the logging library manually (only available with NetX version 9):
- Download the latest version of log4j from the Apache Foundation here: https://logging.apache.org/log4j/2.x/download.html. We recommend using the apache-log4j-2.17.1-bin.zip. Please make sure you follow the instructions to verify the validity of the downloaded archive before deploying it in your environments.
- The following is a list of the affected jars and their path locations — [netx root] is the location of your NetX installation):
[netx root]/webapps/ROOT/WEB-INF/lib/log4j-1.2-api-2.8.1.jar
[netx root]/webapps/ROOT/WEB-INF/lib/log4j-api-2.8.1.jar
[netx root]/webapps/ROOT/WEB-INF/lib/log4j-core-2.8.1.jar
[netx root]/webapps/ROOT/WEB-INF/lib/log4j-jul-2.8.1.jar
While unpatched NetX instances bundled version 2.8.1, we recommend scanning this directory for any matches of “log4j*”; if you find other versions (other than the patched 2.17.1 version), please contact NetX support for further instructions. - Stop the NetX service.
- Remove each of the jars listed above and replace them with the updated 2.17.1 version:
Remove
[netx root]/webapps/ROOT/WEB-INF/lib/log4j-1.2-api-2.8.1.jar
Add
[netx root]/webapps/ROOT/WEB-INF/lib/log4j-1.2.api-2.17.1.jar - Perform this operation for each jar listed above.
- Restart the NetX service.
- Repeat all these steps for every deployed instance of NetX.
Step 3: Patch Solr
- The following is a list of the affected jars and their path locations — [solr root] is the location of your Solr installation:
[solr root]/solr/contrib/prometheus-exporter/lib/log4j-core-2.11.0.jar
[solr root]/solr/contrib/prometheus-exporter/lib/log4j-slf4j-impl-2.11.0.jar
[solr root]/solr/contrib/prometheus-exporter/lib/log4j-api-2.11.0.jar
[solr root]/solr/server/lib/ext/log4j-core-2.11.0.jar
[solr root]/solr/server/lib/ext/log4j-slf4j-impl-2.11.0.jar
[solr root]/solr/server/lib/ext/log4j-api-2.11.0.jar
[solr root]/solr/server/lib/ext/log4j-1.2-api-2.11.0.jar
While the unpatched NetX instances bundled version of Solr 7.7.3 which includes Log4j version 2.11.0, we recommend scanning these directories for any matches of “log4j*”; if you find other versions (other than the patched 2.17.1 version), please contact NetX support for further instructions. - Stop the Solr service.
- Remove each of the jars listed above and replace them with the updated 2.16.0 version:
Remove
[solr root]/solr/server/lib/ext/log4j-core-2.11.0.jar
Add
[solr root]/solr/server/lib/ext/log4j-core-2.17.1.jar - Perform this operation for each jar listed above.
- Restart the Solr service.
Step 4: Remove Possible Old Version of Solr
Older installations of NetX may still contain an unused installation of Solr. Please check if any of the following paths exist (and are not the location of the Solr 7.7.3 server installation)
[netx root]/solr[netx root]/webapps/solr
[netx root]/webapps/solr.war
[netx root]/work/Catalina/localhost/solr
If you see one or more of these paths, please perform the following steps:
- Stop the NetX service.
- Remove each of the paths listed above, and all files under the paths that are directories.
- Restart the NetX service
- Repeat all these steps for every deployed instance of NetX.
Step 5: Turn off MediaRich (if applicable)
If you have purchased MediaRich and use it in conjunction with NetX, NetX recommends disabling MediaRich until Equilibrium can provide a patch. NetX will follow up with customers of MediaRich once we get a course of action from the vendor.
Step 6: Scan Intrusion
NetX highly recommends having industry-standard intrusion detection software installed.
Step 7: Avoid Bringing Back Old Log4J Versions
We recommend that you review all processes for managing your NetX instance(s). For example, assuming you have backups, restoring those backups without re-running all these steps could reimplement the Log4j vulnerability. Carefully consider all IT processes — especially Disaster Recovery operations — to avoid bringing back a vulnerability in the future.
Step 8: Read and Follow CISA Guidelines
CISA has put out an excellent high-level doc for guidance on handling security incidents, particularly relevant as we go into the holidays:
Updates
Jan 4, 2022
NetX released a new patch, version 9.16.11 which includes the latest Log4j patch: 2.17.1. NetX recommends all on-premise customers upgrade to this latest NetX version at their earliest convenience. For our SaaS customers, we will be upgrading all sites to this new version over the next month. There is no urgent need to upgrade because we are not running configurations that are known to be affected by issues addressed in current Log4j patches beyond 2.16.0. It's also worth noting that all NetX SaaS sites are running with an added Java agent override, which removed the JNDI lookup (networking) aspect of the underlying vulnerability.
Dec 20, 2021
NetX was evaluating and testing the latest 2.17.0 Apache Log4j patch. While the the CVSS (severity) score set by the Apache Log4j security team for version 2.16.0 is 7.5 (DOS), the NetX 9.16.9+ patch included an explicit java agent that disabled the JndiLookup class altogether. Furthermore, NetX does not employ any non-default Log4j configurations. Therefore, we believe there is not a current DOS threat when running our SaaS-configured NetX 9.16.9+ using Log4j 2.16.0.
Dec 17, 2021
NetX continued upgrading all customer SaaS instances to 9.16.10 during non-business hours. This includes the latest 2.16.0 Log4j patch, and the java agent override. Furthermore, the Apache Log4j security team increased the CVSS (severity) score for version 2.15.0 from an initial score of 3.7 (limited DOS) to a CVSS score of 9.0 (limited RCE). NetX HIGHLY recommends all on-premise customers upgrade to the latest NetX version, and/or ensure they have patched all Log4j libraries to the latest 2.16.0 version.
Dec 16, 2021
NetX began upgrading all customer SaaS instances to 9.16.9 during non-business hours. This includes the latest 2.16.0 Log4j patch, and the java agent override.
Dec 15, 2021
The latest 2.16.0 Log4j patch removed the message pattern replacement aspect of the vulnerability. NetX also added a java agent override, which removed the JNDI lookup (networking) aspect of the underlying vulnerability. NetX began deploying a Java agent and Log4j version 2.16.0 patches to all non-Production SaaS instances.
Dec 14, 2021
The Apache Log4j team issued a second security patch: version 2.16.0. According to the release notes: “It was found that the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations.” NetX does not deploy Log4j in a non-default configuration, but we patched NetX with this new version of Log4j, as NetX version 9.16.9.
Dec 13, 2021
NetX learned that some on-premise customers left old installations of Solr4 in place when they upgraded from version 8 to 9. Those old versions of Solr4 contained a version 1 bundle of Log4j; while not directly affected by CVE-2021-44228, we recommended that they be removed. NetX sent out a second “Follow-up: Critical 0day exploit in Apache Log4j library (CVE-2021-44228)” email that refined our initial patch instructions, and included explicit instructions for checking and removing any old Solr4 installations.
This email also included additional guidelines and recommendations (see the On-Premise section above for the latest info).
Dec 10, 2021
Informed of the Log4j security issue, we investigated and determined that NetX bundles Log4j versions 2.8 and 2.11 (via Solr) are both vulnerable to the exploit. We patched all SaaS instances with NetX 9.16.8, replacing those two vulnerable versions with the latest Log4j patch (version 2.15.0).
NetX emailed all on-premise customers with instructions for how to upgrade and/or manually patch Log4j.