'WSO2 IdendityServer throwing ThreadDeath Exception

I start the server using the ./wso2server.sh start command but after a few hours the server stops responding and throws a ThreadDeath exception.

I tried modifying the JVM_MEMOPTS environment variable to increase heap size JVM_MEM_OPTS="-Xms1024m -Xmx2048m" and that seemed to allow the server to run longer before the exception happens, but that may be just a coincidence honestly.

Do you have any ideas on how to fix this?

Stacktrace:

TID: [-1234] [authenticationendpoint] [2022-04-27 16:37:30,910] [86418802-68be-45af-af12-fd4bbfd54bf4] ERROR {org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/authenticationendpoint].[jsp]} - Servlet.service() for servlet [jsp] in context with path [/authenticationendpoint] threw exception [javax.servlet.ServletException: java.lang.ThreadDeath] with root cause java.lang.ThreadDeath
        at java.base/java.lang.Thread.stop(Thread.java:937)
        at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.handleStuckThread(CarbonStuckThreadDetectionValve.java:111)
        at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.backgroundProcess(CarbonStuckThreadDetectionValve.java:164)
        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1145)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:829)

INFO:

  • WSO2 is running in a 6GB machine with AlmaLinux hosted at Azure.
  • Tried deleting the virtual machine and creating a new one: got same problem.
  • Java version:
openjdk 11.0.15 2022-04-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.15+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+9-LTS, mixed mode, sharing)
javawso2


Solution 1:[1]

According to the log, there are JSP files with random names.

  • /authenticationendpoint/28oc6hgokjawkyo9zhidiqzmie3.jsp

Therefore, your deployment seems to be compromised with the vulnerability [1]. These JSP files could be executing commands which are running for a longer time and causing the thread deaths.

You can follow the steps provided in the same document to mitigate the vulnerability.

But when a remote code execution vulnerability gets exploited, there are numerous ways a malicious actor could create backdoors within the system and establish persistence. Due to different attack vectors, and techniques used in the exploitation, it is not possible to provide steps that are usable to cleanup an affected deployment with an assurance that the malicious actor would not be able to regain access.

Therefore, the security recommendation is to recreate the instance (without reusing the VM/server instance). However, you may use the existing configuration files from the previous deployment, after reviewing that there are no unintended modifications made to the configuration files.

[1] https://docs.wso2.com/display/Security/Security+Advisory+WSO2-2021-1738

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1

Related Questions