'High CPU usage from the Spring DefaultMessageListenerContainer Thread

I have a application which uses Spring DefaultMessageListenerContainer and IBM mq as the message broker, after running application for certain time, the cpu usage spiked with other threads with similar cpu usage. Below is one of the messageListenerContainer thread from thread dump with top command.

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
29410 tomcat    20   0   10.4g   4.0g  20320 R 25.0 12.6 610:36.22 msgListenerCont
"msgListenerContainer" #265306 prio=5 os_prio=0 cpu=36635917.55ms elapsed=3263601.16s tid=0x00007f8804046800 nid=0x72e2 runnable  [0x00007f87abe79000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.ThreadLocal$ThreadLocalMap.set([email protected]/ThreadLocal.java:487)
    at java.lang.ThreadLocal.set([email protected]/ThreadLocal.java:222)
    at com.ibm.msg.client.commonservices.locking.TraceableLock.incrementDepth(TraceableLock.java:99)
    at com.ibm.msg.client.commonservices.locking.TraceableLock.lock(TraceableLock.java:90)
    at com.ibm.msg.client.commonservices.locking.TraceableReentrantLock.lock(TraceableReentrantLock.java:73)
    at com.ibm.mq.jmqi.remote.util.HconnLock.lock(HconnLock.java:83)
    at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.requestMutex(RemoteProxyQueue.java:367)
    at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.requestMessagesReconnectable(RemoteProxyQueue.java:1046)
    - locked <0x00000007166f9d48> (a com.ibm.mq.jmqi.remote.impl.RemoteSession$RequestMessagesMutex)
    at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.requestMessages(RemoteProxyQueue.java:717)
    at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.flushQueue(RemoteProxyQueue.java:1911)
    at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.proxyMQGET(RemoteProxyQueue.java:2783)
    at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGetInternalWithRecon(RemoteFAP.java:6103)
    at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGetInternal(RemoteFAP.java:5992)
    at com.ibm.mq.jmqi.internal.JmqiTools.getMessage(JmqiTools.java:1371)
    at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGet(RemoteFAP.java:5957)
    at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiGet(InterceptedJmqiImpl.java:1341)
    at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiGet(ESEJMQI.java:602)
    at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.getMsg(WMQConsumerShadow.java:1795)
    at com.ibm.msg.client.wmq.internal.WMQSyncConsumerShadow.receiveInternal(WMQSyncConsumerShadow.java:228)
    at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.receive(WMQConsumerShadow.java:1461)
    at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.receive(WMQMessageConsumer.java:674)
    at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receiveInboundMessage(JmsMessageConsumerImpl.java:1051)
    at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receive(JmsMessageConsumerImpl.java:667)
    at com.ibm.mq.jms.MQMessageConsumer.receive(MQMessageConsumer.java:209)
    at org.springframework.jms.support.destination.JmsDestinationAccessor.receiveFromConsumer(JmsDestinationAccessor.java:132)
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:418)
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:303)
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076)
    at java.lang.Thread.run([email protected]/Thread.java:834)

From above thread stack trace it looks to me that thread is stuck doing something with ThreadLocalMap.set method, but other than that I can't find why it is using so much cpu.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source