'High CPU usage from the Spring DefaultMessageListenerContainer Thread
I have a application which uses Spring DefaultMessageListenerContainer and IBM mq as the message broker, after running application for certain time, the cpu usage spiked with other threads with similar cpu usage. Below is one of the messageListenerContainer thread from thread dump with top command.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29410 tomcat 20 0 10.4g 4.0g 20320 R 25.0 12.6 610:36.22 msgListenerCont
"msgListenerContainer" #265306 prio=5 os_prio=0 cpu=36635917.55ms elapsed=3263601.16s tid=0x00007f8804046800 nid=0x72e2 runnable [0x00007f87abe79000]
java.lang.Thread.State: RUNNABLE
at java.lang.ThreadLocal$ThreadLocalMap.set([email protected]/ThreadLocal.java:487)
at java.lang.ThreadLocal.set([email protected]/ThreadLocal.java:222)
at com.ibm.msg.client.commonservices.locking.TraceableLock.incrementDepth(TraceableLock.java:99)
at com.ibm.msg.client.commonservices.locking.TraceableLock.lock(TraceableLock.java:90)
at com.ibm.msg.client.commonservices.locking.TraceableReentrantLock.lock(TraceableReentrantLock.java:73)
at com.ibm.mq.jmqi.remote.util.HconnLock.lock(HconnLock.java:83)
at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.requestMutex(RemoteProxyQueue.java:367)
at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.requestMessagesReconnectable(RemoteProxyQueue.java:1046)
- locked <0x00000007166f9d48> (a com.ibm.mq.jmqi.remote.impl.RemoteSession$RequestMessagesMutex)
at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.requestMessages(RemoteProxyQueue.java:717)
at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.flushQueue(RemoteProxyQueue.java:1911)
at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.proxyMQGET(RemoteProxyQueue.java:2783)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGetInternalWithRecon(RemoteFAP.java:6103)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGetInternal(RemoteFAP.java:5992)
at com.ibm.mq.jmqi.internal.JmqiTools.getMessage(JmqiTools.java:1371)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGet(RemoteFAP.java:5957)
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiGet(InterceptedJmqiImpl.java:1341)
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiGet(ESEJMQI.java:602)
at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.getMsg(WMQConsumerShadow.java:1795)
at com.ibm.msg.client.wmq.internal.WMQSyncConsumerShadow.receiveInternal(WMQSyncConsumerShadow.java:228)
at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.receive(WMQConsumerShadow.java:1461)
at com.ibm.msg.client.wmq.internal.WMQMessageConsumer.receive(WMQMessageConsumer.java:674)
at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receiveInboundMessage(JmsMessageConsumerImpl.java:1051)
at com.ibm.msg.client.jms.internal.JmsMessageConsumerImpl.receive(JmsMessageConsumerImpl.java:667)
at com.ibm.mq.jms.MQMessageConsumer.receive(MQMessageConsumer.java:209)
at org.springframework.jms.support.destination.JmsDestinationAccessor.receiveFromConsumer(JmsDestinationAccessor.java:132)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:418)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:303)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076)
at java.lang.Thread.run([email protected]/Thread.java:834)
From above thread stack trace it looks to me that thread is stuck doing something with ThreadLocalMap.set method, but other than that I can't find why it is using so much cpu.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|