'Performance Issues Saving ActiveMQTextMessages
I am trying to consume ActiveMQTextMessages (received as Message objects) and store into a file given a very high data rate (9K a second is the target). I am running into performance problems trying to save the messages into a set of files.
I am currently receiving at 30K a second and pushing into a ConcurrentLinkedQueue of which I have a configurable set of workers popping off the queue until they reach a max internal queue amount. This all seems fine. The issue I am having is when I am passing the concatenated text (longest running time I think) to the compression algorithm. I would be willing to store the textmessage received from the message but, when I stored it out, what would be a more efficient way to handle the following pseudo?
FileOutputStream fout = new FileOutputStream(op);
Deflater df = new Deflater(Deflater.BEST_SPEED);
DeflaterOutputStream out = new DeflaterOutputStream(fout, df);
while(!localQueue.isEmpty())
{
Message msg = localQueue.remove();
ActiveMQTextMessage amqtextmesssage = (ActiveMQTextMessage)msg;
String textMessage = amqtextmesssage.getText();
String jmsCorrID = amqtextmesssage.getJMSCorrelationID();
Destination destinationID = amqtextmesssage.getJMSDestination();
long jmsTimeStamp = amqtextmesssage.getJMSTimestamp();
//FORMAT FOR SPECIFIC OUTPUT TO BE POSTPROCESSED
String result = String.format(/*format strings here*/);
InputStream is = new ByteArrayInputStream(result.getBytes());
int i;
while ((i = is.read()) != -1) {
out.write((byte) i);
out.flush();
}
is.close();
}
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|