How to determine the maximum number of messages in *USRQ

Question

It is required to create *USRQ of the largest possible size. According to the documentation, the maximum size for *USRQ is 2Gb. Creating a queue requires specifying the maximum message size, the initial number of messages in the queue, the size (in messages) of the queue expansion, and the maximum number of expansion operations. Let's the message size is 1024 bytes. The initial number is 128 messages. The queue will expand by 128 messages. Estimate the maximum possible number of messages - 2Gb / 128 bytes. Then we subtract the initial number of messages (128) and divide by the size of the extension - 128. As a result, we get the maximum number of extensions - 16,383. We pass these parameters to QUSCRTUQ, after which we look at what we got (call matqat). We see that the maximum number of extensions (mat_template.Max_Extend) is set less than the requested one - 15 306, and the maximum number of messages in the queue is therefore is 1 959 296 Then start filling the queue and at some point get the error "Tried to go larger than storage limit" At the same time, the number of messages in the queue is 1,957,504, the number of extensions used is 15,282. Why does this happen and how to correctly estimate the maximum number of increments when creating a queue?

Answer 1

I think with more experimentation you will find that IBM i always allocates memory in pages. It is probably going through the allocation procedure for each message and allocating space on the *USRQ from the previous memory allocation until the allocated memory is full, then it allocates another block large enough to contain the next allocation.

Let me give you a simple example. A dataarea can be up to 2000 bytes, but no matter what size you make it, it always shows a physical size of 8K bytes. Internally, it knows how much of that 8K space can be used.

So what is happening?

• You create a *USRQ with a minimal size: 1 message 64 bytes/message.
• The system creates an object with a minimal physical size 16K but a smaller logical size 64bytes + overhead
• You add a message
• The system adds a message
• You add another message
• The system checks to see if it has enough memory allocated for the next message and if so extends the logical size of the *USRQ.
• The system then adds the message
• This continues as messages are added until the allocated memory is no longer large enough to hold the next extension. At this point the OS allocates more physical memory in page increments, but at least enough to hold the requested extension.

So there are a couple allocation schemes going on here. The *USRQ routines are extending the logical size of the object based on the *USRQ parameters, and the OS is allocating physical memory in page size blocks as requested by the *USRQ routines. Thus a *USRQ extension does not necessarily require a physical memory allocation. It also points out a possible optimization based on setting the initial size of the queue such that it fills up at least 16K, and also setting the size of the extension such that it fills up at least 4K. That way the API is not running through the extension logic more often than necessary.

Answer 2

Consider that there is (and must be) some internal "overhead" for a queue to keep all of the enqueued messages chained together in the proper LIFO or FIFO order, etc. These internal "linked lists" or "pointers" are not "free".

Create a small test *USRQ, and then do a DMPOBJ of that. Then add a few messages to the queue, and DMPOBJ of that again. Then "de-queue" a few messages, and do another DMPOBJ. Then compare the spool files of these dumps to see what is going on.

Answer 3

This doesn't really answer your question directly, but I don't think my comments are working, so I'll try an answer.

The documentation says

If the number of queue extensions is specified and non-zero, the queue will be extended by the number of additional messages specified until either the number of queue extensions is reached or the storage limit of 2GB is reached.

In other words, the number of extensions you specify in QUSCRTUQ is just a request; the system will stop you before you reach that number if the limit would be exceeded.

As pointed out in Mark's answer, there is going to be overhead which uses up some of that 2GB, so it shouldn't be surprising that MATQAT reports a number of extensions lower than what you requested.

What is surprising (to me, anyway) is that when you actually add data to the queue, you might not even get the number of extensions reported by MATQAT. So the only way to get an accurate number seems to be adding data until you get the error message.

Given that, the next thing I would look at is what the possible user inputs are. Can they enter literally anything? Or do they have to choose among a limited set of values? If the number of possible combinations of user input is "manageable" then you could write a script which tries all of them. Use a huge value for the number of extensions (deliberately requesting more than 2GB) and monitor for the error message to capture the number of extensions you really got. If you can do this for every possible user input, then you can make a lookup table for the number of extensions, and actually use it instead of doing arithmetic.

If there are too many possible user inputs, then you just have to try a representative sample. And instead of a lookup table, do arithmetic, but based on the smallest effective size encountered in the sample (or even smaller, to provide a margin of safety).

Answer 4

Firstly, I want to say thanks to everyone who commented and answered - it helped a lot to find a solution. Special thanks to @MarkSWaterbury - your answer was very helpful in finding the right approach in finding a solution. What we managed to understand and see.

The following parameters were chosen for the experiment:

• the size of one message is 64 bytes
• initial queue capacity - 1 message
• queue increment - 1 message

That. each time a new message is added, the size of the queue should increase.

After the queue is created, its size is 16Kb. This is the same as the size of the "associated space" (which can be obtained via MATMDATA).

Regardless of the size of the message and the size of the queue increment (the number of messages in the increment), the physical size of the queue does not increase every time, but due to lack of capacity and a multiple of 1 page (MATMDATA says that the page size is 4Kb).

In our case, the first 26 messages do not cause a change in the physical size of the queue. The 27th leads to its increase by 4Kb. Further, until the 55th message, the physical size of the queue does not change. The 56th increases it by another 4Kb. Then up to the 86th without changes, the 87th again increases the physical size by 4Kb...

All this allows us to evaluate the alleged Mark S Waterbury "internal overhead". Here, also, it should be noted that the physical size of the message in the queue consists of the user-specified maximum message size and the size of the message header - Message attributes in a structure filled with MATQMSG and containing:

• Message enqueue time Char(8)
• Message length Bin(4)
• Reserved Char(4)

For a KEYED queue, the key size is added to the physical size of the message.

Using the proposed method, we can roughly estimate the "internal overhead" mentioned above, which is 64 bytes (perhaps a little less, because due to the page increment of the physical size of the queue, the exact value is different each time, but with alignment by 16 let it be like this).

In total, we get an additional 80 bytes to the sum of the maximum message size specified by the user and the key size (for KEYED queues, for FIFO and LIFO queues, the key size is 0).

Now, if we count the maximum number of increments, based on the specified maximum queue size (2GB, as indicated in the documentation) and the physical size of the message, defined as described above, we get the correct value, which we pass to QUSCRTUQ. If we then call MATQAT and calculate back the maximum size of messages in the queue, using the values returned to it Initial number of messages, Additional number of messages and Number of queue extensions, we get the real value of the maximum number of messages in the queue, which does not throw exception 1C04 "Object Storage Limit Exceeded" (MCH2804). That. now you can control the degree of filling the queue by the ratio of the number of messages in it and the calculated maximum number of messages without fear of an exception.

And what is all this for? If we have *DTAQ with which everything is much simpler... The thing is, if we don't need all the features of *DTAQ (journaling, saving content to disk, using remote queues...) and we only work with local queues, *USRQ is 4-5 times faster and consumes about the same times less CPU resources.