Cannot allocate memory when running xdmp:document-filter() in spawn-function()

Question

I have a function that runs xdmp:document-filter() on binary documents. I call it like this:

let $doc-metadata-and-text := xdmp:document-filter(fn:doc("/ediscovery/mydocument.doc")/binary())

When I put this in a spawn-function() I get out-of-memory errors. Here is what I see in the log file:

2017-01-10 11:53:52.038 Notice: TaskServer: SVC-PROCESSRUN: xdmp:document-filter(fn:doc("/ediscovery/mydocument.doc")/binary()) -- Process run error: fork: Cannot allocate memory

Although if I call xdmp:document-filter() from an XQuery main, it works fine.

Any suggestions?

Answer 1

xdmp:spawn-function executes the task using a thread from the Task Server. This has it's own settings and would not be the same context as running it via query console or other application servers.

Under your configuration group, have a look at the Task Server configuration and consider altering the max-inference-size.

However, changing memory/thread settings should be done with caution as it changes how the server uses finite resources. In your case, the confusing part is that App Services, the Task Server and also default settings for any other application server all default the setting to the same amount.

https://docs.marklogic.com/admin-help/task-server

Answer 2

I've seen this function fail with the same "Cannot allocate memory" error before when it's run on an app server and not on the task server. One thing to check is your swap space, as xdmp:document-filter is forking a separate process (as indicated by the error).

ML recommendations are at https://help.marklogic.com/knowledgebase/article/View/21/19/swap-space-requirements.