'How to resolve Aerospike error, Operation not allowed at this time?

I am using Aerospike 4.8v and persisting my data on disk and I am making parallel write requests to aerospike, if i make 10 parallel requests then it works fine but when I make 100 parallel requests it gives error i.e. Operation not allowed at this time, with code 22. I think that aerospike must be capable of handling hundreds or thousands of requests in parallel but I don't know whats wrong so if anyone can guide that would be helpful.

Error log:

 error : { AerospikeError: Operation not allowed at this time.
    at Function.fromASError (/data/codebase/lib/node_modules/aerospike/lib/error.js:113:21)
    at QueryCommand.convertError (/data/codebase/lib/node_modules/aerospike/lib/commands/command.js:91:27)
    at QueryCommand.convertResponse (/data/codebase/lib/node_modules/aerospike/lib/commands/command.js:101:24)
    at asCallback (/data/codebase/lib/node_modules/aerospike/lib/commands/command.js:163:24)
  name: 'AerospikeError',
  code: 22,
  command:
   QueryCommand {
     client:
      Client {
        domain: null,
        _events: {},
        _eventsCount: 0,
        _maxListeners: undefined,
        config: [Object],
        as_client: AerospikeClient {},
        connected: true,
        captureStackTraces: false },
     args: [ 'antiSpamming', 'userTargetingMatrix', [Object], undefined ],
     captureStackTraces: false,
     key: undefined,
     ensureConnected: true,
     stream:
      RecordStream {
        aborted: false,
        client: [Object],
        _events: [Object],
        _eventsCount: 3 } },
  func: 'as_query_parse_records_async',
  file: 'src/main/aerospike/aerospike_query.c',
  line: 246,
  inDoubt: false }

Warning logs:

Jan 28 06:03:25 ip-1-0-4-78 asd[32437]: Jan 28 2020 06:03:25 GMT: WARNING (scan): (scan_manager.c:103) at scan threads limit - can't start new scan
Jan 28 06:03:25 ip-1-0-4-78 asd[32437]: Jan 28 2020 06:03:25 GMT: WARNING (scan): (scan.c:676) basic scan job 5614303283813349397 failed to start (22)
aerospikeaerospike-ce


Solution 1:[1]

You are most likely exceeding the default limit related to queries. First, see the definition for error code 22:

For scans with server 4.7 or later, no threads available (scan-threads-limit reached).

Based on your other question, you're doing scans rather than a query with a secondary index. You'll need to increase that limit, as suggested in the 'Additional Information'. However, you have a pretty weak system in terms of CPU, so you should adjust that value and benchmark, comparing performance before and after with the same workload. In a real production system you'd have multiple nodes, probably want more than two CPU cores, and similarly tune the scan threads as needed.

Solution 2:[2]

This happened to me because a job was performing a backup while I was trying to scan the partitions.

Recommendation, do not run an extra job while a backup is in process.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Ronen Botzer
Solution 2 Walter Alonso Alvarado Sandova

Related Questions