'Compose Transporter throws error when collection_filters is set to sync data for current day from DocumentDB/MongoDB to file/ElasticSearch

I am using Compose Transporter to sync data from DocumentDB to ElasticSearch instance in AWS. After one time sync, I added following collection_filters in pipeline.js to sync incremental data daily:

// pipeline.js
var source = mongodb({
  "uri": "mongodb <URI>"
  "ssl": true,
  "collection_filters": '{ "mycollection": { "createdDate": { "$gt": new Date(Date.now() - 24*60*60*1000) } }}',
})

var sink = file({
  "uri": "file://mongo_dump.json"
})

t.Source("source", source, "^mycollection$").Save("sink", sink, "/.*/")

I get following error:

$ transporter run pipeline.js
panic: malformed collection_filters [recovered]
    panic: Panic at 32: malformed collection_filters [recovered]
    panic: Panic at 32: malformed collection_filters

goroutine 1 [running]:
github.com/compose/transporter/vendor/github.com/dop251/goja.(*Runtime).RunProgram.func1(0xc420101d98)
    /Users/JP/gocode/src/github.com/compose/transporter/vendor/github.com/dop251/goja/runtime.go:779 +0x98

When I change collection_filters so that value of "gt" key is single string token (see below), malformed error vanishes but it doesn't fetch any document:

'{ "mycollection": { "createdDate": { "$gt": "new Date(Date.now() - 24*60*60 * 1000)" } }}',

To check if something is fundamentally wrong with the way I am querying, tried simple string filter and that works well:

"collection_filters": '{ "articles": { "createdBy": "author name" }}',

I tried various ways to pass createdDate filter but either getting malformed error or no data. However same filter on mongo shell gives me expected output. Note that I tried with ES as well as file as sink before asking here.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source