ElasticSearch - Spring Boot - Upsert is throwing DocumentMissingException

Question

I am using Spring Boot and ElasticSearch. When I am trying to upsert using Spring, it is throwing DocumentMissingException when there is no document present in the ElasticSearch. The same code works fine when there is a document present in the ElasticSearch.

Exception Stacktrace:

org.springframework.data.elasticsearch.ElasticsearchException: Bulk indexing has failures. Use ElasticsearchException.getFailedDocuments() for detailed messages [{U65929AR1978SGC001748=[company/vteSxfKoRF-k4g982vissw][[company][2]] DocumentMissingException[[_doc][U65929AR1978SGC001748]: document missing], U45309AR2000PTC006288=[company/vteSxfKoRF-k4g982vissw][[company][3]] DocumentMissingException[[_doc][U45309AR2000PTC006288]: document missing],...

Code:

public <S extends Company> void saveAllCustom(Iterable<S> companies) {

    List<UpdateQuery> updateQueries = new ArrayList<UpdateQuery>();

    ObjectMapper oMapper = new ObjectMapper();

    for (S company : companies) {

        Map<String, Object> companyJsonMap = (Map<String, Object>) oMapper.convertValue(company, Map.class);
    
        Map<String, Object> paramsDocument = new HashMap<String, Object>();
        paramsDocument.put("doc", companyJsonMap);
    
        Script script = new Script(
            ScriptType.INLINE
            , "painless"
            , "if (ctx._source.dataAsOf < params.doc.dataAsOf) {ctx._source = params.doc}"
            , paramsDocument);
    
        UpdateRequest updateRequest = new UpdateRequest()
            .index("company")
            .type("_doc")
            .id(company.cin)
            .script(script) // Conditional Update
            .upsert(companyJsonMap);

        UpdateQuery updateQuery = new UpdateQueryBuilder()
            .withIndexName("company")
            .withType("_doc")
            .withId(company.cin)
            .withDoUpsert(true)
            .withClass(Company.class)
            .withUpdateRequest(updateRequest)
            .build();
    
        updateQueries.add(updateQuery);
        
    }

    elasticsearchTemplate.bulkUpdate(updateQueries);
}

But a similar upsert command is working using CURL:

curl -X POST "localhost:9200/company/_doc/10001/_update" -H 'Content-Type: application/json' -d'
{
    "script": {
        "lang": "painless",
        "source": "ctx._source = params.doc",
        "params": {
            "doc": {
                "name": "XYZ LIMITED - updated",
                "cin": "10001"
            }
        }
    },
    "upsert" : {
        "name": "XYZ LIMITED - newly created",
        "cin": "10001"
    }
}'

As per my understanding, when there is no document present in the ElasticSearch, it shouldn't throw a DocumentMissingException because I have added upsert(...) as well in the query.

Elasticsearch version: Version: 6.4.3, Build: default/tar/fe40335/2018-10-30T23:17:19.084789Z, JVM: 1.8.0_191

Plugins installed: None

JVM version: java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

OS version: Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64

Spring Boot Version: 2.1.7.RELEASE

Spring Data ElasticSearch Version: 3.2.0.RC2

Answer 1

This is currently not supported by Spring Data Elasticsearch.

We can see it in the source code of ElasticsearchTemplate that the upsert is not taken into account when a script is used.

Note that this issue has been reported, but it still hasn't been solved.