'Elasticsearch: Bulk request throws error in Elasticsearch 6.1.1

I recently upgraded to Elasticsearch version 6.1.1 and now I can't bulk index documents from a JSON file. When I do it inline, it works fine. Here are the contents of the document:

{"index" : {}}
{"name": "Carlson Barnes", "age": 34}
{"index":{}}
{"name": "Sheppard Stein","age": 39}
{"index":{}}
{"name": "Nixon Singleton","age": 36}
{"index":{}}
{"name": "Sharron Sosa","age": 33}
{"index":{}}
{"name": "Kendra Cabrera","age": 24}
{"index":{}}
{"name": "Young Robinson","age": 20}

When I run this command,

curl -XPUT 'localhost:9200/subscribers/ppl/_bulk?pretty' -H 'Content-Type: application/json' -d @customers_full.json

I get this error:

"error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "The bulk request must be terminated by a newline [\n]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "The bulk request must be terminated by a newline [\n]"
  },
  "status" : 400

It works fine if I send the data inline and in Elasticsearch 5.x. I tried adding newlines as well as the newline character to the end of the file. Doesn't seem to work.

json bulk-load

Solution 1:^[1]

Add empty line at the end of the JSON file and save the file and then try to run the below command

curl -XPOST localhost:9200/subscribers/ppl/_bulk?pretty --data-binary @customers_full.json -H 'Content-Type: application/json'

Solution 2:^[2]

As the document says: use the --data-binary flag instead of plain -d

-d doesn’t preserve newlines and doesn't format the json.

I faced this problem because of JSON formatting.

Solution 3:^[3]

The error is pretty clear:

The bulk request must be terminated by a newline [\n]

So you simply need to add a newline at the end of your customers_full.json file and you'll be ok.

Solution 4:^[4]

I ran into the same issue and spent hours adding and removing newlines before somebody pointed out I mis-typed the file name... So note that curl will throw the same error if the file is not actually present, making this super-confusing.

Solution 5:^[5]

I had a similar issue when working with Elasticsearch 7.3.

Here's how I solved it.

Locate the .json file, say products.json file.
Double click to open the .json file in your text editor.
Scroll to the end of the .json file and then press the ENTER key on your keyboard.
Close the .json file. This will create a new line at the end of .json file.
Go back to your terminal and run the command below.

N/B: For the command below, the .json file name is products.json which I am importing to http://localhost:9200/ecommerce/product

curl -H "Content-type: application/json" -XPOST "http://localhost:9200/ecommerce/product/_bulk?pretty" --data-binary "@products.json"

That's all.

I hope this helps

Solution 6:^[6]

For anyone using postman to make requests to ElasticSearch

Just press enter to create an empty new line!

And voila, problem solved

Solution 7:^[7]

This worked for me:

curl -H "Content-Type: application/x-ndjson" -XPOST "localhost:9200/bank/_bulk?pretty&refresh" --data-binary "@C:\Program Files\Elastic\Elasticsearch\7.2.0\accounts.json"

Solution 8:^[8]

Press Enter end of the line inside the JSON file and run the command again.

curl -H "Content-Type: application/x-ndjson" -XPOST 'localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary @"generated.json"

Solution 9:^[9]

you just need to open json file and then go to the end of the file ( Ctrl+end) and then please Enter to break a new line.

Solution 10:^[10]

I had the same problem running on Windows 10, using ElasticSearch 7.5.1.

I tried all the answers; none of them worked. I was certain I had a newline at the end of my file.

To get it to work, I had to ensure the file I was uploading was using UNIX end-of-line characters (0A only, no 0D), and also the encoding had to be UTF-8.

Using Notepad++, you can edit the metadata of the file.

Finally some good news:

Solution 11:^[11]

I was struggling with this for a hot minute. Mine was caused by a space in my curl request between the --data and the -binary and gave the same error - must end with new line [\\n]}.

So double-check that in the curl req it's --data-binary not --data - binary

Solution 12:^[12]

For me, the issue was only due to the wrong file name. I have used customer_full.json in command whereas the file was named customer_full in my file system (without the extension).

So in my case,this command worked for me:

curl -H "Content-Type: application/x-ndjson" -XPOST 'http://localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary @"customer_full"

Solution 13:^[13]

I faced a similar issue on windows using elastic 7.9.1 When I used below CURL command.

curl -s -H "Content-Type: application/json" -XPOST localhost:9200/accounts/docs/_bulk?filter_path=items.*.error --data-binary  "@textoutES.json"  >> erroredAtES.json"

I tried to manually add Newline at the end of the file but did not work.

I have created my JSON by extracting data from MySQL database like below to make sure my records should end with LINE FEED and CARRIAGE RETURN.

Then, it is working for me:

SELECT CONCAT('{"index":{"_id":"',id,'"}}\r\n',request_data,'\r\n') reqestData FROM cards

More importantly you End-of-File should have a carriage-return and Line-Feed (CRLF)if you are using windows. Also if any line in JSON contains a CR but no LF then you will get parsing exception Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@2d5ed2ca

Windows CRLF and EOF

Solution 14:^[14]

You need to use --data-binary instead of -d in your curl request. Please see: Bulk API

Solution 15:^[15]

I just forgot to add an @ symbol before file name like this

--data-binary "@products.json"

Solution 16:^[16]

This worked in my local set-up.

curl -H "Content-type:application/json" -XPOST "http://localhost:9200/customer/personal/_bulk?pretty" --data-binary @"generated.json"

Solution 17:^[17]

How do you do that if you are not using a data-file? I am having the issue but not sending data from a file.

const data1 = {
    "amount" : "100",
    "@timestamp" : `${UTC_timestamp}`,
    "transaction_attributes" : {
    "channel" : "channel-foobarbaz",
    "session_id" : "session-1234",
    "information" : "iinformation-foobarbaznformation-foobarbaz"
    },
    "currency" : {
    "currency_description" : "my currency description",
    },
    "external_timestamp" : "2021-12-03T11:22:55.206229500Z" };

  
// execute a post
let res = http.post(url,JSON.stringify(data1),params);

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	Sabito éŒ†å…Ž stands with Ukraine
Solution 2	avp
Solution 3	Val
Solution 4	Raya Fratkina
Solution 5
Solution 6	bnahayo
Solution 7	Shayki Abramczyk
Solution 8	David Buck
Solution 9	user6589750
Solution 10	Vince I
Solution 11	Sabito éŒ†å…Ž stands with Ukraine
Solution 12	Sabito éŒ†å…Ž stands with Ukraine
Solution 13	Sabito éŒ†å…Ž stands with Ukraine
Solution 14	Sabito éŒ†å…Ž stands with Ukraine
Solution 15
Solution 16	David Buck
Solution 17	fatkobra

'Elasticsearch: Bulk request throws error in Elasticsearch 6.1.1

Solution 1:[1]

Solution 2:[2]

Solution 3:[3]

Solution 4:[4]

Solution 5:[5]

Solution 6:[6]

Solution 7:[7]

Solution 8:[8]

Solution 9:[9]

Solution 10:[10]

Solution 11:[11]

Solution 12:[12]

Solution 13:[13]

Solution 14:[14]

Solution 15:[15]

Solution 16:[16]

Solution 17:[17]