Unable to send logs from kinesis firehose to opensearch

Question

I have a workflow, where:

• All cloudwatch logs will be streamed to Kinesis Firehose Delivery Stream.
• From Kinesis Firehose Delivery Stream, it will be sent to Opensearch.

However, Kinesis is failed to put data to opensearch and returning the error:

The Amazon OpenSearch Service cluster returned a JsonParseException. Ensure that the data being put is valid.

I have tried all available Lambda blueprints for firehose to transform records as well, but no luck.

Any suggestions are appreciated.

Thank you.

Answer 1

I raised an AWS ticket and got this reply

I understand that you are implementing the below data flow:

Logger Lambda--> Cloudwatch log group--> Firehose (Lambda transformation) --> Streaming to Opensearch

you encountered the below error message from OpenSearch services:

-- Error Message: The Amazon OpenSearch Service cluster returned a JsonParseException. Ensure that the data being put is valid. Error Code: OS.JsonParseException

As per the error message, it indicated the data being sent to the OpenSearch cluster is not a valid JSON object. Cloudwatch log group will send multiple logs in one record against firehose, which resulted invalid JSON being sent to the OpenSearch cluster, because OpenSearch only accepts single JSON object as the data format. Meanwhile, Firehose does not allow passing back more records than what was passed to its transformation Lambda function. Therefore, it is not possible to use the current pipeline.

Suggested workarounds by AWS support

1. CloudWatch Logs -> Kinesis Data Stream -> Lambda -> Firehose -> OpenSearch
2. CloudWatch logs -> Lambda -> Firehose -> OpenSearch
3. CloudWatch logs (subscription filter) -> OpenSearch

I would like to know your thoughts on this.