'Issue in placing DataDog log annotation on deployment via ansible

I am using ansible version 2.7 for kubernetes deployment. For sending logs to datadog on kubernetes one of the way is to configure annotations like below,

template:
    metadata:
      annotations:
        ad.datadoghq.com/nginx.logs: '[{"source":"nginx","service":"webapp"}]'

this works fine and I could see logs in DataDog.

However I would like to achieve above configuration via ansible deployment on kubernetes for which I have used below code

 template:
        metadata:
           annotations:
             ad.datadoghq.com/xxx.logs: "{{ lookup('template', './datadog.json.j2')}}"

and datadog.json.j2 looks like below

'[{{ '{' }}"source":"{{ sourcea }}"{{ ',' }} "service":"{{ serviceb }}"{{ '}' }}]'  **--> sourcea and serviceb are defined as vars**

However the resulting config on deployment is below

template:
    metadata:
      annotations:
        ad.datadoghq.com/yps.logs: |
          '[{"source":"test", "service":"test"}]'

and this config does not allow datadog agent to parse logs failing with below error

[ AGENT ] 2019-xx-xx xx10:50 UTC | ERROR | (kubelet.go:97 in parseKubeletPodlist) | Can't parse template for pod xxx-5645f7c66c-s9zj4: could not extract logs config: in logs: invalid character '\'' looking for beginning of value

if I use ansible code as below (using replace) 

template:
        metadata:
           annotations:
             ad.datadoghq.com/xxx.logs: "{{ lookup('template', './datadog.json.j2', convert_data=False) | string | replace('\n','')}}"

it generates deployment config as below

template:
    metadata:
      annotations:
        ad.datadoghq.com/yps.logs: '''[{"source":"test", "service":"test"}]'''
      creationTimestamp: null
      labels:

Which also fails,

to configure the working config with ansible, I have to either remove leading pipe (|) or three quotes coming when using replace).

I would like to have jinja variables substitution in place so that I could configure deployment with desired source and service at deployment time.

kindly suggest

kubernetesansiblejinja2datadog


Solution 1:[1]

By introducing space in datadog.json.j2 template definition .i.e.

 [{"source":"{{ sourcea }}"{{ ',' }} "service":"{{ serviceb }}"}] (space at start)

and running deployment again I got the working config as below 

template:
    metadata:
      annotations:
        ad.datadoghq.com/yps.logs: ' [{"source":"test", "service":"test"}]'

However I am not able to understand the behaviour if anyone could help me understand it

Solution 2:[2]

The problem is that the YAML being produced is broken. The | character starts an indented scalar ("string" more or less), but the next line includes single-quotes - so the quotes end up being inside the annotation value.

The correct YAML should look like:

template:
    metadata:
      annotations:
        ad.datadoghq.com/yps.logs: |
          [{"source":"test", "service":"test"}]

This looks like a bug in how Ansible is generating the output YAML, and your fix must have worked around the bug.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Ruchir Bharadwaj
Solution 2 jkinkead

Related Questions