Dialogflow log give me an JSON without commas in stackdriver

Question

I'm working with dialogflow for creating chatbots, i want to recover all the messages and send them to bigquery, after i will link bigquery to datastudio to show a data panel to my client.

Dialogflow acctually sends all the log to stackdriver but when the textPayload is open it doesnt have commas to use it as a common JSON.

¿Some idea?

Stackdriver log

Dialogflow Response : id: "3f6c4006-e8fd-4c59-af75-1f62e5785c3c-2dd8e723"
lang: "es"
session_id: "d1eea2ea-3dcc-447c-8883-a23a89d9e570"
timestamp: "2019-06-09T20:21:24.465Z"
result {
  source: "agent"
  resolved_query: "Comunidad"
  score: 1.0
  parameters {
  }
  contexts {
    name: "generic"
    lifespan: 4
    parameters {
      fields {
        key: "facebook_sender_id"
        value {
          string_value: "2665762213453771"
        }
      }
    }
  }
  metadata {
    intent_id: "843d151c-ffaa-43a5-b1af-7537947aeb98"
    intent_name: "CommunityIntent"
    webhook_used: "false"
    webhook_for_slot_filling_used: "false"
    is_fallback_intent: "false"
  }
  fulfillment {
    speech: "Nuestra comunidad busca vincular desarrolladores para crecer como un ecosistema tecnol\303\263gico. Puedes checar m\303\241s sobre nuestros eventos en http://cloudmex.io/servicios/workshops-y-meetups/"
    messages {
      lang: "es"
      type {
        number_value: 0.0
      }
      speech {
        string_value: "Nuestra comunidad busca vincular desarrolladores para crecer como un ecosistema tecnol\303\263gico. Puedes checar m\303\241s sobre nuestros eventos en http://cloudmex.io/servicios/workshops-y-meetups/"
      }
    }
  }
}
status {
  code: 200
  error_type: "success"
}

**UPDATE: Stackdriver log **

{
insertId:  "qd0nynfxfz53o"  
 labels: {
  protocol:  "V2"   
  request_id:  "7034b4ec-3b54-4f0c-9cee-8a37eb117f66-2dd8e723"   
  source:  "facebook"   
  type:  "dialogflow_response"   
 }
 logName:  "projects/cloudy-76f21/logs/dialogflow_agent"  
 receiveTimestamp:  "2019-06-09T19:32:53.416394429Z"  
 resource: {
  labels: {
   project_id:  "cloudy-76f21"    
  }
  type:  "global"   
 }
 severity:  "INFO"  
 textPayload:  "Dialogflow Response : id: "7034b4ec-3b54-4f0c-9cee-8a37eb117f66-2dd8e723"
lang: "es"
session_id: "d1eea2ea-3dcc-447c-8883-a23a89d9e570"
timestamp: "2019-06-09T19:32:53.334Z"
result {
  source: "agent"
  resolved_query: "Prueba"
  action: "input.unknown"
  score: 1.0
  parameters {
  }
  contexts {
    name: "generic"
    lifespan: 4
    parameters {
      fields {
        key: "facebook_sender_id"
        value {
          string_value: "2665762213453771"
        }
      }
    }
  }
  metadata {
    intent_id: "25d76718-8282-47ae-999e-55c40baba860"
    intent_name: "Default Fallback Intent"
    webhook_used: "false"
    webhook_for_slot_filling_used: "false"
    is_fallback_intent: "true"
  }
  fulfillment {
    speech: "Si es necesario puedes pedirme ayuda con solo mencionarlo."
    messages {
      lang: "es"
      type {
        number_value: 0.0
      }
      speech {
        string_value: "Si es necesario puedes pedirme ayuda con solo mencionarlo."
      }
    }
  }
}
status {
  code: 200
  error_type: "success"
}
"  
 timestamp:  "2019-06-09T19:32:53.416394429Z"  
 trace:  "d1eea2ea-3dcc-447c-8883-a23a89d9e570"  
}

Answer 1

When an application writes a log to Stackdriver, it can write either a line of text or a structured payload. It is up to the writing application on what it chooses to write. It appears that Dialog flow is writing a line of text of the form:

Dialogflow Request : <Rest of text>

where <Rest of text> appears to be a JSON string. If this is the case, it is arguable that Dialogflow isn't playing nicely with Stackdriver writing but there could be stories that we aren't familiar with. Either way, we have what we have.

What we now need to do is think about transforming this data into a form you can use. If we assume that all you need is the JSON representation of the Dialogflow request then what we would do is retrieve the records from Stackdriver and find the textPayload lines. If they start with Dialogflow Request : then we strip that off an parse the remainder of the line as JSON and we have our data.

There are many ways to do this and the choices will be based on how you want to process the data. Assuming you don't need real-time live parsing, then I would export the Stackdriver logs as GCS objects (files) and pass them through an ETL tool such as Datafusion (or your own hand-written parser). The resulting data could then be written to new GCS files and imported into BigQuery.

Should you need real-time processing, one though would be to write the Stackdriver log messages to PubSub and have a Cloud Function get each record as it is written. That function could parse our your data and write it to BigQuery.

Answer 2

I realize this post is almost 3 years old now, but I have not been able to find any easy option for cleaning up these logs and exporting to something more standard, and recently wrote my own tool.

Dialogflow Logs Fixer can now be found at https://gitlab.com/amsclark/dialogflow-logs-fixer under an open source license. There is a link to a video in the repo README that explains how to use the tool.