Convert airflow macro 'ts' into datetime object

Question

I encountered a problem with converting the airflow macros reference 'ts' into a datetime object. The problem is with the tz at the end of the string.

from datetime import datetime
timetoday = kwargs['ts']
t = datetime.strptime(timetoday, '%Y-%m-%dT%H:%M:%S%z')

The code didn't execute successfully, instead it throws an error message:

time data '2019-08-24T00:00:00+00:00' does not match format '%Y-%m-%dT%H:%M:%S%z'

If I used the same code with slightly different format, and try with 'ds' macros, and I managed to convert the string into datetime object.

from datetime import datetime
timetoday = kwargs['ds']
t = datetime.strptime(timetoday, '%Y-%m-%d')

Update: Reading the notes from http://strftime.org/ and understand that %z is UTC offset in the form +HHMM or -HHMM (empty string if the the object is naive). This led me wondering whether the additional ':' in between +00:00 (returned string of kwargs['ts'] is the root cause of the error. However, I'm still unsure how to parse it correctly, as there were no other options in strptime.

Therefore, I change my macro to kwargs['ts_nodash'] that doesn't return the timezone, and proceed with my code for now. If anyone knows how to do it, I'm still interested to learn how that can be done correctly! Thanks!

timetoday = kwargs['ts_nodash']
t = datetime.strptime(timetoday, '%Y%m%dT%H%M%S')

Answer 1

Looking at the macros reference and then your use-case, I'd suggest you to use execution_date macro instead of ts macro

{{ ts }}            : same as execution_date.isoformat(). Example: 2018-01-01T00:00:00+00:00

{{ execution_date }}: the execution_date (pendulum.Pendulum)

---

If your'e OK with that, it all boils down to converting the execution_date which is a pendulum object into a Python datetime object, which is as simple as datetime.fromtimestamp(execution_date.timestamp())

Python snippet (conversion) demo:

import pendulum
from pendulum import Pendulum
from datetime import datetime
..
execution_date: Pendulum = pendulum.now()
execution_date_as_datetime: datetime = datetime.fromtimestamp(execution_date.timestamp())

---

References

• execution_date jinja resolving as a string
• How to convert pendulum to datetime.datetime type?

Answer 2

You can just do it the easy way in your dag if needed.

Example airflow output object {{ ts }} from dag

2022-04-27T01:00:00+00:00

Example code in your dag or script

from datetime import datetime as dt 
launch_dt = dt.fromisoformat(kwargs['ts'])
print(launch_dt)

2022-04-27 01:00:00+00:00