Soundbarrier Blog

Personal blog about hardware, software, data and other interesting things.

Airflow Time-spans

by
Sebastian Schaetz
written
A common gotcha when working with airflow is to determine when a DAG is going to execute, and what the execution date-time is going to be when it executes. Especially if a DAG relies on processing data for a given execution date-time only to ensure idempotency. The best way to think about this is not in dates or timestamps but time-spans. An Airflow DAG run is going to cover a times-span - i.e. it is going to process data that was generated/received/added etc. in a given time-span. Continue reading..

Companions of Progress

by
Sebastian Schaetz
written
During the emergence of electronic communication, the two scientists/engineers Carl Friedrich Gauss and Wilhelm Eduard Weber built the first electrical communications line in Goettingen, Germany. They set up a two kilometer two wire copper line between Gauss' workplace – the Astronomical Observatory – and Weber’s laboratory in the physics department of the Goettingen University. Continue reading..

More posts can be found in the archive.