Over the last 3 years I read and listened to a number of books. Some of them stood out to me for one reason or another. I am listing those books here. The list serves the purpose of reminding myself of the interesting tidbits in the book, why I enjoyed it and why I might want to read it/listen to it again. I tried to categorize them but I don't think that is particularly helpful - it is also hard. The box are all fiction books and a mix of history, technology, science, aviation, memoir and biography. Continue reading..
At a high level backfilling in Airflow is a mechanism that allows (re-)execution of Airflow DAG instances for a specific time interval. It is a useful but inherently fragile and brittle process in Airflow. Continue reading..
If you ever wrote software or designed hardware in a regulated industry, it is very possible that you interfaced within a Quality Management System (QMS). I personally have seen a couple of different incarnations of such a system: from paper-based with wet-ink signatures to fully database driven and smart-card sign-off. Especially at smaller companies, paper-based systems are common. Templates, checklist, manuals, SOPs and protocols are created often as Word documents, are edited, sent, printed, signed, revised, edited again, etc. Continue reading..
A common gotcha when working with airflow is to determine when a DAG is going to execute, and what the execution date-time is going to be when it executes. Especially if a DAG relies on processing data for a given execution date-time only to ensure idempotency. The best way to think about this is not in dates or timestamps but time-spans. An Airflow DAG run is going to cover a times-span - i.e. it is going to process data that was generated/received/added etc. in a given time-span. Continue reading..
During the emergence of electronic communication, the two scientists/engineers Carl Friedrich Gauss and Wilhelm Eduard Weber built the first electrical communications line in Goettingen, Germany. They set up a two kilometer two wire copper line between Gauss' workplace – the Astronomical Observatory – and Weber’s laboratory in the physics department of the Goettingen University. Continue reading..
More posts can be found in the archive.