Tag: Airflow

Resources about Airflow collected from the weekly GCP Weekly newsletter — a hand-curated collection of Google Cloud Platform news, technical articles, tutorials, videos and release notes. Each entry below links back to the original GCP Weekly issue and retains its publish date and title, giving you a focused, searchable knowledge base for this topic.

Airflow Cloud Composer Data Analytics Official Blog Sept. 21, 2026

The future of orchestration: Pine59’s journey to Airflow 3 on Google Cloud - Pine59 decided to modernize its monorepo to improve its MLOps capabilities, developer workflow, and pipeline speed.

Agents Airflow Data Analytics Official Blog Sept. 7, 2026

From weeks to minutes: The new agentic era of data pipelines - Build, deploy, and troubleshoot Airflow pipelines in minutes using Google Cloud Data Agent Kit and natural language.

Airflow Cloud Composer Paywall June 29, 2026

Optimising Airflow: Cutting DAG parse time even more through configuration — a practical guide (Part 2) - This guide details how to optimize Airflow DAG parsing times in large environments by tuning scheduler-level configurations, building on code-level improvements. It explains specific settings like `parsing_processes` and `min_file_process_interval` to enhance scheduler responsiveness and overall platform health. The article also addresses the standalone DAG processor in Airflow 3.0, the use of `.airflowignore`, and the role of underlying infrastructure in performance.

Airflow BigQuery Cloud Storage June 22, 2026

Backfilling 30 Million Files from GCS to BigQuery - This article details the complex journey of backfilling 30 million files from Google Cloud Storage into BigQuery, highlighting challenges often overlooked in data engineering. It describes the iterative process of building a robust pipeline, overcoming issues like API rate limits, data corruption, and system interruptions through strategies like idempotency and separating file discovery from ingestion.

Airflow BigQuery Paywall June 8, 2026

Designing a Medallion Architecture Pipeline for Spotify Listening Analytics - From Spotify API ingestion to BigQuery transformations and genre-level aggregation with Airflow orchestration.

Airflow Vertex AI Search March 9, 2026

Escaping Airflow Migration Chaos: Building an AI-Powered Autonomous Migration Agent - The Jira ticket had been sitting in the “Backlog” column for six months, staring at me like a unexploded ordnance.

Airflow Cloud Composer Data Analytics Official Blog Streaming Jan. 26, 2026

Improving workflow orchestration with Apache Airflow 3.1 in Cloud Composer - For Cloud Composer users, new features in Apache Airflow 3.1 are designed to increase oversight, improve reliability, and support global teams.

Airflow Cloud Composer Paywall Oct. 13, 2025

When “Scale Up” Just Scales the Pain: Debugging GCP Apache Meltdowns - The article provides a practical guide to debugging Apache meltdowns on Google Cloud Platform instead of just scaling up resources.

Airflow Cloud Functions Oct. 6, 2025

Orchestrating Google Cloud Functions with Apache Airflow - Exploring 3 Airflow Operators to Invoke Cloud Functions via HTTP, with Examples.

Airflow BigQuery dbt June 16, 2025

Advanced Security Strategies for Data Pipelines with Airflow, dbt, and BigQuery on GCP - Protect your data pipelines beyond access controls: learn deep security techniques for Airflow, dbt, and BigQuery in a modern GCP stack.

Airflow Cloud Composer GKE June 9, 2025

Apache Airflow on Helm in GKE + Cloud Functions Integration - This article explains how to deploy Apache Airflow on Google Kubernetes Engine (GKE) using Helm, offering more control and flexibility compared to Google Cloud Composer.

Airflow Cloud Composer Security June 2, 2025

Runtime Security in Cloud Composer: Enforcing Per-App DAG Isolation with External Policies - In a multi-tenant Cloud Composer environment, the author proposes enforcing per-DAG security policies at runtime to isolate applications. This is achieved by tagging DAGs, defining allowed operators and GCP connections in a JSON policy file, and using Airflow's `task_policy` hook to dynamically enforce these policies, enhancing security and manageability.

Airflow BigQuery Docker May 12, 2025

Fix: “Compute Engine Metadata server unavailable” in Airflow Docker for GCP Authentication - Running Airflow in Docker locally? Trying to reach BigQuery and getting errors? You’re not alone. Here’s how to solve it.

Airflow Cloud Run dbt April 28, 2025

Execute DBT with Airflow and Cloud Run - This article explains how to trigger a DBT pipeline in Cloud Run directly from Apache Airflow and Cloud Composer.

Airflow Cloud Composer Feb. 17, 2025

Expose Cloud Composer database to Looker Studio - In this article, we'll show you how to expose your Cloud Composer database to Looker Studio. We'll create a proxy VM with an external IP and run a TCP proxy to forward traffic to the database. We'll also set up a HAProxy config file and copy it to the compute instance. Finally, we'll set up the connection in Looker Studio and verify that the Airflow tables appear.

Airflow Cloud Composer Cloud Monitoring Feb. 10, 2025

XCom in Cloud Composer 2: A Blessing and a Curse for Airflow Data Sharing - XCom, a feature in Apache Airflow and Google Cloud Composer, enables tasks within a DAG to exchange small amounts of data. While useful for sharing metadata or small result sets, its misuse can lead to environment update/upgrade failures and snapshot issues.

Airflow BigQuery Billing dbt FinOps Nov. 25, 2024

Monitoring BigQuery Costs at Plum: A Detailed Breakdown - The article explores monitoring and managing BigQuery costs by analyzing metadata from tools like dbt, Apache Airflow, Looker, and user-generated queries.

Airflow GKE Kubernetes Oct. 28, 2024

Spark on GKE: A Guide to using GKEStartPodOperator for Spark workloads - Learn how to efficiently run your Spark applications on Google Kubernetes Engine using the GKEStartPodOperator from the Google Kubernetes Engine Operators for Apache Airflow.

Airflow Cloud Composer Oct. 7, 2024

Cloud Composer 3: Truly “serverless”? - An overview of Cloud Composer third generation.

Airflow Cloud Composer Data Analytics Official Blog Sept. 23, 2024

Apache Airflow ETL in Google Cloud - Apache Airflow is a popular choice for running complex tasks like ETL or data analytics pipelines. There are three different ways to run Apache Airflow on Google Cloud: Compute Engine, GKE Autopilot, and Cloud Composer. Each approach has its own advantages and disadvantages in terms of cost, performance, and availability.

Airflow BigQuery dbt Aug. 26, 2024

Dagster: A complete replacement for dbt Cloud automations - Dagster is a complete replacement for dbt Cloud automation. Combined with BigQuery, it offers cost-effective automation and enhanced features compared to dbt Cloud.

Airflow Cloud Composer Data Analytics Official Blog Streaming Aug. 26, 2024

Scalable alerting for Apache Airflow to improve data orchestration reliability and performance - This guide reviews the hierarchy of alerting on Cloud Composer and various alerting options available to Google Cloud engineers using Cloud Composer and Apache Airflow.

Airflow Cloud Composer Data Analytics Official Blog Streaming Aug. 12, 2024

Announcing Apache Airflow operators for Google generative AI - Apache Airflow now has operators to interact with Vertex AI's generative models. These operators enable the integration of Vertex AI's generative models into data pipelines orchestrated by Apache Airflow and Cloud Composer.

Airflow Cloud Composer Aug. 5, 2024

Logging new Airflow DAG entires in Cloud Composer - DAG Upload Audit.

Airflow Cloud Composer Data Analytics Official Blog Streaming July 29, 2024

Understanding Airflow DAG and task concurrency on Google Cloud Composer - Airflow DAG and task concurrency are crucial for optimizing Cloud Composer performance. This guide provides comprehensive insights into concurrency settings across four levels: Composer environment, Airflow installation, DAG, and task. By understanding these settings, you can ensure efficient resource utilization, scalability, and fault tolerance in your data pipelines.

Airflow BigQuery dbt June 24, 2024

How to choose between dbt clone and dbt defer. And how we clone for all contributors. - This blog post discusses the challenges of using production data in development environments for dbt projects and explores two approaches offered by dbt to address these challenges: defer and clone.

Airflow June 3, 2024

Data platform from scratch on GCP - Solvimon's bespoke analytics experience.

Airflow Cloud Composer Docker April 29, 2024

Lessons in adopting Airflow - Booking.com’s AdTech team’s learnings in adopting Airflow on GCP Composer.

Airflow GKE Kubernetes Tutorial April 29, 2024

Airflow on GKE using Helm - A tutorial on deploying Apache Airflow (tested with 2.8.4) on Google Kubernetes Engine (GKE) using the official Helm chart.

Airflow Cloud Composer Feb. 26, 2024

Avoid Autopilot in Cloud Composer 2 - A simple way to run your Aiflow DAGs in a standard GKE cluster under Cloud Composer 2 to reduce costs.

Showing 1–30 of 93 resources

 

Latest Issues




Contact

Zdenko Hrček
Třebanická 183
Prague, Czech Republic
Phone: +420 777 283 075
Email: [email protected]