apache / airflow
Apache Airflow makes scheduled workflows inspectable, replayable, and operational
Airflow lets teams define workflows as Python DAGs, schedule task instances, track state in a metadata database, and inspect runs through its user interface and APIs. It is powerful for finite, dependency-driven batch work, but adopting the repository also means owning a scheduler, execution layer, database, provider set, and upgrade process.
WHAT TO KNOW FIRST
- Airflow is an orchestrator for batch-oriented workflows: it schedules and records tasks but should not become the place where large data payloads are passed between them.
- DAG code, provider packages, executor choice, metadata database, logs, secrets, and deployment topology form one operational system and must be versioned together.
- Retries and backfills are safe only when tasks are idempotent, time boundaries are understood, and external side effects have explicit deduplication rules.
01Airflow is a control plane for finite work
Apache Airflow's stated purpose is to programmatically author, schedule, and monitor workflows. A workflow is represented as a directed acyclic graph, or DAG, whose tasks and dependencies describe an order of work. The scheduler creates task instances for eligible runs, an execution layer runs them, the metadata database records state, and the web interface and APIs expose what happened. The platform coordinates work rather than replacing the systems that perform it.
That distinction prevents a common design mistake. A task can submit a warehouse query, invoke a service, transform a file, or start a container, but Airflow should not carry a large dataset through its metadata records. Small messages and references can communicate state between tasks. Durable payloads belong in databases, object stores, or other systems designed for them. Airflow becomes the inspectable map and history of the process.
02DAGs are Python, with scheduler consequences
DAG files are Python modules, which gives authors normal language tools for constructing tasks and dependencies. The TaskFlow API can express task relationships with decorated Python functions, while operators represent established kinds of work. Dynamic DAG generation and task mapping cover cases where structure depends on configuration or runtime collections. The flexibility is real, but the scheduler must repeatedly parse DAG definitions to understand them.
Top-level DAG code should therefore be deterministic, quick to import, and free of unnecessary network or database calls. Parsing is not a convenient hidden startup hook. Heavy discovery work can delay scheduling and make failures difficult to diagnose. Put runtime I/O inside tasks, use supported configuration and connection mechanisms, and keep generated identifiers stable. A code review should be able to explain the graph without executing unrelated business operations.
03Scheduling is organized around logical intervals
Airflow scheduling is easier to use once a run's logical time and data interval are separated from the wall-clock moment when a task happens to start. A scheduled run usually represents a period of data, and the scheduler creates it according to the timetable after the relevant interval. Templates and context let tasks address that interval explicitly. This is why a daily job should read the intended day's partition instead of asking the operating system for now.
Catchup and backfill features can create historical runs that were not previously completed. That is valuable for repairing a pipeline or applying new logic, and dangerous when a task sends email, charges an account, overwrites an unversioned object, or appends without deduplication. Define side effects by logical run identity. Test boundaries around daylight-saving changes, time zones, manual runs, and partial intervals before a schedule becomes production policy.
04Task design decides whether retries are safe
Airflow records attempts and can retry failures according to task configuration. A retry reruns code; it does not roll back an external service. Tasks should be idempotent where possible, writing to deterministic partitions, using transaction boundaries, or checking an operation key before creating a side effect. Keep tasks large enough to represent meaningful operational units but small enough that retrying one failure does not repeat an entire day's unrelated work.
Dependencies should describe data or control requirements rather than merely preserve the order of a legacy script. Explicit inputs, outputs, and ownership make the graph useful during an incident. Timeouts, pools, concurrency limits, and priority weights can protect shared systems, but they cannot repair a task that opens unlimited connections or ignores cancellation. Each operator must respect the capacity and failure behavior of the service it calls.
05Executors change where tasks actually run
Airflow separates scheduling from execution through executors. Local development can run tasks on one machine, while production deployments may use distributed workers or Kubernetes-based execution according to the versioned options and provider setup. Executor choice affects isolation, scaling, queue behavior, worker dependencies, logs, networking, and failure recovery. It is an architectural choice, not a performance toggle to postpone until launch.
Package the code and dependencies so a task sees the same environment wherever it lands. A scheduler successfully queuing work does not prove a worker can import the DAG's packages, reach a private endpoint, or fetch credentials. Test worker loss, duplicate delivery assumptions, log retrieval, and task termination. Capacity planning must cover both scheduler health and the external systems tasks can overwhelm when many historical runs become eligible at once.
06Providers keep integrations modular and versioned
The Airflow project publishes provider packages for integrations with databases, clouds, messaging systems, and other services. Separating providers from core allows integration releases to move on their own schedules. It also creates a compatibility matrix. A deployment is defined not only by the Airflow core version, but by its selected providers, executor-related packages, Python version, constraints, and any custom plugins.
Install only providers the deployment uses and maintain an inventory. An upgrade can change an operator default, hook behavior, dependency, or connection field even when DAG code remains unchanged. Read provider changelogs, run DAG import checks, and exercise representative external calls in staging. Custom operators should use public interfaces and narrow dependencies. Copying an internal helper from Airflow may save a few lines and create a brittle upgrade later.
07The metadata database is operational state
DAG runs, task instances, schedules, connections, variables, serialized definitions, and other platform state rely on Airflow's metadata database. Production deployments need a supported database, migrations, backups, monitoring, and access control. The UI depends on this state to tell the truth about the system. Treat database health and scheduler health as first-class service indicators rather than waiting for a delayed job to reveal a problem.
Logs may be local or remote depending on deployment. Confirm that an operator can reach the logs for completed, retried, and failed tasks after workers disappear. Define retention for metadata and logs, because unlimited history has cost while aggressive cleanup can weaken incident analysis. Secrets should come from supported connections or secret backends and should not be embedded in DAG source, task arguments displayed in the UI, or templates that reach logs.
08Operations and security are part of the product
A production Airflow installation exposes a web application and API, runs privileged orchestration code, stores credentials or references to them, and can reach valuable data systems. Configure authentication, authorization, network boundaries, encryption, secret backends, and audit practices appropriate to that role. Limit who can deploy DAGs, because Python workflow code can be executable infrastructure. Review plugins and providers with the same care as application dependencies.
Upgrades require database migrations and compatibility testing across the complete deployment. The project distinguishes official source releases under Apache Software Foundation policy from convenience packages and images. Use the versioned installation documentation and constraints, verify artifacts according to organizational policy, back up state, and rehearse rollback limits. A green unit test for DAG code is only one layer of upgrade evidence.
09Know when a smaller scheduler is better
Airflow carries substantial machinery because it solves substantial orchestration problems: dependency graphs, schedules, retries, backfills, state history, concurrency controls, integrations, and operator visibility. A handful of independent scripts may be clearer under a managed scheduler or operating-system timer. Adding Airflow too early can turn simple jobs into a platform migration and create more failure modes than the workflows originally had.
It is also not a streaming processor. A DAG can coordinate deployment or maintenance of a streaming system, but Airflow's model centers on finite task instances and runs. Choose it when the operational questions match its strengths: what should run, for which interval, after what dependency, with what retry history, and under whose control. If those questions matter daily, the repository offers a mature language and control plane for answering them.
A SENSIBLE FIRST HOUR
Start small enough to learn the repo
- Read the versioned installation guide and use the official constraints for the chosen Airflow and Python versions instead of an unconstrained pip install.
- Run a disposable local environment, open the UI, and inspect one example DAG's graph, scheduled run, task instances, logs, retries, and clear-state behavior.
- Write a small DAG with two idempotent tasks and a declared data dependency, then test manual runs, a failure, a retry, and a backfill boundary.
- Before production, select an executor and metadata database, inventory provider packages, configure secrets and remote logs, and rehearse backup and upgrade procedures.
SOURCE LEDGER
What this review is built on
We use the project repository and first-party documentation. Access, licenses and project direction can change, so recheck the linked source before making a production decision.