Orchestration interview questions

1. A.A Dagster abstraction where you declare the data assets a pipeline produces and their dependencies, and Dagster derives the execution graph.
2. B.A per-task rule (all_success, one_failed, all_done) controlling when a task fires based on the states of its upstream tasks.
3. C.The Prefect component that executes tasks, configurable for sequential, concurrent, or distributed (Dask, Ray) execution.
4. D.a directed acyclic graph of tasks defining execution order

Answer + AI explanation with Pro

1. A.pool
2. B.Airflow DAG
3. C.sensor
4. D.Prefect flow

Answer + AI explanation with Pro

1. A.Airflow DAG — A Dagster abstraction where you declare the data assets a pipeline produces and their dependencies, and Dagster derives the execution graph.
2. B.Airflow DAG — A Dagster component that polls external state and launches runs when conditions are met, the event-driven complement to schedules.
3. C.Airflow DAG — A single execution instance of a DAG for a given logical date or trigger, containing the task instances and their states.
4. D.Airflow DAG — a directed acyclic graph of tasks defining execution order

Answer + AI explanation with Pro

1. A.a template defining one task's work (PythonOperator, BashOperator, sensor)
2. B.Airflow's mechanism for passing small amounts of data between tasks
3. C.A Dagster abstraction where you declare the data assets a pipeline produces and their dependencies, and Dagster derives the execution graph.
4. D.The Prefect component that executes tasks, configurable for sequential, concurrent, or distributed (Dask, Ray) execution.

Answer + AI explanation with Pro

1. A.Airflow operator
2. B.TaskGroup
3. C.Prefect flow
4. D.pool

Answer + AI explanation with Pro

1. A.Airflow operator — A Dagster abstraction where you declare the data assets a pipeline produces and their dependencies, and Dagster derives the execution graph.
2. B.Airflow operator — A per-task rule (all_success, one_failed, all_done) controlling when a task fires based on the states of its upstream tasks.
3. C.Airflow operator — a template defining one task's work (PythonOperator, BashOperator, sensor)
4. D.Airflow operator — Designing pipelines around the data assets produced and consumed rather than around task ordering, the model behind Dagster assets and Airflow assets.

Answer + AI explanation with Pro

1. A.the component that scans DAGs and queues eligible tasks
2. B.A resilience pattern that retries a failed task after increasing delays, reducing load on a struggling dependency while improving success odds.
3. C.A Dagster abstraction where you declare the data assets a pipeline produces and their dependencies, and Dagster derives the execution graph.
4. D.an operator that waits for an external condition before succeeding

Answer + AI explanation with Pro

1. A.sensor poke vs reschedule
2. B.data-aware orchestration
3. C.sensor
4. D.retry with backoff

Answer + AI explanation with Pro

1. A.sensor — A Dagster abstraction where you declare the data assets a pipeline produces and their dependencies, and Dagster derives the execution graph.
2. B.sensor — a directed acyclic graph of tasks defining execution order
3. C.sensor — an operator that waits for an external condition before succeeding
4. D.sensor — A task designed so that rerunning it with the same inputs produces the same result without duplicating data, essential for safe retries and backfills.

Answer + AI explanation with Pro

1. A.a directed acyclic graph of tasks defining execution order
2. B.An Airflow concurrency-limiting mechanism that caps how many task instances tagged to it can run at once, protecting shared resources.
3. C.skipping the run of missed historical intervals when a DAG is first enabled
4. D.A resilience pattern that retries a failed task after increasing delays, reducing load on a struggling dependency while improving success odds.

Answer + AI explanation with Pro

1. A.Airflow operator
2. B.catchup=False
3. C.Airflow scheduler
4. D.Dagster sensor

Answer + AI explanation with Pro

1. A.catchup=False — Two sensor modes: poke holds a worker slot while checking, reschedule frees the slot between checks; deferrable triggers improve on both.
2. B.catchup=False — skipping the run of missed historical intervals when a DAG is first enabled
3. C.catchup=False — A per-task rule (all_success, one_failed, all_done) controlling when a task fires based on the states of its upstream tasks.
4. D.catchup=False — An Airflow construct that visually and logically groups related tasks inside a DAG, replacing SubDAGs without their scheduling drawbacks.

Answer + AI explanation with Pro

1. A.a directed acyclic graph of tasks defining execution order
2. B.Airflow's mechanism for passing small amounts of data between tasks
3. C.a template defining one task's work (PythonOperator, BashOperator, sensor)
4. D.A per-task rule (all_success, one_failed, all_done) controlling when a task fires based on the states of its upstream tasks.

Answer + AI explanation with Pro

1. A.software-defined asset
2. B.Prefect flow
3. C.XCom
4. D.retry with backoff

Answer + AI explanation with Pro

1. A.XCom — Designing pipelines around the data assets produced and consumed rather than around task ordering, the model behind Dagster assets and Airflow assets.
2. B.XCom — Airflow's mechanism for passing small amounts of data between tasks
3. C.XCom — The Prefect component that executes tasks, configurable for sequential, concurrent, or distributed (Dask, Ray) execution.
4. D.XCom — A per-task rule (all_success, one_failed, all_done) controlling when a task fires based on the states of its upstream tasks.

Answer + AI explanation with Pro

1. A.A Dagster component that polls external state and launches runs when conditions are met, the event-driven complement to schedules.
2. B.a directed acyclic graph of tasks defining execution order
3. C.the component that scans DAGs and queues eligible tasks
4. D.a template defining one task's work (PythonOperator, BashOperator, sensor)

Answer + AI explanation with Pro

1. A.Airflow operator
2. B.trigger rule
3. C.Airflow scheduler
4. D.DAG run

Answer + AI explanation with Pro

1. A.Airflow scheduler — A task designed so that rerunning it with the same inputs produces the same result without duplicating data, essential for safe retries and backfills.
2. B.Airflow scheduler — the component that scans DAGs and queues eligible tasks
3. C.Airflow scheduler — Airflow's mechanism for passing small amounts of data between tasks
4. D.Airflow scheduler — a template defining one task's work (PythonOperator, BashOperator, sensor)

Answer + AI explanation with Pro

1. A.Airflow's mechanism for passing small amounts of data between tasks
2. B.A task designed so that rerunning it with the same inputs produces the same result without duplicating data, essential for safe retries and backfills.
3. C.A resilience pattern that retries a failed task after increasing delays, reducing load on a struggling dependency while improving success odds.
4. D.An Airflow construct that visually and logically groups related tasks inside a DAG, replacing SubDAGs without their scheduling drawbacks.

Answer + AI explanation with Pro

1. A.idempotent task
2. B.Prefect flow
3. C.TaskGroup
4. D.Airflow scheduler

Answer + AI explanation with Pro

1. A.TaskGroup — A single execution instance of a DAG for a given logical date or trigger, containing the task instances and their states.
2. B.TaskGroup — An Airflow construct that visually and logically groups related tasks inside a DAG, replacing SubDAGs without their scheduling drawbacks.
3. C.TaskGroup — A task designed so that rerunning it with the same inputs produces the same result without duplicating data, essential for safe retries and backfills.
4. D.TaskGroup — the component that scans DAGs and queues eligible tasks

Answer + AI explanation with Pro

1. A.A Dagster abstraction where you declare the data assets a pipeline produces and their dependencies, and Dagster derives the execution graph.
2. B.A Dagster component that polls external state and launches runs when conditions are met, the event-driven complement to schedules.
3. C.An Airflow concurrency-limiting mechanism that caps how many task instances tagged to it can run at once, protecting shared resources.
4. D.Designing pipelines around the data assets produced and consumed rather than around task ordering, the model behind Dagster assets and Airflow assets.

Answer + AI explanation with Pro

1. A.software-defined asset
2. B.Dagster sensor
3. C.XCom
4. D.pool

Answer + AI explanation with Pro

1. A.pool — An Airflow construct that visually and logically groups related tasks inside a DAG, replacing SubDAGs without their scheduling drawbacks.
2. B.pool — A task designed so that rerunning it with the same inputs produces the same result without duplicating data, essential for safe retries and backfills.
3. C.pool — A Prefect-decorated Python function representing a workflow of tasks, with dynamic, native-Python control flow and observability.
4. D.pool — An Airflow concurrency-limiting mechanism that caps how many task instances tagged to it can run at once, protecting shared resources.

Answer + AI explanation with Pro

1. A.A Dagster abstraction where you declare the data assets a pipeline produces and their dependencies, and Dagster derives the execution graph.
2. B.An Airflow construct that visually and logically groups related tasks inside a DAG, replacing SubDAGs without their scheduling drawbacks.
3. C.the component that scans DAGs and queues eligible tasks
4. D.The Prefect component that executes tasks, configurable for sequential, concurrent, or distributed (Dask, Ray) execution.

Answer + AI explanation with Pro

1. A.data-aware orchestration
2. B.sensor
3. C.Prefect flow
4. D.software-defined asset

Answer + AI explanation with Pro

1. A.software-defined asset — An Airflow construct that visually and logically groups related tasks inside a DAG, replacing SubDAGs without their scheduling drawbacks.
2. B.software-defined asset — A Dagster abstraction where you declare the data assets a pipeline produces and their dependencies, and Dagster derives the execution graph.
3. C.software-defined asset — A task designed so that rerunning it with the same inputs produces the same result without duplicating data, essential for safe retries and backfills.
4. D.software-defined asset — A Dagster component that polls external state and launches runs when conditions are met, the event-driven complement to schedules.

Answer + AI explanation with Pro

1. A.An Airflow concurrency-limiting mechanism that caps how many task instances tagged to it can run at once, protecting shared resources.
2. B.skipping the run of missed historical intervals when a DAG is first enabled
3. C.A Prefect-decorated Python function representing a workflow of tasks, with dynamic, native-Python control flow and observability.
4. D.The Prefect component that executes tasks, configurable for sequential, concurrent, or distributed (Dask, Ray) execution.

Answer + AI explanation with Pro

1. A.DAG run
2. B.XCom
3. C.Prefect flow
4. D.catchup=False

Answer + AI explanation with Pro

1. A.Prefect flow — A resilience pattern that retries a failed task after increasing delays, reducing load on a struggling dependency while improving success odds.
2. B.Prefect flow — A Prefect-decorated Python function representing a workflow of tasks, with dynamic, native-Python control flow and observability.
3. C.Prefect flow — An Airflow concurrency-limiting mechanism that caps how many task instances tagged to it can run at once, protecting shared resources.
4. D.Prefect flow — Two sensor modes: poke holds a worker slot while checking, reschedule frees the slot between checks; deferrable triggers improve on both.

Answer + AI explanation with Pro