81 real Spark SQL & Catalyst questions from the Big Data bank, as asked in Indian campus drives and tech interviews. Every question has a verified answer and an AI-tutor explanation on placd — free to start.
1. What is Whole-stage code generation?
Junior
A.a Catalyst optimization that compiles a chain of operators in a stage into a single fused Java function to cut virtual-call and iterator overhead
B.the Catalyst component (spark.sql.cbo.enabled) that reorders joins and picks physical operators using collected table and column statistics
C.a Spark table layout that hash-distributes rows into a fixed number of files per column so joins and aggregations on that column can skip the shuffle
D.an optimization that pushes filter conditions down to the data source so fewer rows or files are read
2. Which term means: "a Catalyst optimization that compiles a chain of operators in a stage into a single fused Java function to cut virtual-call and iterator overhead"?
A.Whole-stage code generation — the Catalyst component (spark.sql.cbo.enabled) that reorders joins and picks physical operators using collected table and column statistics
B.Whole-stage code generation — a Catalyst optimization that compiles a chain of operators in a stage into a single fused Java function to cut virtual-call and iterator overhead
C.Whole-stage code generation — a Common Table Expression, a named temporary result defined with WITH that improves readability and can be referenced multiple times in a query
D.Whole-stage code generation — a Spark SQL hint (/*+ BROADCAST(t) */) forcing the optimizer to broadcast a small table to all executors for a map-side join
A.the tree of relational operators Spark builds and rewrites through analysis and optimization rules before producing physical plans
B.an optimization that pushes filter conditions down to the data source so fewer rows or files are read
C.the resolved tree of relational operators produced before physical planning, on which rule-based optimizations are applied
D.the ROWS or RANGE clause defining which rows around the current row a window function aggregates, such as ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
A.Logical plan — a Spark SQL hint (/*+ BROADCAST(t) */) forcing the optimizer to broadcast a small table to all executors for a map-side join
B.Logical plan — collected row counts and column histograms (gathered via ANALYZE TABLE) that the cost-based optimizer uses to choose join order and strategy
C.Logical plan — the tree of relational operators Spark builds and rewrites through analysis and optimization rules before producing physical plans
D.Logical plan — the resolved tree of relational operators produced before physical planning, on which rule-based optimizations are applied
A.the executable plan of physical operators (chosen join strategies, exchanges) selected by Catalyst using cost and rules
B.the Spark SQL mode (default in Spark 4) that enforces standard SQL semantics, throwing errors on overflow, divide-by-zero, and invalid casts instead of silently returning null
C.an optimization that pushes filter conditions down to the data source so fewer rows or files are read
D.the Catalyst technique that compiles a chain of operators into a single fused Java function to remove virtual-call and iterator overhead
A.Physical plan — the Spark SQL mode (default in Spark 4) that enforces standard SQL semantics, throwing errors on overflow, divide-by-zero, and invalid casts instead of silently returning null
B.Physical plan — the executable plan of physical operators (chosen join strategies, exchanges) selected by Catalyst using cost and rules
C.Physical plan — a Spark SQL hint (/*+ BROADCAST(t) */) forcing the optimizer to broadcast a small table to all executors for a map-side join
D.Physical plan — an optimization that pushes filter conditions down to the data source so fewer rows or files are read
A.the executable plan of physical operators (chosen join strategies, exchanges) selected by Catalyst using cost and rules
B.an optimization that pushes filter conditions down to the data source so fewer rows or files are read
C.a Spark SQL hint (/*+ BROADCAST(t) */) forcing the optimizer to broadcast a small table to all executors for a map-side join
D.a SQL construct computing a value over a frame of rows related to the current row (using OVER with PARTITION BY and ORDER BY) without collapsing them into a group
A.Predicate pushdown — a Common Table Expression that references itself to traverse hierarchies, supported in Spark SQL from version 3.5 onward
B.Predicate pushdown — the Parquet and ORC reader path in Spark that decodes batches of column values at once into in-memory column vectors for cache-friendly processing
C.Predicate pushdown — an optimization that pushes filter conditions down to the data source so fewer rows or files are read
D.Predicate pushdown — an optimization that reads only the columns referenced by a query, especially effective on columnar formats like Parquet
A.an AQE feature that splits oversized skewed partitions into smaller sub-partitions so one task does not dominate a join
B.an optimization that reads only the columns referenced by a query, especially effective on columnar formats like Parquet
C.an AQE-related optimization that prunes fact-table partitions at runtime using the filtered keys of a joined dimension table
D.the ROWS or RANGE clause defining which rows around the current row a window function aggregates, such as ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
A.Column pruning — a SQL construct computing a value over a frame of rows related to the current row (using OVER with PARTITION BY and ORDER BY) without collapsing them into a group
B.Column pruning — a Catalyst optimization that compiles a chain of operators in a stage into a single fused Java function to cut virtual-call and iterator overhead
C.Column pruning — an optimization that reads only the columns referenced by a query, especially effective on columnar formats like Parquet
D.Column pruning — the Catalyst technique that compiles a chain of operators into a single fused Java function to remove virtual-call and iterator overhead
A.a Catalyst optimization that compiles a chain of operators in a stage into a single fused Java function to cut virtual-call and iterator overhead
B.the Spark SQL mode (default in Spark 4) that enforces standard SQL semantics, throwing errors on overflow, divide-by-zero, and invalid casts instead of silently returning null
C.an AQE-related optimization that prunes fact-table partitions at runtime using the filtered keys of a joined dimension table
D.a Catalyst mode that uses table and column statistics to pick join order and join strategies
A.Dynamic partition pruning — the tree of relational operators Spark builds and rewrites through analysis and optimization rules before producing physical plans
B.Dynamic partition pruning — an AQE-related optimization that prunes fact-table partitions at runtime using the filtered keys of a joined dimension table
C.Dynamic partition pruning — an optimization that reads only the columns referenced by a query, especially effective on columnar formats like Parquet
D.Dynamic partition pruning — the Catalyst component (spark.sql.cbo.enabled) that reorders joins and picks physical operators using collected table and column statistics
A.a Common Table Expression, a named temporary result defined with WITH that improves readability and can be referenced multiple times in a query
B.the ROWS or RANGE clause defining which rows around the current row a window function aggregates, such as ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
C.a Catalyst mode that uses table and column statistics to pick join order and join strategies
D.the resolved tree of relational operators produced before physical planning, on which rule-based optimizations are applied
A.Cost-based optimization — a Catalyst mode that uses table and column statistics to pick join order and join strategies
B.Cost-based optimization — an AQE-related optimization that prunes fact-table partitions at runtime using the filtered keys of a joined dimension table
C.Cost-based optimization — a SQL construct computing a value over a frame of rows related to the current row (using OVER with PARTITION BY and ORDER BY) without collapsing them into a group
D.Cost-based optimization — a Common Table Expression that references itself to traverse hierarchies, supported in Spark SQL from version 3.5 onward
A.the Catalyst technique that compiles a chain of operators into a single fused Java function to remove virtual-call and iterator overhead
B.an AQE-related optimization that prunes fact-table partitions at runtime using the filtered keys of a joined dimension table
C.an AQE feature that splits oversized skewed partitions into smaller sub-partitions so one task does not dominate a join
D.a SQL construct computing a value over a frame of rows related to the current row (using OVER with PARTITION BY and ORDER BY) without collapsing them into a group
A.Skew join optimization — the tree of relational operators Spark builds and rewrites through analysis and optimization rules before producing physical plans
B.Skew join optimization — an AQE feature that splits oversized skewed partitions into smaller sub-partitions so one task does not dominate a join
C.Skew join optimization — a Common Table Expression, a named temporary result defined with WITH that improves readability and can be referenced multiple times in a query
D.Skew join optimization — the executable plan of physical operators (chosen join strategies, exchanges) selected by Catalyst using cost and rules
A.ANALYZE TABLE t COMPUTE STATISTICS — collects table and column statistics so the cost-based optimizer can choose better plans
B.ANALYZE TABLE t COMPUTE STATISTICS — the command that prints Spark's parsed, analyzed, optimized, and physical plans in a structured layout for tuning
C.ANALYZE TABLE t COMPUTE STATISTICS — marks a DataFrame to be stored in memory on first action using the default MEMORY_AND_DISK storage level
D.ANALYZE TABLE t COMPUTE STATISTICS — prints the parsed, analyzed, optimized, and physical plans for a query in a structured layout
Showing 30 of 81 Spark SQL & Catalyst questions — the full set, with answers, explanations and an AI tutor on every question, is inside.
Free to start
Answers, AI explanations, and a scored voice mock interview
Sign up free to check your answers with explanations, ask the AI tutor anything on any question, and take one full AI mock interview — scored like a real panel.