Spark Tuning interview questions

1. A.it spills to disk, serializes data and transfers it over the network
2. B.the garbage collector recommended for large Spark executor heaps, dividing the heap into regions to keep pause times bounded better than the older parallel collector
3. C.adding a random prefix to a hot key to spread skew across partitions
4. D.an uneven key distribution that makes one task a straggler

Answer + AI explanation with Pro

1. A.why a shuffle is expensive
2. B.spark.memory.fraction
3. C.shuffle tracking
4. D.Garbage collection tuning

Answer + AI explanation with Pro

1. A.why a shuffle is expensive — an AQE feature that merges many small shuffle partitions into fewer right-sized ones based on runtime statistics
2. B.why a shuffle is expensive — a task that runs much slower than its peers in a stage and delays the whole stage from completing
3. C.why a shuffle is expensive — it spills to disk, serializes data and transfers it over the network
4. D.why a shuffle is expensive — writing an RDD or DataFrame to reliable storage to truncate its lineage, preventing stack-overflow and recomputation in long iterative or streaming jobs

Answer + AI explanation with Pro

1. A.the mechanism (spark.dynamicAllocation.shuffleTracking.enabled) that lets dynamic allocation release executors safely without a separate external shuffle service by tracking which executors hold shuffle blocks
2. B.the act of writing shuffle or aggregation data from memory to disk when it exceeds the available execution memory, slowing the job
3. C.the config capping the number of bytes packed into a single partition when reading files, defaulting to 128 MB and controlling read-side parallelism
4. D.the setting controlling post-shuffle partition count (default 200)

Answer + AI explanation with Pro

1. A.spark.sql.shuffle.partitions
2. B.Garbage collection tuning
3. C.dynamic allocation
4. D.why a shuffle is expensive

Answer + AI explanation with Pro

1. A.spark.sql.shuffle.partitions — the AQE feature that detects skewed join partitions at runtime and splits them into smaller subpartitions to balance task time
2. B.spark.sql.shuffle.partitions — the config that sets how many concurrent task slots each Spark executor gets, commonly tuned to 4-5 to balance HDFS throughput against memory pressure
3. C.spark.sql.shuffle.partitions — the config capping the number of bytes packed into a single partition when reading files, defaulting to 128 MB and controlling read-side parallelism
4. D.spark.sql.shuffle.partitions — the setting controlling post-shuffle partition count (default 200)

Answer + AI explanation with Pro

1. A.an uneven key distribution that makes one task a straggler
2. B.the garbage collector recommended for large Spark executor heaps, dividing the heap into regions to keep pause times bounded better than the older parallel collector
3. C.a skew-mitigation technique that appends a random prefix to hot keys to spread them across more reducers before aggregating
4. D.writing an RDD or DataFrame to reliable storage to truncate its lineage, preventing stack-overflow and recomputation in long iterative or streaming jobs

Answer + AI explanation with Pro

1. A.spark.executor.memory
2. B.dynamic allocation
3. C.partition sizing
4. D.data skew

Answer + AI explanation with Pro

1. A.data skew — an uneven key distribution that makes one task a straggler
2. B.data skew — it spills to disk, serializes data and transfers it over the network
3. C.data skew — the AQE setting that merges small post-shuffle partitions at runtime so the output partition count matches the actual data volume
4. D.data skew — adding a random prefix to a hot key to spread skew across partitions

Answer + AI explanation with Pro

1. A.an uneven distribution of keys across partitions that overloads a few tasks and creates straggler-driven slowness
2. B.the act of writing shuffle or aggregation data from memory to disk when it exceeds the available execution memory, slowing the job
3. C.adding a random prefix to a hot key to spread skew across partitions
4. D.adjusting JVM collector and heap settings to reduce long GC pauses that stall Spark executors

Answer + AI explanation with Pro

1. A.salting
2. B.checkpointing
3. C.why a shuffle is expensive
4. D.spark.executor.memoryOverhead

Answer + AI explanation with Pro

1. A.salting — adding a random prefix to a hot key to spread skew across partitions
2. B.salting — the config that sets the JVM heap size per Spark executor, with task working memory drawn from it after reserving overhead and storage
3. C.salting — a task that runs much slower than its peers in a stage and delays the whole stage from completing
4. D.salting — an AQE feature that merges many small shuffle partitions into fewer right-sized ones based on runtime statistics

Answer + AI explanation with Pro

1. A.the size limit (default 10MB) under which Spark auto-broadcasts a table
2. B.the AQE feature that detects skewed join partitions at runtime and splits them into smaller subpartitions to balance task time
3. C.a performance issue where too many tiny output files create excessive task and metadata overhead on reads
4. D.the AQE setting that merges small post-shuffle partitions at runtime so the output partition count matches the actual data volume

Answer + AI explanation with Pro

1. A.spark.sql.files.maxPartitionBytes
2. B.autoBroadcastJoinThreshold
3. C.Bucketed join
4. D.Salting

Answer + AI explanation with Pro

1. A.autoBroadcastJoinThreshold — the garbage collector recommended for large Spark executor heaps, dividing the heap into regions to keep pause times bounded better than the older parallel collector
2. B.autoBroadcastJoinThreshold — the LRU process by which Spark drops the least-recently-used cached blocks from storage memory when space is needed, with MEMORY_AND_DISK spilling rather than recomputing
3. C.autoBroadcastJoinThreshold — the practice of keeping each Spark partition near 128-200 MB so tasks are neither too small (scheduler overhead) nor too large (spilling and skew)
4. D.autoBroadcastJoinThreshold — the size limit (default 10MB) under which Spark auto-broadcasts a table

Answer + AI explanation with Pro

1. A.memory managed outside the JVM heap by Spark (enabled with spark.memory.offHeap.enabled and sized by spark.memory.offHeap.size) to reduce garbage collection pressure on large workloads
2. B.the AQE feature that detects skewed join partitions at runtime and splits them into smaller subpartitions to balance task time
3. C.an uneven distribution of keys across partitions that overloads a few tasks and creates straggler-driven slowness
4. D.a task that runs much slower than its peers in a stage and delays the whole stage from completing

Answer + AI explanation with Pro

1. A.localCheckpoint
2. B.Data skew
3. C.checkpointing
4. D.data skew

Answer + AI explanation with Pro

1. A.Data skew — an uneven distribution of keys across partitions that overloads a few tasks and creates straggler-driven slowness
2. B.Data skew — the config reserving off-heap memory per executor for JVM internals, shuffle buffers, and Python/native processes, defaulting to about 10 percent of executor memory
3. C.Data skew — a skew-mitigation technique that appends a random prefix to hot keys to spread them across more reducers before aggregating
4. D.Data skew — a faster Spark checkpoint variant that truncates lineage using executor local storage instead of reliable storage, trading fault tolerance for speed

Answer + AI explanation with Pro

1. A.a skew-mitigation technique that appends a random prefix to hot keys to spread them across more reducers before aggregating
2. B.a shuffle-free join made possible when both tables are pre-bucketed on the join key with the same number of buckets
3. C.the fraction of usable heap shared by execution and storage in Spark unified memory management, defaulting to 0.6
4. D.an uneven key distribution that makes one task a straggler

Answer + AI explanation with Pro

1. A.Spill
2. B.shuffle tracking
3. C.Bucketed join
4. D.Salting

Answer + AI explanation with Pro

1. A.Salting — a skew-mitigation technique that appends a random prefix to hot keys to spread them across more reducers before aggregating
2. B.Salting — the garbage collector recommended for large Spark executor heaps, dividing the heap into regions to keep pause times bounded better than the older parallel collector
3. C.Salting — an uneven key distribution that makes one task a straggler
4. D.Salting — writing an RDD or DataFrame to reliable storage to truncate its lineage, preventing stack-overflow and recomputation in long iterative or streaming jobs

Answer + AI explanation with Pro

1. A.an uneven key distribution that makes one task a straggler
2. B.the act of writing shuffle or aggregation data from memory to disk when it exceeds the available execution memory, slowing the job
3. C.the LRU process by which Spark drops the least-recently-used cached blocks from storage memory when space is needed, with MEMORY_AND_DISK spilling rather than recomputing
4. D.the config that sets how many concurrent task slots each Spark executor gets, commonly tuned to 4-5 to balance HDFS throughput against memory pressure

Answer + AI explanation with Pro

1. A.salting
2. B.off-heap memory
3. C.Spill
4. D.data skew

Answer + AI explanation with Pro

1. A.Spill — adding a random prefix to a hot key to spread skew across partitions
2. B.Spill — the Spark feature (spark.speculation) that relaunches copies of slow-running stragglers on other executors and keeps whichever finishes first to limit tail latency
3. C.Spill — a faster Spark checkpoint variant that truncates lineage using executor local storage instead of reliable storage, trading fault tolerance for speed
4. D.Spill — the act of writing shuffle or aggregation data from memory to disk when it exceeds the available execution memory, slowing the job

Answer + AI explanation with Pro

1. A.a task that runs much slower than its peers in a stage and delays the whole stage from completing
2. B.an AQE feature that merges many small shuffle partitions into fewer right-sized ones based on runtime statistics
3. C.the Spark feature (spark.dynamicAllocation.enabled) that scales executors up and down at runtime based on pending and idle tasks, requiring an external or shuffle-tracking service to preserve shuffle data
4. D.a faster Spark checkpoint variant that truncates lineage using executor local storage instead of reliable storage, trading fault tolerance for speed

Answer + AI explanation with Pro

1. A.localCheckpoint
2. B.Salting
3. C.Coalescing post-shuffle partitions
4. D.speculative execution

Answer + AI explanation with Pro

1. A.Coalescing post-shuffle partitions — an AQE feature that merges many small shuffle partitions into fewer right-sized ones based on runtime statistics
2. B.Coalescing post-shuffle partitions — the size limit (default 10MB) under which Spark auto-broadcasts a table
3. C.Coalescing post-shuffle partitions — the practice of keeping each Spark partition near 128-200 MB so tasks are neither too small (scheduler overhead) nor too large (spilling and skew)
4. D.Coalescing post-shuffle partitions — a faster Spark checkpoint variant that truncates lineage using executor local storage instead of reliable storage, trading fault tolerance for speed

Answer + AI explanation with Pro

1. A.an uneven distribution of keys across partitions that overloads a few tasks and creates straggler-driven slowness
2. B.the config reserving off-heap memory per executor for JVM internals, shuffle buffers, and Python/native processes, defaulting to about 10 percent of executor memory
3. C.the config that sets how many concurrent task slots each Spark executor gets, commonly tuned to 4-5 to balance HDFS throughput against memory pressure
4. D.a performance issue where too many tiny output files create excessive task and metadata overhead on reads

Answer + AI explanation with Pro

1. A.Salting
2. B.Bucketed join
3. C.Small files problem
4. D.dynamic allocation

Answer + AI explanation with Pro

1. A.Small files problem — a performance issue where too many tiny output files create excessive task and metadata overhead on reads
2. B.Small files problem — the mechanism (spark.dynamicAllocation.shuffleTracking.enabled) that lets dynamic allocation release executors safely without a separate external shuffle service by tracking which executors hold shuffle blocks
3. C.Small files problem — adding a random prefix to a hot key to spread skew across partitions
4. D.Small files problem — it spills to disk, serializes data and transfers it over the network

Answer + AI explanation with Pro