DATABRICKS-DEA Question 55

Community Discussion

Written by other learners. Comments can disagree with the marked answer — check the official documentation when in doubt.

• Chose A0

  - Correct answer : A — Increase `spark.sql.shuffle.partitions` so each shuffle partition holds a smaller amount of data and disk spill is reduced. A . Raising the shuffle partition count spreads a fixed volume of data across more, smaller partitions, so each task handles less data and spills less to local disk. This directly targets the observed multi-gigabyte-per-task spill pattern. B . A 2 TB dataset is far larger than any practical broadcast threshold, and forcing a broadcast of that size would exhaust executor memory rather than fix the spill. The broadcast threshold governs join planning, not aggregation shuffle sizing. C . `spark.default.parallelism` applies to RDD-based operations and does not override the SQL-specific shuffle partition setting for DataFrame or SQL aggregations. Raising it would not change the aggregation's shuffle partition count. D . Shuffle spill happens on the executors that hold the shuffle data, not on the driver, so increasing driver memory has no effect on executor-side spill during this aggregation.