Databricks Updated Databricks-Certified-Data-Engineer-Associate Exam Questions and Answers by millie-mae

The small country lookup DataFrame should be broadcast to the executors. A broadcast join copies the small side of the join to each worker, allowing every executor to join its partition of the billion-row events DataFrame locally. This avoids shuffling the very large DataFrame across the network and is effective when one relation is small enough to fit comfortably in executor memory. Broadcasting events_df would be impractical because it contains billions of rows. Caching or persisting both DataFrames can help when data is reused, but neither action by itself changes the join strategy or removes the expensive shuffle. Databricks exposes the PySpark broadcast function specifically to mark a DataFrame as suitable for a broadcast join. Therefore, option D is the correct optimization.

================

DataFrame.dropna() removes rows containing NULL or NaN values. Its default how= " any " behavior removes a row when any column included in the specified subset is NULL. By passing [ " customer_email " , " customer_phone " ] as the subset, the operation removes a record when either required field is missing, while ignoring NULL values in unrelated columns. Option B can produce the same filtering result, but it uses two chained where() calls and therefore does not meet the single-method-call requirement. Option C uses how= " all " and would remove a row only when all evaluated columns are NULL. Option D contains one filter() operation but requires multiple column-function calls. Consequently, option A is the direct PySpark method designed for this requirement.

================