Databricks Updated Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Questions and Answers by remy

na.drop(how='any') drops any row that has at least one null value.

This is exactly what's needed when the goal is to retain only fully complete records.

Usage:CopyEdit

filtered_df = users_raw_df.na.drop(how='any')

Explanation of incorrect options:

A: thresh=0 is invalid — thresh must be ≥ 1.

B: how='all' drops only rows where all columns are null (too lenient).

D: spark.na.drop doesn't support mixing how and thresh in that way; it's incorrect syntax.

RDD actions like reduce() aggregate values across all partitions and return the result to the driver.

To compute the maximum processing time, reduce() is ideal because it combines results from all tasks efficiently.

Example:

max_time = rdd_times.reduce(lambda x, y: max(x, y))

This aggregates maximum values from all executors into a single result on the driver.

Why the other options are incorrect:

A: Broadcast variables distribute read-only data; they cannot aggregate results.

B: Spark UI provides visualization, not programmatic collection.

D: Accumulators support additive operations only (e.g., counters, sums), not non-associative ones like max.

In PySpark, when performing a join between two DataFrames, the columns do not have to share the same name. You can explicitly provide a join condition by comparing specific columns from each DataFrame.

This syntax is correct and fully supported:

df1.join(df2, df1.employee_id == df2.emp_id, how='inner')

This will perform an inner join between df1 and df2 using the employee_id from df1 and emp_id from df2.

The correct way to aggregate information (e.g., max value) from distributed workers back to the driver is using RDD actions such as reduce() or aggregate().

From the documentation:

“To perform global aggregations on distributed data, actions like reduce() are commonly used to collect summaries such as min/max/avg.”

Accumulators (Option B) do not support max operations directly and are not intended for such analytics.

Broadcast (Option C) is used to send data to workers, not collect from them.

Spark UI (Option D) is a monitoring tool — not an analytics collection interface.

Final Answer: A