Databricks Updated Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Questions and Answers by yousuf

To flatten an array of structs into individual rows and access fields within each struct, you must:

Use explode() to expand the array so each struct becomes its own row.

Access the struct fields via dot notation (e.g., record_exploded.sensor_id).

Option C does exactly that:

First, explode the record array column into a new column record_exploded.

Then, access fields of the struct using the dot syntax in select.

This is standard practice in PySpark for nested data transformation.

Final Answer: C

Apache Spark follows a lazy evaluation model, meaning transformations (like filter(), select(), map()) are not executed immediately. Instead, they build a logical plan (lineage graph) that represents the sequence of operations to be applied.

Execution only begins when an action (e.g., count(), collect(), save(), show()) is called. At that point, Spark’s engine:

Optimizes the logical plan into a physical plan.

Divides it into stages and tasks.

Executes them across the cluster.

This design helps Spark optimize execution paths and avoid unnecessary computations.

Why the other options are incorrect:

A: Transformations do not execute immediately; they are deferred.

B: Optimization happens during job execution (after an action), not during transformations.

D: Execution starts automatically once an action is triggered, no manual intervention needed.

To sort a Spark DataFrame by multiple columns, use .orderBy() (or .sort()) with column expressions.

Correct syntax for descending and ascending mix:

from pyspark.sql.functions import col

df1.orderBy(col("count").desc(), col("Name").asc())

This sorts primarily by count in descending order and secondarily by Name in ascending order (alphabetically).

Why the other options are incorrect:

B/C: Default sort order is ascending; won’t place highest counts first.

D: Reverses sorting logic — sorts Name descending, not required.

In Apache Spark’s execution hierarchy, the relationships are structured as follows:

Job: Created when an action (e.g., count(), collect(), save()) is triggered on an RDD or DataFrame.

Stage: Each job is divided into one or more stages, separated by shuffle boundaries (e.g., after a reduceByKey or join).

Task: Each stage consists of multiple tasks, one per partition, executed in parallel on executors.

Execution Hierarchy:

Job → Stage(s) → Task(s)

So, a job contains multiple stages, and each stage contains multiple tasks.

Why the other options are incorrect:

A: A job does not directly contain tasks without stages.

B: A stage cannot contain multiple jobs; it belongs to a single job.

C: Tasks do not contain jobs.

References (Databricks Apache Spark 3.5 – Python / Study Guide):

Spark Architecture Overview — Execution Hierarchy: Jobs, Stages, and Tasks.

Databricks Exam Guide (June 2025): Section “Apache Spark Architecture and Components” — describes execution hierarchy and lazy evaluation.

===========