Databricks Updated Databricks-Certified-Associate-Developer-for-Apache-Spark-3.0 Exam Questions and Answers by zaviyar

Explanation

Pay attention here to which variables are quoted. fileSchema is a variable and thus should not be in quotes. parquet is not a variable and therefore should be in quotes.

SparkSession.read (here referenced as spark.read) returns a DataFrameReader which all subsequent calls reference - the DataFrameReader is not callable, so you should not use parentheses

here.

Finally, there is no open method in PySpark. The method name is load.

Static notebook | Dynamic notebook: See test 1, QUESTION NO: 44 (Databricks import instructions)

Explanation

This QUESTION NO: deals with the parameters of Spark's split operator for strings.

To solve this question, you first need to understand the difference between DataFrame.withColumn() and DataFrame.withColumnRenamed(). The correct option here is DataFrame.withColumn()

since, according to the question, we want to add a column and not rename an existing column. This leaves you with only 3 answers to consider.

The second gap should be filled with the name of the new column to be added to the DataFrame. One of the remaining answers states the column name as itemNameBetweenSeparators, while the

other two state it as "itemNameBetweenSeparators". The correct option here is "itemNameBetweenSeparators", since the other option would let Python try to interpret itemNameBetweenSeparators

as the name of a variable, which we have not defined. This leaves you with 2 answers to consider.

The decision boils down to how to fill gap 5. Either with 4 or with 5. The QUESTION NO: asks for arrays of maximum four strings. The code in gap 5 relates to the limit parameter of Spark's split

operator

(see documentation linked below). The documentation states that "the resulting array's length will not be more than limit", meaning that we should pick the answer option with 4 as the code in the

fifth gap here.

On a side note: One answer option includes a function str_split. This function does not exist in pySpark.

More info: pyspark.sql.functions.split — PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 38 (Databricks import instructions)

Explanation

Correct code block:

transactionsDf.filter(transactionsDf.storeId.isNotNull()).groupBy("storeId").pivot("productId", [2, 3]).mean("predError").orderBy("storeId")

Output of correct code block:

+-------+----+----+

|storeId| 2| 3|

+-------+----+----+

| 2| 6.0|null|

| 3|null|null|

| 25| 3.0| 3.0|

+-------+----+----+

This QUESTION NO: is quite convoluted and requires you to think hard about the correct order of operations. The pivot method also makes an appearance - a method that you may not know all

that much

about (yet).

At the first position in all answers is code block 4, so the QUESTION NO: is essentially just about the ordering of the remaining 4 code blocks.

The QUESTION NO: states that the returned DataFrame should be sorted by column storeId. So, it should make sense to have code block 3 which includes the orderBy operator at the very end of

the code

block. This leaves you with only two answer options.

Now, it is useful to know more about the context of pivot in PySpark. A common pattern is groupBy, pivot, and then another aggregating function, like mean. In the documentation linked below you

can see that pivot is a method of pyspark.sql.GroupedData - meaning that before pivoting, you have to use groupBy. The only answer option matching this requirement is the one in which code

block 2 (which includes groupBy) is stated before code block 5 (which includes pivot).

More info: pyspark.sql.GroupedData.pivot — PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 43 (Databricks import instructions)

Explanation

Correct code block:

transactionsDf.drop("predError", "productId", "value")

Static notebook | Dynamic notebook: See test 1, QUESTION NO: 37 (Databricks import instructions)