Databricks Updated Databricks-Machine-Learning-Associate Exam Questions and Answers by agnes

To determine the number of machine learning models that can be trained in parallel, we need to calculate the total number of combinations of hyperparameters. The given hyperparameter grid includes:

• Hyperparameter 1: [2, 5, 10] (3 values)
• Hyperparameter 2: [50, 100] (2 values)

The total number of combinations is the product of the number of values for each hyperparameter:3 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=63 (values of Hyperparameter 1)×2 (values of Hyperparameter 2)=6

With 3-fold cross-validation, each combination of hyperparameters will be evaluated 3 times. Thus, the total number of models trained will be:6 (combinations)×3 (folds)=186 (combinations)×3 (folds)=18

However, the number of models that can be trained in parallel is equal to the number of hyperparameter combinations, not the total number of models considering cross-validation. Therefore, 6 models can be trained in parallel.

References:

• Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

To parallelize the inference of group-specific models using the Pandas Function API in PySpark, you can use theapplyInPandasfunction. This function allows you to apply a Python function on each group of a DataFrame and return a DataFrame, leveraging the power of pandas UDFs (user-defined functions) for better performance.

prediction_df = ( df.groupby("device_id") .applyInPandas(apply_model, schema=apply_return_schema) )

In this code:

• groupby("device_id"): Groups the DataFrame by the "device_id" column.
• applyInPandas(apply_model, schema=apply_return_schema): Applies theapply_modelfunction to each group and specifies the schema of the return DataFrame.

References:

• PySpark Pandas UDFs Documentation

Tree of Parzen Estimators (TPE) is a sequential model-based optimization algorithm that selects hyperparameter values based on the outcomes of previous trials. It models the probability density of good and bad hyperparameter values and makes informed decisions about which hyperparameters to try next.

This approach contrasts with methods like random search and grid search, which do not use information from previous trials to guide the search process.

References:

• Hyperopt and TPE