Databricks Updated Databricks-Certified-Data-Engineer-Associate Exam Questions and Answers by marco

The File Arrival trigger and Auto Loader perform separate, complementary functions. The trigger monitors the configured storage location and starts the Lakeflow Job after detecting new files. After the notebook starts, Auto Loader examines its own checkpoint and file-tracking state to determine which files have not yet been committed successfully. This separation provides incremental processing and exactly-once ingestion behavior. With availableNow=True, Auto Loader processes all data available when the query starts and then stops, allowing the job compute to terminate and control costs. File Notification mode is not required solely because a File Arrival trigger is used. Disabling checkpoints would remove progress tracking and could cause duplicate processing. The trigger also does not eliminate the need to configure the Auto Loader source path. Therefore, option A is correct.

================

 Option A is the correct answer because it uses the FILTER higher-order function correctly to filter out employees with more than 5 years of experience from the array column “employees”. It applies a lambda function i - > i.years_exp > 5 that checks if the years of experience of each employee in the array is greater than 5. If this condition is met, the employee is included in the new array column “exp_employees”.

The use of higher-order functions like FILTER can be referenced from Databricks documentation on Higher-Order Functions.

Pipeline A has a driver-side failure because collect() transfers every row to the driver process. Databricks states that collect() should be used only when the result is expected to be small. For a 100 GB dataset, the preferred correction is to remove the full collection and use distributed processing, aggregation, sampling, or limit; increasing driver capacity alone is not a durable solution. Pipeline B is an executor-side problem during a shuffle-intensive join. Increasing spark.sql.shuffle.partitions can divide the shuffled data into smaller task partitions, while workers with more memory per core can provide additional executor memory. Increasing the number of executors does not solve Pipeline A’s driver bottleneck, and increasing driver memory does not address Pipeline B’s executor failures. Therefore, option D correctly distinguishes the affected Spark components and appropriate remedies.

Option A is correct because Lakeflow Connect provides a built-in, fully managed Salesforce connector. It ingests Salesforce data into Databricks, supports incremental processing, and publishes governed destination tables in Unity Catalog. Managed SaaS connectors handle source-specific authentication, incremental reads, schema evolution, and automated retries, reducing the custom engineering required for API pagination, state tracking, and failure recovery. Calling the Salesforce REST API from a notebook would require the team to develop and maintain those capabilities itself. Auto Loader is designed to discover and incrementally process files arriving in cloud object storage; it does not connect directly to Salesforce. Using an exported Amazon S3 path would also require a separate export system outside the requested managed Databricks ingestion experience. Therefore, option A is the low-code capability that directly satisfies the Salesforce ingestion requirement.

================