Databricks Updated Databricks-Generative-AI-Engineer-Associate Exam Questions and Answers by ronnie

Problem Context : After receiving positive feedback for the RAG application prototype, the next step is to formally evaluate the system to pinpoint areas for improvement.

Explanation of Options :

Option A : While cosine similarity scores are useful, they primarily measure similarity rather than the overall performance of an RAG system.

Option B : This option provides a systematic approach to evaluation by testing both retrieval and generation components separately. This allows for targeted improvements and a clear understanding of each component ' s performance, using MLflow’s metrics for a structured and standardized assessment.

Option C : Benchmarking multiple LLMs does not focus on evaluating the existing system’s components but rather on comparing different models.

Option D : Using an LLM as a judge is subjective and less reliable for systematic performance evaluation.

Option B is the most comprehensive and structured approach, facilitating precise evaluations and improvements on specific components of the RAG system.

The task involves improving a Retrieval-Augmented Generation (RAG) application’s performance by experimenting with embedding models. The choice of embedding model impacts retrieval accuracy, which is critical for RAG systems. Let’s evaluate the options based on Databricks Generative AI Engineer best practices.

Option A: Pick an embedding model trained on related domain knowledge

Embedding models trained on domain-specific data (e.g., industry-specific corpora) produce vectors that better capture the semantics of the application’s context, improving retrieval relevance. For RAG, this is a key strategy to enhance performance.

Databricks Reference : " For optimal retrieval in RAG systems, select embedding models aligned with the domain of your data " ( " Building LLM Applications with Databricks, " 2023).

Option B: Pick the most recent and most performant open LLM released at the time

LLMs are not embedding models; they generate text, not embeddings for retrieval. While recent LLMs may be performant for generation, this doesn’t address the embedding step in RAG. This option misunderstands the component being selected.

Databricks Reference : Embedding models and LLMs are distinct in RAG workflows: " Embedding models convert text to vectors, while LLMs generate responses " ( " Generative AI Cookbook " ).

Option C: Pick the embedding model ranked highest on the Massive Text Embedding Benchmark (MTEB) leaderboard hosted by HuggingFace

The MTEB leaderboard ranks models across general tasks, but high overall performance doesn’t guarantee suitability for a specific domain. A top-ranked model might excel in generic contexts but underperform on the engineer’s unique data.

Databricks Reference : General performance is less critical than domain fit: " Benchmark rankings provide a starting point, but domain-specific evaluation is recommended " ( " Databricks Generative AI Engineer Guide " ).

Option D: Pick an embedding model with multilingual support to support potential multilingual user questions

Multilingual support is useful only if the application explicitly requires it. Without evidence of multilingual needs, this adds complexity without guaranteed performance gains for the current use case.

Databricks Reference : " Choose features like multilingual support based on application requirements " ( " Building LLM-Powered Applications " ).

Conclusion : Option A is the best strategy because it prioritizes domain relevance, directly improving retrieval accuracy in a RAG system—aligning with Databricks’ emphasis on tailoring models to specific use cases.

The Prompt Registry preserves identifiable prompt versions, allowing the team to recover the exact template associated with a historical execution. Linking inference activity to the prompt’s name and version establishes which registered template the application actually used. MLflow also supports lineage between prompt versions, application versions, and execution traces. However, satisfying the entire audit requires retaining the actual runtime variable values or the fully rendered prompt in the relevant execution records. Registry history alone preserves the template, not every value substituted into it. Therefore, D is the best option, provided the logging captures those inputs. Model webhooks, experiment tags, or table history alone do not provide the same explicit prompt-version lineage and complete historical reconstruction. Databricks documentation

Separate Genie Spaces provide domain-specific access to structured business datasets through Databricks-native conversational querying. The Sales specialist calls the Sales Space, and the HR specialist calls the HR Space. Each Space should contain the appropriate datasets and business context, with Unity Catalog permissions enforcing access to its underlying resources. Separating Spaces makes domain boundaries explicit and avoids relying only on an LLM’s routing instructions. Knowledge Assistants primarily support document-based knowledge retrieval, making Genie the better fit for structured business questions. A shared Space with distinct permissions can enforce security, but separate Spaces better express the requested domain separation. The video’s phrase “HR Analyst for stock questions” appears inconsistent; “staff questions” is an inferred typing correction. Databricks documentation