ISTQB Updated CT-AI Exam Questions and Answers by montague

The question asks which deficiency is most likely to be discovered by the test expert given the scenario of poor real-world performance despite good isolated accuracy.

A lack of similarity between the training and testing data (A): This is a common issue in ML where the model performs well on training data but poorly on real-world data due to a lack of representativeness in the training data. This leads to poor generalization to new, unseen data.

The input data has not been tested for quality prior to use for testing (B): While data quality is important, this option is less likely to be the primary reason for the described issue compared to the representativeness of training data.

A lack of focus on choosing the right functional-performance metrics (C): Proper metrics are crucial, but the issue described seems more related to the data mismatch rather than metric selection.

A lack of focus on non-functional requirements testing (D): Non-functional requirements are important, but the scenario specifically mentions issues with detecting real cancer cases, pointing more towards data issues.

A. Regression

Regression is used to predict a continuous value and is not suitable for grouping people based on similar tastes.

B. Association

Association is used to find relationships between variables in large datasets, often in the form of rules (e.g., market basket analysis). It does not directly group individuals but identifies patterns of co-occurrence.

C. Clustering

Clustering is an unsupervised learning method used to group similar data points based on their features. It is ideal for identifying groups of people with similar tastes without prior knowledge of the group labels. This technique will help the company segment its customer base effectively.

D. Classification

Classification is a supervised learning method used to categorize data points into predefined classes. It requires labeled data for training, which is not the case here as we want to identify groups without predefined labels.

Therefore, the correct answer isCbecause clustering is the most suitable method for grouping people with similar tastes for targeted product marketing​​.

The ISTQB CT-AI syllabus states in Section2.10 – Explainability, Transparency, and Interpretabilitythat interpretability isuser-dependent, meaning different users understand explanations differently. This is because interpretability depends not only on the ML algorithm but also on the user’s domain knowledge, experience, and expectations. OptionBdirectly reflects this syllabus principle: interpretability testing must consideruser background, and explanations must be comprehensible to the intended user group.

Option A is incorrect because explainability testing requires substantial information about the model, data, and expected behavior—not just exploratory effort. Option C is incorrect because explainability isnotgenerally quantifiable through dynamic testing alone, and the syllabus does not assert model-type specificity in this way. Option D exaggerates LIME’s capabilities. LIME offers approximate local explanations, but cannotpreciselystate root causes; the syllabus emphasizes itslimitationsand that explanations areapproximations, not exact reasons.

Therefore,Option Bis the most syllabus-aligned and correct statement.

The ISTQB CT-AI syllabus (Section3.2 – Model Evaluation) specifies the correct usage oftraining,validation, andtestdatasets. It emphasizes that thetest dataset must be representative of the real operational dataand must beequivalent in distribution to the training and validation sets, ensuring a fair and unbiased evaluation. Option D precisely matches this requirement.

Option A contradicts the syllabus because validation and test sets servedifferent purposes: validation is for tuning, test is for final evaluation. Combining them undermines the reliability of results. Option B is incorrect because even with limited data, the syllabus recommends maintaining a test set or using techniques such ascross-validationrather than eliminating testing. Option C is wrong because equal distribution (33/33/33) isnot recommended; typically, the training set is much larger (e.g., 70–80%).

Thus, OptionDis the only statement aligned with the syllabus' guidance.