ISTQB Updated CT-AI Exam Questions and Answers by lylah

A. Add 10% of the rows randomly and create another model and compare the R-Square scores of both the models.

Adding more data to the training set can affect the R-Square score, but it does not directly verify the correctness of the implementation.

B. Train various models by changing the order of input features and verify that the R-Square score of these models vary significantly.

Changing the order of input features should not significantly affect the R-Square score if the implementation is correct, but this approach is more about testing model robustness rather than correctness of the implementation.

C. Compare the R-Square score of the model obtained using two different implementations that utilize two different programming languages while using the same algorithm and the same training and testing data.

This approach directly compares the performance of two implementations of the same algorithm. If both implementations produce similar R-Square scores on the same training and testing data, it suggests that the new implementation "X" is correct.

D. Drop 10% of the rows randomly and create another model and compare the R-Square scores of both the models.

Dropping data can lead to variations in the R-Square score but does not directly verify the correctness of the implementation.

Therefore, optionCis the most appropriate strategy because it directly compares the performance of the new implementation "X" with another implementation using the same algorithm and datasets, which helps in verifying the correctness of the implementation​​.

When verifyingautonomous AI-based systems, a critical aspect is ensuring that they maintain an appropriate level of autonomy whileonly requesting human intervention when necessary. If an AI system unnecessarily asks for human input, it defeats the purpose of autonomy and can:

Slow down operations.

Reduce trust in the system.

Indicate improper confidence thresholds in decision-making.

This is particularly crucial inautonomous vehicles, AI-driven financial trading, and robotic process automation, where excessive human intervention would hinder performance.

A. Test cases to verify that the system automatically confirms the correct classification of training data→ This is relevant for verifying training consistency but not for autonomy validation.

B. Test cases to detect the system appropriately automating its data input→ While relevant, data automation does not directly address the verification of autonomy.

D. Test cases to verify that the system automatically suppresses invalid output data→ This focuses on output filtering rather than decision-making autonomy.

Why are the other options incorrect?Thus, the mostcritical test casefor verifyingautonomous AI-based systemsis ensuring that itdoes not unnecessarily request human intervention.

Section 8.2 - Testing Autonomous AI-Based Systemsstates that it is crucial to testwhether the system requests human intervention only when necessaryand does not disrupt autonomy​.

Reference from ISTQB Certified Tester AI Testing Study Guide:

 AI-Based Autonomous Functions:An AI-based autonomous system is one that can respond to its environment without human intervention. The other options either involve human decisions or do not use AI at all.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Autonomy and Testing Autonomous AI-Based Systems​​​​.

A. The difficulty of defining criteria for improvement before the model can be accepted.

Defining criteria for improvement is a challenge in the acceptance of AI models, but it is not directly related to the performance drop in real-world scenarios. It relates more to the evaluation and deployment phase rather than affecting the model's real-time performance post-deployment.

B. The fast pace of change did not allow sufficient time for testing.

This can significantly affect the model's performance. If the system is self-learning, it needs to adapt quickly, and insufficient testing time can lead to incomplete learning and poor performance.

C. The unknown nature and insufficient specification of the operating environment might have caused the poor performance.

This is highly likely to affect performance. Self-learning AI systems require detailed specifications of the operating environment to adapt and learn effectively. If the environment is insufficiently specified, the model may fail to perform accurately in real-world scenarios.

D. There was an algorithmic bias in the AI system.

Algorithmic bias can significantly impact the performance of AI systems. If the model has biases, it will not perform well across different scenarios and data distributions.

Given the context of the self-learning nature and the need for real-time adaptability, optionAis least likely to describe the fall in performance because it deals with acceptance criteria rather than real-time performance issues​​.