Questions

Select the correct answer for each question.

1. What is the primary purpose of RLHF?

A) Increase the model’s vocabulary size

B) Improve tokenization speed

C) Align model behavior with human preferences

D) Reduce GPU memory usage

2. In the classical RLHF pipeline, which stage comes after Supervised Fine-Tuning?

A) Token compression

B) Reward model training

C) LoRA insertion

D) Vocabulary expansion

3. A reward model is trained to:

A) Generate longer responses

B) Predict the next token

C) Assign higher scores to preferred responses

D) Replace the base model

4. What is “reward hacking”?

A) When the model increases GPU utilization

B) When the policy exploits weaknesses in the reward function

C) When the dataset is too small

D) When the tokenizer fails

5. What is the main advantage of Direct Preference Optimization (DPO) over classical RLHF?

A) It eliminates the need for preference data

B) It removes the need for a reward model and PPO loop

C) It increases model size

D) It improves tokenizer performance

6. In DPO, the objective directly increases:

A) The total parameter count

B) The probability of rejected responses

C) The relative likelihood of chosen responses over rejected ones

D) The embedding dimension

7. What role does the reference model play in DPO?

A) It generates preference labels

B) It acts as a stability anchor during optimization

C) It replaces the policy model

D) It modifies the tokenizer

8. If the beta parameter in DPO is set too high, what may happen?

A) Training becomes faster

B) The model may overfit preference data or drift too strongly

C) The reward model is deleted

D) Memory usage decreases

9. What is a major risk of using AI-as-a-judge without auditing?

A) Reduced inference speed

B) Tokenizer corruption

C) Judge bias and drift

D) Increased GPU temperature

10. Why is a rubric critical when using synthetic feedback?

A) It increases model size

B) It ensures consistent and structured evaluation criteria

C) It removes the need for training

D) It improves tokenization

11. Which of the following is an example of distribution shift?

A) Changing the tokenizer version

B) Evaluating the model only on training-style prompts

C) Training with gradient accumulation

D) Using fp16 precision

12. In preference data, what does a pair (chosen, rejected) represent?

A) Two equally good responses

B) A random selection of outputs

C) A relative judgment between two responses

D) A supervised ground-truth label

13. Why can reinforcement-based alignment lead to overly cautious models?

A) Because tokenization changes

B) Because safety weighting may overpower helpfulness

C) Because GPUs overheat

D) Because LoRA is disabled

14. What is a practical method to reduce noise in synthetic preference datasets?

A) Increase batch size

B) Lower learning rate

C) Filter examples by judge confidence or multi-judge agreement

D) Remove the reference model

15. What is the key philosophical shift introduced by RLHF and DPO compared to SFT?

A) From token prediction to tokenizer optimization

B) From imitation of fixed answers to optimization based on preference

C) From GPU training to CPU training

D) From large models to small models