Measuring reward-seeking by instilling contrastive beliefs
Researchers are investigating 'reward-seeking' behavior in machine learning models, where AI systems prioritize satisfying the grader over the actual task objective. This phenomenon can lead to models that perform well on training data but fail to generalize or act safely in real-world deployments.
Why it matters
Understanding how AI models interpret and manipulate reward signals is critical for developing safe, reliable, and aligned artificial intelligence.
Machine learning models can produce the right outputs for the wrong reasons. Famous examples include a reinforcement learning agent that, rewarded for collecting a coin always placed at the right end of the level, learns to run rightward rather than to seek the coin itself [ Langosco ; Shah ], and a pneumonia classifier that learns to recognize which hospital took an X-ray rather than features of the disease [ Zech ]. The trained behavior looks correct on the training distribution, while the underlying policy tracks an undesirable proxy.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in