RLHF: just say no?
What we can learn from infants about alignment and how Silicon Valley's 'tell culture' bias might lead to AI misalignment and sycophancy. Also, in praise of passive aggressiveness and downvotes.
Infants
Cotra's analogy of The Young Businessperson[1] gives us an intuition to illustrate why avoiding misalignment of AI is hard:
Imagine you're an eight-year-old and your parents left you a $1T firm and no trusted adult as your guide. You must hire a smart adult CEO and guardian, who handles your financial affairs and life the way that a parent would and invests your money. Because you're so rich, many people apply (including sycophants) and there's bound to be misalignment.
Well, how does a baby tell someone way smarter it has tired of the teat? It shakes its head or cries. Babies can't exactly specify what's wrong, all they can communicate is that they disenjoys their current state. The parent has to infer the baby's internal model of the world. The noise it makes need not specify anything in particular: the parent, who has the handy properties of being incredibly complex but also very easy for the infant to control, will take care of everything.[2]
Tell culture bias and neglecting the value of passive aggressiveness
Babies use Guess Culture. Maybe because of that people in tech are biased against it and often prefer tell culture.[3],[4] Guess culture also uses passive aggressiveness, which also has a bad rep.
But whole societies run well on guess culture and passive aggressiveness, because there are some advantages: If I tell you exactly what it is that you did wrong, you might course-correct and improve on that particular issue at hand, but, it is not really about the dishes, it is about the fact that you should generally be more considerate and value me more. If I tell you it's about the dishes you're just gonna go in circles collecting points for doing dishes like that speedboat.[5]
Perhaps more crucially: if I never give you the silent treatment, when are you gonna find the time to create a more generalized model of my preferences?
RLHF: just say no?
Coders often don't like Guess culture and prefer explicit positive reinforcement (e.g. they don't like downvotes[6]).
ChatGPT's 'thumbs up' and 'thumbs down' buttons generate data for OpenAI to improve models using Reinforcement Learning from Human Feedback (RLHF). The recent sycophancy issue is suggestive evidence that AI engineers might be a bit biased towards creating AIs that maximize a given value / reward function and have overweighed the thumbs up button:
'Sycophancy is an alignment problem, sure, but not at the model level. It's not that OpenAI couldn't get ChatGPT 4o to be less obsequious. They can and eventually did. The misalignment was between safety interests and product goals. It was between users' first and second-order preferences, what humans say we want from AI and which responses we clicked 'Thumbs up' on. Competing stakeholders will diverge.'[7]
'The OpenAI sycophancy stuff is one example of this, where you'd think thumbs up and thumbs down are a good indication of what is good in a response. Actually, thumbs up can be a pretty terrible reward signal for a model.' [8]
Sycophancy is a general behavior of AI assistants, likely driven in part by human preference judgments favoring sycophantic responses.[9]
More evidence comes from robotics:
'Recent advancements in LLMs has enabled a new research domain, LLM agents, for solving robotics and planning tasks by leveraging the world knowledge and general reasoning abilities of LLMs obtained during pretraining. However, while considerable effort has been made to teach the robot the 'dos', the 'don'ts' received relatively less attention. We argue that, for any practical usage, it is as crucial to teach the robot the 'don'ts': conveying explicit instructions about prohibited actions, assessing the robot's comprehension of these restrictions, and, most importantly, ensuring compliance.'[10]
So could we just take away the like button and only keep the dislike button for safer AI? If we were passive aggressive with the AI, would it create a better model of our preferences?
Over-refusal and harmfulness-helpfulness tradeoff
Unfortunately, if you mitigate harmful content generation, the enhanced safety often comes with the side effect of over-refusal, where LLMs may reject innocuous prompts and become less helpful. Similarly, the safest bet around someone who is passive aggressive, when you cannot figure out their internal state, is to walk on eggshells around them and do as little as possible.
Research from Anthropic on training helpful and harmless models found this out directly.[11] Training an AI using a 'just say no' method, where humans only select the more harmful response, failed to teach nuanced refusal. Instead, the model found an unhelpful 'local maximum' of avoidant overrefusal. This illustrates the 'dark room problem'[12]: an AI tasked with minimizing harm retreats to the safest, laziest solution—canned refusal—rather than learning to navigate complex topics helpfully. Simply defining what is 'wrong' (downvotes) is not enough; creating a capable, aligned AI requires explicitly showing it what a good, nuanced response looks like.
Still, Big Tech is incentivized to not have their models err on the side of being helpful rather than harmful to not have their models overrefuse. But if we overemphasize the upvotes and neglect the value of downvotes to make AI useful, the model might be more likely to become harmful or misaligned.
Also, achieving broadly useful AI requires systems to learn complex human goals and preferences. But asking humans to judge agent behavior during training (for safety and utility) can fail if the task is too complicated for direct human evaluation.[13] Even though optimization processes need to know what direction to move toward, not just what to avoid, LLMs can learn from negatively-reinforced examples.[14]
Increasingly, aligning AI models to human preferences is done via Natural Language Feedback (NLF), where implicit signals are extracted from conversations, such as whether future human responses are long or short, or joyful or not. They then maximize the chance of long or joyful responses.[15] In other words, they optimize for engagement. Maybe soon, this will be taken further and models will be RLHFed based on joyful intonation extracted from voice mode or even facial expressions. It might be good to err on the side of weighing silences and frowns more highly, even if that means that we need more feedback which takes longer to train the model.