Researchers have found a novel way to approach reward models (RMs) in reinforcement learning from human feedback (RLHF), making use of synthetic critiques.

In the paper titled ‘Improving Reward Models with Synthetic Critiques’, researchers from Cohere and the University of Oxford proposed using large language models (LLMs) to align other language models, reducing the cost and time needed in using human annotation.

“RMs are trained to predict a score reflecting human preference, which requires significant time and cost for human annotation. Additionally, RMs tend to quickly overfit on superficial features in the training set, hindering their generalisation performance on unseen distributions,” the researchers said.

To rectify this, the researchers propose using LLMs to generate critiques, assessing the relationship between prompts and the generated output and predicting scalar awards. Through experimentation, they found that the synthetic critiques were able to score higher on existing RM benchmarks.

These synthetic critiques provided additional feedback in various aspects like instruction following, correctness, and style. Taking these into consideration, the reward models received better signals and features, allowing them to better assess and score the language models. 

Interestingly, as seen in the figure, GPT-4o was able to match and, in certain benchmarks, even surpass RMs with no critiques. Additionally, they warned that using poor models could be detrimental to the overall training process, as seen in the case of LLaMa-2 7B.

They also highlighted that the use of synthetic critiques meant an increase in data efficiency. “In these settings one high-quality critique-enhanced preference pair is worth 40 non-enhanced preference pairs. As these critiques are generated without human labour, this approach could make it much more cost-effective to obtain competitive reward models,” they said.

There have been several discussions on improving how reward models work, making them less reliant on RLHF. Currently, all major AI players rely on the reward approach to align their LLMs, including Google, OpenAI and Meta.

Research on replacing RLHF with reinforcement learning from AI feedback (RLAIF) has also been ongoing at Google Research, who found that their approach still meant that RLHF had an edge over AI. With this paper, however, the tide could be turning in favour of AI-based critiquing.

Researchers have found a novel way to approach reward models (RMs) in reinforcement learning from human feedback (RLHF), making use of synthetic critiques.

In the paper titled ‘Improving Reward Models with Synthetic Critiques’, researchers from Cohere and the University of Oxford proposed using large language models (LLMs) to align other language models, reducing the cost and time needed in using human annotation.

“RMs are trained to predict a score reflecting human preference, which requires significant time and cost for human annotation. Additionally, RMs tend to quickly overfit on superficial features in the training set, hindering their generalisation performance on unseen distributions,” the researchers said.

To rectify this, the researchers propose using LLMs to generate critiques, assessing the relationship between prompts and the generated output and predicting scalar awards. Through experimentation, they found that the synthetic critiques were able to score higher on existing RM benchmarks.

These synthetic critiques provided additional feedback in various aspects like instruction following, correctness, and style. Taking these into consideration, the reward models received better signals and features, allowing them to better assess and score the language models. 

Interestingly, as seen in the figure, GPT-4o was able to match and, in certain benchmarks, even surpass RMs with no critiques. Additionally, they warned that using poor models could be detrimental to the overall training process, as seen in the case of LLaMa-2 7B.

They also highlighted that the use of synthetic critiques meant an increase in data efficiency. “In these settings one high-quality critique-enhanced preference pair is worth 40 non-enhanced preference pairs. As these critiques are generated without human labour, this approach could make it much more cost-effective to obtain competitive reward models,” they said.

There have been several discussions on improving how reward models work, making them less reliant on RLHF. Currently, all major AI players rely on the reward approach to align their LLMs, including Google, OpenAI and Meta.

Research on replacing RLHF with reinforcement learning from AI feedback (RLAIF) has also been ongoing at Google Research, who found that their approach still meant that RLHF had an edge over AI. With this paper, however, the tide could be turning in favour of AI-based critiquing.