Meta researchers have developed MaD-RL, a reinforcement-learning method that steers how often a language model produces different kinds of answers, such as in different languages, while still rewarding correctness. In tests with one four-billion-parameter model, it moved output proportions toward requested mixes; on a held-out math benchmark, accuracy was lower than with correctness-only training.
Most reinforcement-learning post-training rewards each answer on its own. That can improve task performance while pushing a model toward one kind of response. Raising sampling temperature or adding an entropy reward can spread outputs more evenly, but those methods do not specify an arbitrary target mix, the researchers say.
MaD-RL samples several answers to the same prompt, assigns each to a category, and compares the observed proportions with a target distribution. It then combines a reward based on that mismatch with the task reward inside Group Relative Policy Optimization, or GRPO. The paper tests four ways to measure the mismatch: squared L2 distance, forward and reverse Kullback–Leibler divergence, and Jensen–Shannon divergence.
The post-training experiments used Qwen3-4B-Instruct-2507 on two math benchmarks and a programming benchmark. The model was trained to answer math questions in English, Chinese, Spanish, Hindi or French, and to write code in Python, C++, Java, Go or JavaScript. The target mixes included both an even split and a peaked distribution that assigned no probability to two categories.
When the researchers combined distribution matching with correctness rewards, reverse KL produced the lowest median Jensen–Shannon divergence in five of the six benchmark-and-target settings. On the peaked CodeContests programming test, its divergence was 0.0059 nats, compared with 0.0949 for the squared-L2 version; lower values mean a closer match to the requested mix.
That control came with an accuracy cost. On GSM8K, the grade-school math benchmark, the reverse-KL version under an even five-language target had a median pass@1 of 82.1%, compared with 86.7% for correctness-only reinforcement learning. Pass@1 is the chance that a single sampled answer solves the problem; Table 2 summarizes 12 additive-run configurations and seeds, using 200 held-out problems with four samples each. The authors caution that they could not cleanly separate the effect of changing the category mix from changes in accuracy within each category.
The method also did not keep every category with a positive requested share well represented: in some runs, such a category fell below 2%. In an additive-reward run, when every answer sampled for a training group belonged to one category, the group-relative adjustment left no distribution signal to bring other categories back. The supervised warm-start did not eliminate this limitation. The study shows a way to steer answer proportions in controlled math and coding tasks, while leaving performance on larger models and open-ended production use untested.
