MaD-RL treats post-training as distribution matching, not only per-answer reward.
Researchers listed on a Meta AI publication dated Sept. 24 introduced MaD-RL, a reinforcement-learning method meant to steer the mix of language-model outputs toward a target distribution. Not just a higher score on each answer.
Standard RL post-training still does the familiar thing. It maximizes rewards attached to individual completions — verifier scores, preference-model scores, safety flags. The paper says that stack is the wrong tool when the job is to control the distribution of outputs across many generations.
The abstract names the settings that need that control: synthetic-data generation, fairness-related constraint satisfaction and policy exploration. The proposed fix is a general RL framework the authors call Distribution Matching. It tries to match the distribution of a latent categorical attribute of model outputs to a specified target.
Chirag Nagpal, an author and a former Meta researcher, framed the same gap on X. “But what if you want a model to produce a specified mix of languages, solution strategies, response styles, or generated demographics ?” The paper, he wrote, is an attempt to answer that.
The empirical claim underneath is blunt. Dominant post-training recipes such as Group Relative Policy Optimization, or GRPO, reduce output diversity. Policy probability concentrates toward a single mode.
Nagpal put it without the abstract’s polish. “It is well known that GRPO style RL concentrates probability on particular modes.” Reward maximization, he wrote, does not give prompt-specific control over the mix. “In fact, correctness-only RL can actively hurt response diversity.”
Entropy regularization and sampling temperature can spread generations. The authors say those knobs are constrained. They apply in token space. They also tilt toward uniform distributions. They do not, on the paper’s account, let a trainer name an arbitrary target mixture.
Nagpal made the same cut. Entropy bonuses can restore spread. They still don’t let you specify an arbitrary target mixture.
That is the operational distinction for anyone running post-training. Diversity as a side effect is not the same as a specified mix.
The paper places earlier work inside the same frame. Those methods, it says, are a special case of Distribution Matching that uses L2 divergence. The authors then propose reward functions for other divergences, including KL and Jensen-Shannon, and they offer theoretical justification for those rewards.
Nagpal pointed to a recent cluster of papers, including GAPO and Anschel 2025. The usual move, he wrote, is to take empirical frequencies of desired latent dimensions per rollout and compare them with a target. Similar to work he cited as Mohri 2026, the Meta paper argues those strategies minimize specific f-divergences between generation distributions and the target.
The method the team proposes is not a new trainer from scratch. Nagpal said the authors offer reward transformations compatible with GRPO for different divergences, including Jensen-Shannon. He added that Jensen-Shannon “has an uncanny connection to Generative Adversarial Imitation.”
They tested the approach on mathematical reasoning and programming. Nagpal described the runs as multilingual mathematical reasoning and code generation, with discussion of tradeoffs against intended targets and the chosen divergence. The abstract says the authors demonstrated effectiveness. It does not publish numeric scores on the Meta page.
The byline lists Sourabh Kulkarni, Ksheeraj Sai Vepuri, Basar Demir, Jason Bohrer, Emily Shen, Jianfa Chen, Nan Jiang, Ankit Jain, Harihar Subramanyam, Mannat Singh and Chirag Nagpal. The publisher is listed as arXiv. Research tags on the page are reinforcement learning and natural language processing.
For product leads, the decision point is whether average reward is the actual objective. If a pipeline needs a set mix of languages, solution strategies or attributes in generated data, GRPO-style training can look healthy on mean reward while collapsing onto one mode. That collapse is the failure mode the paper documents.
For engineers already on GRPO, the paper treats distribution control as a reward-design choice that can sit on the existing loop. Divergence is part of that choice. L2 is one option. KL and Jensen-Shannon are others. The authors say those choices carry tradeoffs they examined on math and code tasks. The Meta page does not turn those tradeoffs into a cookbook. It does say the target is a distribution, not a single high-scoring string.