Reinforcement learning for bidding: a practical primer on pacing, auto-bidding and evaluation

A single bid looks like a prediction problem: estimate the impression's value, then bid. The budget changes that. Money spent in the morning is gone by evening, when the same audience might be cheaper. The bid landscape, the distribution of winning prices, also shifts hour by hour. Each bid changes the state the next bid faces, which is exactly the setting RL is built for.

Most practical systems act on a multiplier every few minutes, not on every bid. Cai et al. note that the large real-world auction volume forced them to approximate state values with neural networks (Cai et al., WSDM 2017). Wu et al. found that the immediate reward "is misleading under a critical resource constraint". They designed a learned reward instead (Wu et al., CIKM 2018, arXiv).

Before any RL, a bidder needs a baseline. The common one bids predicted value scaled by a multiplier and adjusts the multiplier with feedback. In a first-price auction the result is then shaded; in a second-price auction it can be bid as is. Theory supports this design: for budget-constrained bidding in repeated auctions, a dual-variable multiplier that learns from spend gives strong guarantees (Balseiro and Gur, Management Science, 2019). Aggarwal, Badanidiyuru and Mehta analysed autobidding under several constraints in the same spirit (WINE 2019).

The loop below is the shape of such a controller. It is pseudo-code for illustration, not any platform's implementation.

Published production work refines this loop rather than replacing it. A Google-authored field guide compares ways to combine budget pacing with return-on-spend pacing. It finds that taking the minimum of two separate multipliers performs almost as well as a fully coupled dual method (Balseiro et al., arXiv 2023, ICML 2024). A 2025 paper adds hysteresis to proportional feedback and reports 13% lower pacing error and 54% lower multiplier volatility in real auctions (Apparaju, Niu and Qi, arXiv, 29 September 2025).

Some platforms describe the same tools in plain terms. Topsort's documentation lists two pacing mechanisms: throttling, which excludes a campaign from some auctions, and discount pacing, which lowers bids to slow spending (Topsort docs, accessed 9 November 2026). Smart Pacing, from Yahoo researchers, learned pacing rates from data and was implemented in a production DSP (Xu et al., KDD 2015, arXiv).

An RL policy is worth deploying only if it beats a well-tuned version of this baseline on the same traffic. The shading step in the last line is covered in bid shading explained.

The table lists the papers most cited in this area, with authors, venue and what each one says about real deployment. "Deployment evidence" quotes the paper's own claim; none has been independently verified.

The pattern is clear. Most production evidence comes from one company, Alibaba, on its own closed auctions. That matters, because a platform that runs the auction sees every competitor's bid. An independent DSP bidding into other people's exchanges sees far less of the market.

Few ad platforms publish engineering detail on bidding. The table separates what is described as reinforcement learning from what is described as control or rules.

Vendors that sell "AI bidding" rarely say which method they use. Ask whether the system learns a policy over time, or predicts value per impression and paces with a controller. Both are valid; they fail in different ways. How to question such a system is covered in explainable AI in bidding.

Write the environment down before choosing an algorithm. The specification below is our illustrative format. It fixes what the policy sees, what it controls, how it is scored and which limits it can never break.

The hard_limits block is the most important part. Guardrails live in the bidder, not in the model, as described in RTB bidder architecture.

RL needs interaction, and interaction costs money. Teams therefore train on simulators or on logged data first. Both have known failure modes.

A simulator replays historical auctions and models competitors' responses. Mou et al. showed that gaps between such a virtual system and the real one make offline training unreliable. They proposed bounded, safe exploration in the live system instead (NeurIPS 2022). A 2025 paper from the same group describes the field moving from simulator-based RL to offline RL on fixed real datasets, then to learned environment models. It notes that offline policies are "limited by the dataset's state space coverage" (Mou et al., arXiv, 22 June 2025).

The practical consequence for a DSP is simple. Your logs contain only the bids your current policy made. A new policy that bids very differently is asking about situations the logs never saw. Keep candidate policies close to the logged one at first, and widen the range as live data accumulates. Logs with full bid and price detail are the raw material; their structure is covered in log-level data.

A learned bidder should pass four gates, in order. Each gate is cheaper than the next and catches different failures.

1. Offline replay. Estimate the new policy's results on logged data. Li et al.'s replay method gives unbiased estimates when the logging policy explored at random (Li et al., WSDM 2011, arXiv). Bidding logs rarely meet that condition, so treat replay results as optimistic.
2. Benchmark environment. Compare against baselines on a shared environment such as AuctionNet, which includes linear programming, RL and generative baselines (Su et al., 2024).
3. Budget-split online test. Ordinary A/B tests are biased in auctions, because treatment and control campaigns compete for the same impressions. Splitting each campaign's budget between arms removes that bias. Liu, Mao and Kang show the design is unbiased wherever buyers have a defined budget (Liu, Mao and Kang, arXiv, 16 December 2020). DoorDash used budget-split experiments to validate its pacing change (Garg et al., 2025).
4. Staged rollout with guardrails. Move a small share of campaigns, with hard bid and spend caps, an automatic fallback to the baseline and a daily review of overspend and cost per acquisition.

The interaction risk is specific to auctions. When most buyers use learning bidders, each one's environment is the others' policies. A bidder that tests well alone can behave differently once rivals adapt to it. To see how inputs move a single bid, try the explainable AI RTB simulator.

1. Skipping the baseline. Without a tuned controller on the same traffic, an RL gain cannot be measured.
2. Plain A/B tests in an auction. Treatment and control compete for the same impressions; use budget-split designs.
3. Letting the model own the limits. Bid caps and spend caps belong in the bidder, outside any learned component.
4. Training on features not available at bid time. Leaked future data makes offline results look far better than live ones.
5. Rewarding only volume. A reward without a cost term teaches the policy to spend the budget.
6. Trusting replay estimates from non-random logs. Logged bids reflect the old policy; estimates far from it are guesses.
7. Retraining without a changelog. If a policy changes weekly, record each version so audits can match bids to policies.

1. Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu and Defeng Guo, "Real-Time Bidding by Reinforcement Learning in Display Advertising" (arXiv, 10 January 2017; WSDM 2017). https://arxiv.org/abs/1701.02490
2. Di Wu, Xiujun Chen, Xun Yang, Hao Wang, Qing Tan, Xiaoxun Zhang, Jian Xu and Kun Gai, "Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising" (arXiv, 23 February 2018; CIKM 2018). https://arxiv.org/abs/1802.08365
3. Junqi Jin, Chengru Song, Han Li, Kun Gai, Jun Wang and Weinan Zhang, "Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising" (arXiv, 27 February 2018; CIKM 2018). https://arxiv.org/abs/1802.09756
4. Jun Zhao, Guang Qiu, Ziyu Guan, Wei Zhao and Xiaofei He, "Deep Reinforcement Learning for Sponsored Search Real-time Bidding" (arXiv, 1 March 2018; KDD 2018). https://arxiv.org/abs/1803.00259
5. Yue He, Xiujun Chen, Di Wu, Junwei Pan, Qing Tan, Chuan Yu et al., "A Unified Solution to Constrained Bidding in Online Display Advertising" (KDD 2021, 14 August 2021). https://doi.org/10.1145/3447548.3467199
6. Zhiyu Mou, Yusen Huo, Rongquan Bai, Mingzhou Xie, Chuan Yu, Jian Xu and Bo Zheng, "Sustainable Online Reinforcement Learning for Auto-bidding" (arXiv, 13 October 2022; NeurIPS 2022). https://arxiv.org/abs/2210.07006
7. Jiayan Guo, Yusen Huo, Zhilin Zhang, Tianyu Wang, Chuan Yu, Jian Xu, Yan Zhang and Bo Zheng, "AIGB: Generative Auto-bidding via Conditional Diffusion Modeling" (arXiv, 25 May 2024; KDD 2024). https://arxiv.org/abs/2405.16141
8. Kefan Su, Yusen Huo, Zhilin Zhang, Shuai Dou, Chuan Yu, Jian Xu, Zongqing Lu and Bo Zheng, "AuctionNet: A Novel Benchmark for Decision-Making in Large-Scale Games" (arXiv, 14 December 2024). https://arxiv.org/abs/2412.10798
9. Zhiyu Mou, Miao Xu, Wei Chen, Rongquan Bai, Chuan Yu and Jian Xu, "Permutation Equivariant Model-based Offline Reinforcement Learning for Auto-bidding" (arXiv, 22 June 2025). https://arxiv.org/abs/2506.17919
10. Jian Xu, Kuang-chih Lee, Wentong Li, Hang Qi and Quan Lu, "Smart Pacing for Effective Online Ad Campaign Optimization" (arXiv, 18 June 2015; KDD 2015). https://arxiv.org/abs/1506.05851