Research

Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost. We study when a treatment effect estimated on LLM outcomes can recover the effect for the human population of interest. Distributional equivalence between LLM and human outcomes would make any standard estimator valid but is unrealistic. We therefore develop a statistical framework that adapts surrogate endpoint theory to LLMs, showing that calibrating LLM outcomes to human outcomes identifies the average treatment effect under surrogacy and comparability conditions that are jointly weaker than distributional equivalence. We present a falsification test for surrogacy and a bound on the worst-case bias from limited overlap between the LLM and human samples. We further show that the stochasticity inherent to LLMs can weaken surrogacy for identification while also introducing bias and variance during estimation, but that using an average over multiple LLM draws per unit as the surrogate mitigates these issues. Simulations validate the results, and an empirical application to the Upworthy Research Archive dataset shows that raw LLM outputs recover only 39% of the human treatment effect while nonparametric calibration closes the gap. A central takeaway is that A/B testing on LLM responses is correct only by assumption, whereas A/B testing on humans is correct by design, and that the required assumptions are hardest to justify precisely where LLMs promise the greatest benefit. We discuss the choice of LLM, prompting, and temperature as design variables, the compounded challenge posed by long-term outcomes, and how to size human pilot studies for validation.

Off-policy evaluation (OPE) estimates the value of a target treatment policy (e.g., a recommender system) using data collected by a different logging policy. It enables high-stakes experimentation without live deployment, yet in practice accuracy depends heavily on the logging policy used to collect data for computing the estimate. We study how to design logging policies that minimize OPE error for given target policies. We characterize a fundamental reward-coverage tradeoff: concentrating probability mass on high-reward actions reduces variance but risks missing signal on actions the target policy may take. We propose a unifying framework for logging policy design and derive optimal policies in canonical informational regimes where the target policy and reward distribution are (i) known, (ii) unknown, and (iii) partially known through priors or noisy estimates at logging time. Our results provide actionable guidance for firms choosing among multiple candidate recommendation systems. We demonstrate the importance of treatment selection when gathering data for OPE, and describe theoretically optimal approaches when this is a firm's primary objective. We also distill practical design principles for selecting logging policies when operational constraints prevent implementing the theoretical optimum.

Heterogeneous treatment effects (HTEs) are increasingly estimated using machine learning models that produce highly personalized predictions of treatment effects. In practice, however, predicted treatment effects are rarely interpreted, reported, or audited at the individual level but, instead, are often aggregated to broader subgroups, such as demographic segments, risk strata, or markets. We show that such aggregation can induce systematic bias of the group-level causal effect: even when models for predicting the individual-level conditional average treatment effect (CATE) are correctly specified and trained on data from randomized experiments, aggregating the predicted CATEs up to the group level does not, in general, recover the corresponding group average treatment effect (GATE). We develop a unified statistical framework to detect and mitigate this form of group bias in randomized experiments. We first define group bias as the discrepancy between the model-implied and experimentally identified GATEs, derive an asymptotically normal estimator, and then provide a simple-to-implement statistical test. For mitigation, we propose a shrinkage-based bias-correction, and show that the theoretically optimal and empirically feasible solutions have closed-form expressions. The framework is fully general, imposes minimal assumptions, and only requires computing sample moments. We analyze the economic implications of mitigating detected group bias for profit-maximizing personalized targeting, thereby characterizing when bias correction alters targeting decisions and profits, and the trade-offs involved. Applications to large-scale experimental data at major digital platforms validate our theoretical results and demonstrate empirical performance.

Marketing decisions that once relied on expert judgment are increasingly informed by data and algorithms. One prominent application is the audience-wide promotion problem faced by content publishers (e.g., newspapers, magazines, independent producers), who sequentially over time select a few items out of many to promote uniformly to their audience based on past information. In this paper, we consider the problem of using data from such expert decisions to learn new decision policies that improve upon them. We collaborate with a leading Swiss newspaper and use six weeks of high-frequency panel data that record all editorial promotion decisions, all information shown to editors at decision time, and all outcomes from their decisions. Informed by the setting, we frame the editorial decision process as a contextual bandit resulting in observational data with selection bias but strong variation across contexts and treatments. We leverage this framing to inform the application of causal machine learning methods to estimate conditional average treatment effects (CATEs) of promotion, evaluate counterfactual policies motivated by bandit algorithms, and measure the editors' alignment with those policies. We find that editors act "as if" they make decisions based on CATEs and their uncertainty, yet deploying algorithms that actually optimize for such signals could have increased annual revenue by USD 0.28–1.41 million. Our study shows how causal machine learning can be utilized with observational data from expert decisions to measure what latent reward signals experts optimize for, and quantify what surplus machine-learned policies that algorithmically optimize for such signals could bring.

Platform companies run thousands of online randomized experiments ("A/B tests") internally each year to test product innovations and decide which ones to roll out to their users. A systematic literature review finds that the vast majority of relevant research since 2000 has focused on questions around statistical inference within individual experiments, rather than on maximizing returns from many experiments. This paper proposes new research directions to bring the latter perspective to fruition. We anchor our proposals in the recent return-aware framework, which treats experimentation as a resource-allocation decision problem: given a finite user base and a set of possible ideas for innovations, how should a firm allocate experimentation effort and make rollout decisions to maximize expected returns? We outline two research directions not captured by the theoretical framework. The first concerns the statistical evidence and priors that enter decision-making, focusing on the use of non-experimental signals to pre-screen ideas for A/B testing and on how to account for ex-ante unknown errors in A/B tests themselves. We discuss partial identification as a promising approach. The second concerns frictions that arise when organizations aim to implement return-maximizing decision rules, including congestion and diminished user experience from frequent product innovations, as well as misaligned incentives between decentralized product teams and the organization's objectives. We call for research clarifying or resolving these frictions. Our aim is to inspire academic and industry research on how to align the practice of internal experimentation on platforms with intended business objectives, drawing upon concepts from statistics and economics.

Algorithms are increasingly used to automate or aid human decisions, yet recent research shows that these algorithms may exhibit bias across legally protected demographic groups. However, data on these groups may be unavailable to organizations or external auditors due to privacy legislation. This paper studies bias detection using an unsupervised bias detection tool when data on demographic groups are unavailable. We collaborated with the Dutch Executive Agency for Education to audit an algorithm that was used to assign risk scores to college students at the national level in the Netherlands between 2012-2023. Our audit covers more than 250,000 students across the country. The unsupervised bias detection tool highlights known disparities between students with a non-European migration background and students with a Dutch or European-migration background. Our contributions are two-fold: (1) we assess bias in a real-world, large-scale, and high-stakes decision-making process by a governmental organization; (2) we provide the unsupervised bias detection tool in an open-source library for others to use to complete bias audits. Our work serves as a starting point for a deliberative assessment by human experts to evaluate potential discrimination in algorithmic decision-making.

Large language models (LLMs) present an enormous evolution in the strategic potential of conversational recommender systems (CRS). Yet to date, research has predominantly focused upon technical frameworks to implement LLM-driven CRS, rather than end-user evaluations or strategic implications for firms, particularly from the perspective of a small to medium enterprises (SME) that makeup the bedrock of the global economy. In the current paper, we detail the design of an LLM-driven CRS in an SME setting, and its subsequent performance in the field using both objective system metrics and subjective user evaluations. While doing so, we additionally outline a short-form revised ResQue model for evaluating LLM-driven CRS, enabling replicability in a rapidly evolving field. Our results reveal good system performance from a user experience perspective (85.5% recommendation accuracy) but underscore latency, cost, and quality issues challenging business viability. Notably, with a median cost of $0.04 per interaction and a latency of 5.7s, cost-effectiveness and response time emerge as crucial areas for achieving a more user-friendly and economically viable LLM-driven CRS for SME settings. One major driver of these costs is the use of an advanced LLM as a ranker within the retrieval-augmented generation (RAG) technique. Our results additionally indicate that relying solely on approaches such as Prompt-based learning with ChatGPT as the underlying LLM makes it challenging to achieve satisfying quality in a production environment. Strategic considerations for SMEs deploying an LLM-driven CRS are outlined, particularly considering trade-offs in the current technical landscape.

Dynamic treatment regimes (DTRs) are used in medicine to tailor sequential treatment decisions to patients by considering patient heterogeneity. Common methods for learning optimal DTRs, however, have shortcomings: they are typically based on outcome prediction and not treatment effect estimation, or they use linear models that are restrictive for patient data from modern electronic health records. To address these shortcomings, we develop two novel methods for learning optimal DTRs that effectively handle complex patient data. We call our methods DTR-CT and DTR-CF. Our methods are based on a data-driven estimation of heterogeneous treatment effects using causal tree methods, specifically causal trees and causal forests, that learn non-linear relationships, control for time-varying confounding, are doubly robust, and explainable. To the best of our knowledge, our paper is the first that adapts causal tree methods for learning optimal DTRs. We evaluate our proposed methods using synthetic data and then apply them to real-world data from intensive care units. Our methods outperform state-of-the-art baselines in terms of cumulative regret and percentage of optimal decisions by a considerable margin. Our work improves treatment recommendations from electronic health record and is thus of direct relevance for personalized medicine.

Introduction. Human mobility was considerably reduced during the COVID-19 pandemic. To support disease surveillance, it is important to understand the effect of mobility on transmission. Aim. We compared the role of mobility during the first and second COVID-19 wave in Switzerland by studying the link between daily travel distances and the effective reproduction number (Rt) of SARS-CoV-2. Methods. We used aggregated mobile phone data from a representative panel survey of the Swiss population to measure human mobility. We estimated the effects of reductions in daily travel distance on Rt via a regression model. We compared mobility effects between the first (2 March-7 April 2020) and second wave (1 October-10 December 2020). Results. Daily travel distances decreased by 73% in the first and by 44% in the second wave (relative to February 2020). For a 1% reduction in average daily travel distance, Rt was estimated to decline by 0.73% (95% credible interval (CrI): 0.34-1.03) in the first wave and by 1.04% (95% CrI: 0.66-1.42) in the second wave. The estimated mobility effects were similar in both waves for all modes of transport, travel purposes and sociodemographic subgroups but differed for movement radius. Conclusion. Mobility was associated with SARS-CoV-2 Rt during the first two epidemic waves in Switzerland. The relative effect of mobility was similar in both waves, but smaller mobility reductions in the second wave corresponded to smaller overall reductions in Rt. Mobility data from mobile phones have a continued potential to support real-time surveillance of COVID-19.

To manage the COVID-19 epidemic effectively, decision-makers in public health need accurate forecasts of case numbers. A potential near real-time predictor of future case numbers is human mobility; however, research on the predictive power of mobility is lacking. To fill this gap, we introduce a novel model for epidemic forecasting based on mobility data, called mobility marked Hawkes model. The proposed model consists of three components: (1) A Hawkes process captures the transmission dynamics of infectious diseases. (2) A mark modulates the rate of infections, thus accounting for how the reproduction number R varies across space and time. The mark is modeled using a regularized Poisson regression based on mobility covariates. (3) A correction procedure incorporates new cases seeded by people traveling between regions. Our model was evaluated on the COVID-19 epidemic in Switzerland. Specifically, we used mobility data from February through April 2020, amounting to approximately 1.5 billion trips. Trip counts were derived from large-scale telecommunication data, i.e., cell phone pings from the Swisscom network, the largest telecommunication provider in Switzerland. We compared our model against various state-of-the-art baselines in terms of out-of-sample root mean squared error. We found that our model outperformed the baselines by 15.52%. The improvement was consistently achieved across different forecast horizons between 5 and 21 days. In addition, we assessed the predictive power of conventional point of interest data, confirming that telecommunication data is superior. To the best of our knowledge, our work is the first to predict the spread of COVID-19 from telecommunication data. Altogether, our work contributes to previous research by developing a scalable early warning system for decision-makers in public health tasked with controlling the spread of infectious diseases.

In response to the novel coronavirus disease (COVID-19), governments have introduced severe policy measures with substantial effects on human behavior. Here, we perform a large-scale, spatio-temporal analysis of human mobility during the COVID-19 epidemic. We derive human mobility from anonymized, aggregated telecommunication data in a nationwide setting (Switzerland; February 10 - April 26, 2020), consisting of ~1.5 billion trips. In comparison to the same time period from 2019, human movement in Switzerland dropped by 49.1%. The strongest reduction is linked to bans on gatherings of more than 5 people, which is estimated to have decreased mobility by 24.9%, followed by venue closures (stores, restaurants, and bars) and school closures. As such, human mobility at a given day predicts reported cases 7-13 days ahead. A 1% reduction in human mobility predicts a 0.88-1.11% reduction in daily reported COVID-19 cases. When managing epidemics, monitoring human mobility via telecommunication data can support public decision-makers in two ways. First, it helps in assessing policy impact; second, it provides a scalable tool for near real-time epidemic surveillance, thereby enabling evidence-based policies.