Model-Free Reinforcement Learning¶
Two-Armed Bandit Task - 1Q1α1β Rescorla-Wagner Model¶
This example demonstrates parameter recovery for a simple two-armed bandit reinforcement learning task. On each trial the participant chooses between two options with different reward probabilities. Choices are generated with a Rescorla–Wagner model using an inverse-temperature parameter (beta) and a learning rate (alpha).
$$ \textbf{Expected value update:}\quad Q_{t+1}^k = Q_t^k + \alpha \cdot \delta_t \tag{1} $$
$$ \textbf{Prediction error:}\quad \delta_t = r_t - Q_t^k \tag{2} $$
$$ \textbf{Choice rule (softmax):}\quad p_t(\text{choose }k)=\frac{\exp(\beta \cdot Q_t^k)}{\sum_{k=1}^{K}\exp(\beta \cdot Q_t^k)} \tag{3} $$
import numpy as np
from pyem import EMModel
from pyem.models.rl_mf import rw1a1b_model
from params import build_params
# ModelSpec bundles this model's identity, description, and sim/fit entry points
print(f"Model ID : {rw1a1b_model.id}")
print(f"Description: {rw1a1b_model.desc}")
print(f"Spec : {rw1a1b_model.spec}")
Model ID : rw1a1b
Description: Rescorla-Wagner model with a single learning rate.
Choices are generated via a softmax over learned expected values.
Free parameters: inverse temperature (beta), learning rate (alpha).
Spec : {'rl': {'softmax': ['beta'], 'rw': ['alpha']}}
# simulate computer agents completing the two-armed task
nsubjects, nblocks, ntrials = 100, 6, 24
# generate "true" parameters using the shared parameter registry (examples/params.py)
param_names, param_xform, true_params = build_params(["beta", "alpha"], nsubjects)
# use EMModel.recover to run simulation, fitting and recovery metrics
model = EMModel(all_data=None, fit_func=rw1a1b_model.fit,
param_names=param_names,
param_xform=param_xform,
simulate_func=rw1a1b_model.sim)
recovery = model.recover(true_params,
pr_inputs=['choices','rewards'],
nblocks=nblocks, ntrials=ntrials
)
# scatter plot of recovered parameters
fig = model.plot_recovery(recovery)
recover(): this model's data and outfit now reflect the recovery fit; the recovered estimates are also in the returned dict['recovery_model'].
The recovery dictionary also contains numerical summaries such as
recovery['correlation'], which provides a correlation coefficient for each
parameter.
Model-Free Reinforcement Learning in Social Contexts¶
The following two examples demonstrate parameter recovery for different versions of a two-armed bandit reinforcement learning task, in which the agent learns how their actions affect themselves and another target. On each trial the agent chooses between different options with different outcome probabilities. Choices are generated with an adapted Rescorla–Wagner model using an inverse-temperature parameter (beta) and distinct learning rates (alpha) depending on the target.
Prosocial Learning Task (Lockwood et al., 2016, PNAS) - 1Q3α1β Model¶
Participants learn to make choices to win points for themselves (self), a stranger (other), or no one (no one) in separate conditions. Rewards include gaining points (+100) or not gaining points (0) on any given trial, which are later converted to real-world payouts.
The model is an adapated Rescorla Wagner model with 3 different learning rates $\alpha_i^\ast$ for each target (self, other, noone):
$$ \textbf{Expected value update:}\quad Q_{t+1}^k = Q_t^k + \alpha_i^\ast \cdot \delta_t \tag{1} $$
$$ \textbf{Prediction error:}\quad \delta_t = r_t - Q_t^k \tag{2} $$
$$ \textbf{Learning rates:}\quad \alpha_i^\ast = \begin{cases} \alpha_{\text{self}}, & \text{if } i=\text{self}\\[4pt] \alpha_{\text{other}}, & \text{if } i=\text{other}\\[4pt] \alpha_{\text{no one}}, & \text{if } i=\text{no one} \end{cases} \tag{3} $$
$$ \textbf{Choice rule (softmax):}\quad p_t(\text{choose }k)=\frac{\exp(\beta \cdot Q_t^k)}{\sum_{k=1}^{K}\exp(\beta \cdot Q_t^k)} \tag{4} $$
from pyem.models.rl_mf import rw3a1b_model
print(f"Model ID : {rw3a1b_model.id}")
print(f"Description: {rw3a1b_model.desc}")
print(f"Spec : {rw3a1b_model.spec}")
Model ID : rw3a1b
Description: Two-option task (A/B) with three binary outcome channels
(self, other, noone). Rescorla-Wagner learning combines self/other/no-one
prediction errors into a single expected-value update (Lockwood et al., 2016).
Free parameters: inverse temperature (beta), alpha_self, alpha_other, alpha_noone.
Spec : {'rl': {'softmax': ['beta'], 'rw': ['alpha_self', 'alpha_other', 'alpha_noone']}}
# simulate computer agents completing the prosocial learning task (Lockwood et al., 2016)
nsubjects, nblocks, ntrials = 100, 9, 16
# generate "true" parameters using the shared parameter registry (examples/params.py)
param_names, param_xform, true_params = build_params(
["beta", "alpha_self", "alpha_other", "alpha_noone"], nsubjects
)
# use EMModel.recover to run simulation, fitting and recovery metrics
model = EMModel(all_data=None, fit_func=rw3a1b_model.fit,
param_names=param_names,
param_xform=param_xform,
simulate_func=rw3a1b_model.sim)
recovery = model.recover(true_params,
pr_inputs=['choices','rewards'],
nblocks=nblocks, ntrials=ntrials
)
# scatter plot of recovered parameters
fig = model.plot_recovery(recovery)
recover(): this model's data and outfit now reflect the recovery fit; the recovered estimates are also in the returned dict['recovery_model'].
Dual-Outcome Social Learning Task (Rhoads et al., 2025, Nature Communications) - 1Q4α1β Model¶
Participants learn to make choices to earn points for themselves (self) and a stranger (other) simultaneously. Outcomes include gaining points (+100), losing points (-100), and receiving no points (0), which are later converted to real-world payouts.
Possible options yield the following types of outcomes probabilistically (independently assigned to self and other targets):
- A: ++ Mutual Benefit (self gain, other gain)
- B: +- Instrumental Harm (self gain, other loss)
- C: -+ Altruism (self loss, other gain)
- D: -- Mutual Cost (self loss, other loss)
In the two-option version of the task, agents learn to choose between each pair of options, $4!/(2! * (4-2)!) = 6$ pairs
The model is an adapated Rescorla Wagner model with four different learning rates $\alpha_i^\ast$ for different targets (self, other) and valence (positive, negative), and predicts how agents learn how their actions affect dual outcomes (self, other) across mutually beneficial, instrumentally harmful, altruistic, and mutually costly scenarios:
$$ \textbf{Expected value update:}\quad Q_{t+1}^k = Q_t^k + \alpha_i^\ast \cdot \delta_t \tag{1} $$
$$ \textbf{Prediction error:}\quad \delta_t = r_t - Q_t^k \tag{2} $$
$$ \textbf{Learning rates:}\quad \alpha_i^\ast = \begin{cases} \alpha_{\text{self}}^{+}, & \text{if } i=\text{self}, \delta_t \ge 0\\ \alpha_{\text{other}}^{+}, & \text{if } i=\text{other}, \delta_t \ge 0\\ \alpha_{\text{self}}^{-}, & \text{if } i=\text{self}, \delta_t < 0\\ \alpha_{\text{other}}^{-}, & \text{if } i=\text{other}, \delta_t < 0 \end{cases} \tag{3} $$
$$ \textbf{Choice rule (softmax):}\quad p_t(\text{choose }k)=\frac{\exp(\beta \cdot Q_t^k)}{\sum_{k=1}^{K}\exp(\beta \cdot Q_t^k)} \tag{4} $$
from pyem.models.rl_mf import rw4a1b_model
print(f"Model ID : {rw4a1b_model.id}")
print(f"Description: {rw4a1b_model.desc}")
print(f"Spec : {rw4a1b_model.spec}")
Model ID : rw4a1b
Description: Four-option task where each trial shows a pair of options;
one shared inverse temperature and four learning rates split by outcome
recipient (self/other) and valence (positive/negative) (Rhoads et al., 2025).
Free parameters: beta, alpha_self_pos, alpha_self_neg, alpha_other_pos, alpha_other_neg.
Spec : {'rl': {'softmax': ['beta'], 'rw': ['alpha_self_pos', 'alpha_self_neg', 'alpha_other_pos', 'alpha_other_neg']}}
# simulate computer agents completing the prosocial learning task (Rhoads et al., 2025)
nsubjects, nblocks, ntrials = 50, 12, 20
# generate "true" parameters using the shared parameter registry (examples/params.py)
param_names, param_xform, true_params = build_params(
["beta", "alpha_self_pos", "alpha_self_neg", "alpha_other_pos", "alpha_other_neg"], nsubjects
)
# use EMModel.recover to run simulation, fitting and recovery metrics
model = EMModel(all_data=None, fit_func=rw4a1b_model.fit,
param_names=param_names,
param_xform=param_xform,
simulate_func=rw4a1b_model.sim)
recovery = model.recover(true_params,
pr_inputs=['choices','outcomes_self','outcomes_other','option_pairs'],
nblocks=nblocks, ntrials=ntrials
)
# scatter plot of recovered parameters
fig = model.plot_recovery(recovery)
recover(): this model's data and outfit now reflect the recovery fit; the recovered estimates are also in the returned dict['recovery_model'].
References
Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction.
Rescorla, R. A. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and non-reinforcement. Classical conditioning, Current Research and Theory, 2, 64-69.
Lockwood, P. L., Apps, M. A., Valton, V., Viding, E., & Roiser, J. P. (2016). Neurocomputational mechanisms of prosocial learning and links to empathy. Proceedings of the National Academy of Sciences, 113(35), 9763-9768. https://doi.org/10.1073/pnas.1603198113
Rhoads, S. A., Gan, L., Berluti, K., OConnell, K., Cutler, J., Lockwood, P., & Marsh, A. (2025). Neurocomputational basis of learning when choices simultaneously affect both oneself and others. Nature Communications. 16, 9350. https://doi.org/10.1038/s41467-025-64424-9