A survey on deep reinforcement learning for human-robot interaction

1. INTRODUCTION

Human-robot interaction (HRI) control spans two coupled layers: a physical layer, in which a robot exchanges forces, positions, and motion with a human partner in direct or teleoperated contact, and a cognitive/social layer, in which the robot must interpret intent, adapt to preferences, and coordinate actions with a human whose behaviour is only partially predictable. Classical solutions to the physical layer, impedance and admittance control^[1], render the robot’s response to contact forces through a fixed or scheduled mechanical relationship. These schemes are provably stable, but they are built on a model of the environment (and implicitly, of the human) that is rarely accurate: human arm stiffness, reaction time, and intent all vary across users and across a single task^[2].

Deep reinforcement learning (DRL) offers an alternative: rather than hand-specifying an interaction law, the robot learns a policy directly from interaction data, allowing the impedance/admittance behaviour itself to become adaptive and human-specific^[3,4]. This shift from designed to learned interaction control is the central subject of this survey.

The shift is not free, and the price is paid in a currency that HRI can least afford. A learned interaction law is a black box placed in mechanical contact with a person, so the two questions that classical control answered by construction, will it remain stable and will it remain safe, must now be answered statistically, from finite data, about a policy that continues to change after deployment. Three consequences organise this survey. First, sample efficiency stops being an engineering convenience and becomes an ethical constraint, because every sample is a physical interaction with a human body; this is what makes demonstration data, offline pre-training, and simulation transfer central rather than optional. Second, the environment is non-stationary in a specific and awkward way: the human adapts to the robot while the robot adapts to the human, so the stationarity assumption underlying the Markov decision process is violated by the very success of the learning process. Third, guarantees must be stated in a form that survives learning, which is why uncertainty quantification and set-invariance methods have moved from the periphery of this literature to its centre in the last three years.

The same pattern, a perception-decision-action loop whose reliability must be certified because a person is on the receiving end, is visible well beyond manipulation. Reviews of embodied AI in clinical settings organise the field along exactly this axis and reach a similar conclusion about the translation bottleneck, namely that the barrier is evaluation and standardisation rather than raw capability^[5]. We draw on that parallel where it is informative, while keeping the scope of this survey on physical and cognitive HRI control.

Four developments justify a new survey rather than a simple update of our 2021 book Human-Robot Interaction Control Using Reinforcement Learning^[6]: (i) safe and stability-certified RL has matured from Lyapunov-style heuristics into formal H₂/uniform ultimate boundedness (UUB) guarantees^[7]; (ii) conformal-prediction-based uncertainty quantification has entered control design^[8]; (iii) large language and vision-language-action models now condition manipulation policies on natural-language instruction^[9,10]; and (iv) residual and hybrid reinforcement learning (RL), learning a correction on top of a model-based controller rather than a policy from scratch, has become the dominant paradigm for sample-efficient, safety-preserving deployment^[4,11].

Table 1 organises the literature underlying this survey into nine paradigms that trace the field’s progression from classical, fixed-law HRI control to certified and language-conditioned DRL. This classification also fixes the vocabulary used throughout the sections that follow.

The remainder of this survey is organised as follows. Section 2 bridges classical HRI control and DRL. Section 3 proposes a four-part classification of DRL approaches for HRI. Section 4 discusses emerging directions, Section 5 lists open challenges, and Section 6 concludes.

2. FROM CLASSICAL HRI CONTROL TO DEEP RL

Admittance and impedance control regulate the dynamic relationship between interaction force F_h (applied by the human) and robot motion. A standard admittance model relates force to a commanded position correction x_d through a virtual mass-damper-spring system,

where M, D, and K are the designer-chosen virtual inertia, damping, and stiffness^[1]. Stability of Equation (1) is straightforward when F_h is bounded and slowly varying, but M, D, K are typically fixed offline; they do not adapt to a specific human’s stiffness, fatigue, or intent. A large body of work addresses this by scheduling the gains online: variable admittance schemes modulate D from measured interaction power or estimated user intent^[12,13], and variable impedance learning treats the stiffness profile itself as the object to be adapted^[3,14]. In all of these the interaction law still relies on a parametric structure fixed at design time.

Reinforcement learning removes that structural assumption. The HRI problem is cast as a Markov decision process ($$ \mathcal{S} $$, $$ \mathcal{A} $$, P, r, γ), where the state s_t ∈ $$ \mathcal{S} $$ includes robot and (where observable) human kinematic/force data, the action a_t ∈ $$ \mathcal{A} $$ is a motion or force command, and the policy π_θ(a_t | s_t) is trained to maximize expected discounted return,

with the reward r(s_t, a_t) typically shaped to penalize excessive interaction force, task error, and unsafe configurations. Three properties of the human partner make Equation (2) attractive over the fixed model in Equation (1):

• Unknown human dynamics. The mapping from robot behaviour to human response is rarely known in closed form; RL treats it as part of the environment transition P rather than requiring an explicit model. Evidence from human motor control supports this treatment: people co-adapt force and impedance rather than holding a fixed mechanical law^[15], so the mapping being learned is genuinely dynamic rather than merely unknown.

• Non-stationarity. Human stiffness, attention, and intent drift within and across sessions. Model-free policies can be updated online as this drift occurs, whereas fixed-gain admittance laws [Equation (1)] cannot.

• Contact uncertainty. Under unknown or time-varying environment constraints, residual strategies that learn a correction on top of a nominal force controller^[4] and demonstration-guided exploration^[16,17] have been shown to recover stable behaviour faster than learning from scratch.

Figure 1 illustrates this shift schematically: the classical loop (dashed) computes x_d directly from F_h via Equation (1); the DRL loop (solid) instead routes (s_t, F_h) through a learned policy π_θ, whose parameters are updated from the interaction reward, closing the loop at the learning level rather than only at the control level.

Figure 1. Classical admittance control (dashed) vs. DRL-based HRI control (solid). In the classical loop the human force F_h is the input to the fixed law [Equation (1)], whose output is the commanded correction x_d; the DRL loop replaces that fixed map with a policy π_θ updated from the interaction reward. DRL: Deep reinforcement learning; HRI: human-robot interaction.

3. DEEP RL APPROACHES FOR HRI

We organise the DRL-for-HRI literature into four sub-categories, summarised in Table 2 and illustrated in Figure 2, that differ in what is being learned (a control law, a policy from data, a joint human-robot strategy, or a certified policy) rather than in application domain. The four categories build on the general DRL-for-robotics methodology established by policy-search and end-to-end visuomotor formulations^[31,32], which they specialise to the case of a human in the loop.

Figure 2. Classification of DRL approaches for HRI. Consistent with the 3+1 reading developed in the main text, categories 3.1-3.3 answer where human information enters and are drawn as siblings, whereas category 3.4 is not a fourth sibling but a certification layer (dashed band) that any of the other three can carry. Corresponds to Table 2. DRL: Deep reinforcement learning; HRI: human-robot interaction.

The four categories are not disjoint, and their overlaps are significant for design. Demonstration learning (Section 3.2) and shared autonomy (Section 3.3) differ mainly in when the human’s influence enters: in the former the human supplies an offline dataset or a preference label that shapes r before deployment, whereas in the latter the human is an agent whose action a_t^h appears inside the value function at run time. The boundary blurs as soon as demonstrations are collected online, at which point a demonstration-learning system is a shared-autonomy system with a particular arbitration rule. Safe RL (Section 3.4) is better read as orthogonal to the other three than as a peer of them: it is a constraint layer that can be applied to any of them. Concretely, demonstration data is what makes a safety filter tractable, because it supplies the calibration set that a conformal bound [Equation (7)] needs^[28], while shared autonomy is what makes a safety filter necessary, because a non-stationary human partner is precisely the disturbance that a fixed-margin barrier cannot absorb^[29]. Reading the four categories as a 3+1 structure, three answers to “where does human information enter” plus one certification layer, explains why the most deployment-ready systems in Table 3 combine categories rather than choosing between them.

Three observations follow from Table 3.

Deployment readiness runs inverse to conceptual ambition. The category that assumes least about the human, admittance shaping, is the one actually running on industrial hardware, while the category that models the human most richly, shared autonomy, has produced almost no deployed systems. This is not a transient state of maturity. Modelling the human as a rational agent buys expressiveness at the cost of an assumption that cannot be validated on the small cohorts these studies use, so the added realism is difficult to falsify in practice. In selecting a method for a given application, the assumptions listed in the second column are therefore the appropriate starting point.

The assumptions fail together, not independently. Each row’s assumption is violated by the same underlying fact, that the human adapts. Non-stationarity invalidates the fixed gains of 3.1, shifts the demonstration distribution of 3.2, breaks the stationary-policy premise of 3.3, and destroys the exchangeability that the conformal certificates of 3.4 require. Combining categories therefore does not average the risk away, because the failures are correlated: a system that pairs offline demonstrations with a conformal safety filter has two components that become miscalibrated at the same moment and for the same reason. Reported robustness gains from combining methods should be read with this in mind.

The data requirements in column four are not commensurable, which makes published comparisons unreliable. A demonstration costs human time, a preference label costs human attention, and a simulated rollout costs neither. Sample counts are nonetheless reported in a single currency, so a method requiring 10⁴ simulated episodes is routinely described as less data-hungry than one requiring 10² human comparisons, when for a rehabilitation patient the ordering is reversed. Until the field reports human-cost separately from environment-interaction count, claims of improved sample efficiency in HRI are not comparable across papers. We regard this as a more tractable near-term fix than any algorithmic change, and as a concrete instance of the benchmarking challenge discussed in Section 5.

4. EMERGING DIRECTIONS

Three directions extend the classification of Section 3 beyond what was addressed in^[6] and are summarised in Table 4.

Large language model (LLM)/Vision-language-action (VLA)-conditioned policies. Vision-language-action models such as RT-2^[9] condition a manipulation policy directly on a natural-language instruction and an image observation, π_θ(a_t | s_t, ℓ), where ℓ is the instruction embedding. Combined with task-and-motion planning^[10], this allows an HRI system to accept unstructured verbal requests rather than a single fixed reward function, a qualitative departure from every method in Section 3, all of which assume a task-specific r fixed at training time. The trajectory here is not unique to robotics: medical vision-language analysis has passed through the same three stages, from task-specific models, through adapter- and prompt-tuned variants, to generalist foundation models, and the reported gains there are in data efficiency and cross-domain generalisation rather than in peak single-task accuracy^[38]. That parallel is worth taking seriously as a prediction: if manipulation follows the same curve, the argument for VLA-conditioned policies in HRI will rest on few-shot adaptation to a new user, not on outperforming a well-tuned task-specific controller on the task it was tuned for.

Digital-twin-based sim-to-real transfer. Real-world DRL training in physical contact with a human is costly and risky; a digital twin of the human-robot-environment system allows π_θ to be pre-trained under Equation (2) in simulation, with the safety mechanisms of Section 3.4 used to certify the policy before, rather than during, real-world deployment. The transfer techniques this relies on, domain randomisation, system identification, and progressive adaptation, are surveyed in^[37]; surveys of real-world DRL deployment in robotics consistently flag the residual sim-to-real gap in human-facing settings as an open bottleneck^[33], motivating this direction as a priority rather than a solved problem.

4.1. Data-efficient and offline learning for HRI

The methods of Section 3 share a dependence that Table 3 makes explicit: they need interaction data, and in HRI every sample is a physical interaction with a person. This section covers the four bodies of work that attack that constraint directly.

Offline RL is the structurally indicated response, not an optional extension. If human-in-the-loop rollouts cannot be reset or parallelised, then learning must proceed from a fixed dataset of previously logged interactions; this is precisely the offline RL setting^[39,40]. The central difficulty there, distributional shift, is that a policy trained on logged data queries the value function at actions the behaviour policy never took, and the resulting overestimation is unrecoverable without online correction. Conservative and constraint-based estimators address this by penalising out-of-distribution actions^[40]. HRI aggravates the problem along an axis the offline RL literature does not treat: the shift is not only between behaviour policy and learned policy, but between the logged human and the deployed human. A dataset collected from one cohort encodes that cohort’s stiffness, reaction latency, and intent distribution, so an offline HRI policy is conservative with respect to the wrong reference. This is the generalisation-across-users challenge of Section 5 restated in offline terms, and it suggests that per-user conservatism, rather than per-action conservatism, is the appropriate regulariser. We are not aware of an offline RL method formulated this way.

Data efficiency beyond offline learning. Three of the mechanisms already surveyed are data-efficiency mechanisms in disguise: demonstrations and preferences replace environment interaction with human supervision (Section 3.2), sim-to-real transfer replaces it with simulated interaction^[37], and residual RL replaces it with a nominal controller that is already approximately correct [Equation (8)]. Their limitations are complementary rather than shared, which is why they compose: imitation fails off-distribution, simulation fails where contact dynamics are mismodelled, and residual formulations fail when the nominal controller is badly wrong. What has been missing is measurement. Benchmarks that evaluate offline methods on real robot hardware rather than in simulation^[41] report substantially smaller gains than simulated benchmarks suggest, which is a caution worth carrying into HRI, where the gap between simulated and real interaction is wider still because the human is the least faithfully simulated component.

Foundation models change what counts as a data requirement. The VLA policies of Section 4 are trained on cross-embodiment corpora aggregated across many robots and laboratories^[42], so the marginal cost of a new task shifts from collecting a task-specific dataset to specifying an instruction. For HRI, the interesting claim is not peak task performance but few-shot adaptation to a new user, which is the axis on which the methods of Section 3 generalise worst. The same three-stage progression, from task-specific models through adapter-tuned variants to generalist foundation models, has already played out in medical vision-language analysis, where the reported gains were likewise in data efficiency and cross-domain transfer rather than in single-task accuracy^[38]. Whether physical interaction follows that curve is open: contact dynamics are not obviously compressible in the way that visual semantics are.

Uncertainty-aware learning. The fourth strand is treated at length in Section 3.4, where conformal prediction sets and their composition with barrier functions and predictive safety filters are surveyed^[26-30]. We note here only the connection to the present section: a conformal guarantee is a statement about a calibration set, so it inherits exactly the coverage limitations of offline data discussed above. An offline HRI policy and its conformal certificate can be miscalibrated for the same reason and at the same time, which is a failure mode that neither literature currently isolates.

Residual RL for hybrid control. Rather than learning π_θ from scratch, residual RL composes a nominal model-based action a_t^model {e.g., an admittance law [Equation (1)] or an inverse-dynamics controller} with a learned correction,

(8) $$ a_t = a_t^{\mathrm{model}} + \Delta a_t^{\mathrm{RL}}(s_t), \qquad\Delta a_t^{\mathrm{RL}} \sim \pi_\theta(\cdot \mid s_t), $$

which preserves the nominal controller’s stability properties when Δa_t^RL is small, while still allowing the policy to compensate for unmodeled human or contact dynamics. Figure 3 shows the resulting architecture, in which the online constraint [Equation (7)] is applied to the composed action before it reaches the plant. The formulation was introduced for contact-rich robot control in^[4,11], and we expect it to be the most practical bridge between the certified guarantees of Section 3.4 and the flexibility of end-to-end DRL.

Figure 3. Residual RL architecture [Equation (8)], combining a nominal model-based controller with a learned correction. Only the set-invariance/conformal constraint [Equation (7)] is executable as an online filter that modifies a_t; the UUB bound [Equation (6)] is a property of the resulting closed loop established by analysis, not a block in the signal path, and is therefore shown separately. RL: Reinforcement learning; UUB: uniform ultimate boundedness.

5. OPEN CHALLENGES

Despite the progress organised in Tables 1 and 2, five challenges remain largely open across the DRL-for-HRI literature we surveyed. Table 5 summarises them; each is discussed below.

Each challenge below is stated as a specific unresolved question rather than a topic, together with what would count as an answer. Several were surfaced by the comparative analysis of Section 3 and are, to our knowledge, not posed in this form elsewhere.

C1. Is per-user conservatism the right regulariser for offline HRI? Offline RL penalises actions outside the behaviour distribution^[39,40], but Section 4.1 argued that the shift that matters in HRI is between the logged human and the deployed human, not between behaviour and learned policy. The concrete question: can a conservatism penalty be defined over an estimated user-parameter posterior, for instance over arm stiffness and reaction latency, rather than over actions? An answer would be an offline method whose pessimism relaxes as the deployed user’s estimated parameters approach the logged cohort, and it could be evaluated today on existing real-robot offline benchmarks^[41] extended with multi-user data.

C2. Can conformal coverage be made valid in closed loop? Section 3.4 showed that conformal sets now compose with set-invariance guarantees in HRI^[26-29] but not with Lyapunov/UUB stability, because the robot’s action changes the human’s next motion and so destroys exchangeability. The specific open problem is to bound the disturbance term in $$ \dot{V} $$(e_t) using a coverage statement that remains valid under this feedback. Adaptive conformal prediction^[27] shows one route, abandoning fixed calibration; a second, untried route is to calibrate on interventional rather than observational data, that is, on trajectories in which the robot’s action was deliberately randomised. The answer would be a UUB bound whose ε in Equation (6) carries an explicit, finite-sample confidence level.

C3. Do the four categories fail independently? The synthesis following Table 3 argued they do not: human adaptation invalidates the assumptions of all four simultaneously. If correct, this predicts that a system pairing offline demonstrations with a conformal safety filter will show correlated degradation, both components decalibrating within the same session. This is directly testable, and to our knowledge untested: instrument a combined system, log calibration error and policy regret separately, and measure their correlation across a session in which the user adapts. A near-zero correlation would falsify the claim and justify treating the categories as independent risk reducers; a high correlation would mean that published robustness gains from combining methods are overstated.

C4. How many users, of what diversity, are enough? Most reported results are single-user or small-cohort^[12,22]. The unresolved question is not qualitative but quantitative: what is the sample complexity of user generalisation? A concrete first step is a scaling study holding the algorithm fixed and varying cohort size and diversity, reporting held-out-user performance as a curve rather than a single number. Without such a curve there is no principled basis for the cohort sizes currently used, which appear to be set by recruitment convenience.

C5. Report human cost separately from interaction count. As argued after Table 3, demonstrations, preference labels and simulated rollouts are aggregated into a single sample count even though they draw on incommensurable resources, so a method needing 10⁴ simulated episodes is described as more efficient than one needing 10² human comparisons. This item requires no new algorithm. We propose that HRI-DRL papers report a three-element cost vector: environment interactions, human minutes, and number of distinct participants. Unlike a full benchmark suite, which requires community coordination^[33,35], this convention can be adopted by an individual paper without prior agreement across the field.

6. CONCLUSION

This survey organised DRL for HRI into four paradigms, physical HRI via admittance shaping, learning from demonstration and feedback, multi-agent and shared-autonomy control, and safe, stability-certified RL, and traced their evolution from the fixed-law control reviewed in our 2021 book^[6] to the certified, language-conditioned, and residual formulations emerging today. Relative to recent DRL-robotics surveys in which HRI appears as one application area among many^[33], and to surveys addressing a specific HRI setting or a specific communication channel rather than control-theoretic guarantees^[34,35], we focused specifically on what a robot can be guaranteed to do while interacting with a human. Section 3.4 argues that the residual weakness is not the absence of distribution-free uncertainty quantification in HRI, which now exists^[26-29], but its confinement to set-invariance rather than stability guarantees.

Three directions are, in our view, the most consequential for closing that gap: reconciling conformal-prediction coverage^[8], which is now routinely composed with set-invariance guarantees in HRI^[26-29], with Lyapunov/UUB stability results^[7] into a single certification framework, which requires a coverage argument that survives the loss of exchangeability in closed loop; using digital twins to move safety validation from real-world trials into simulation before deployment; and adopting residual RL [Equation (8)] as a default architecture, since it inherits the guarantees of a nominal controller while retaining the adaptability that motivated moving beyond classical admittance control [Equation (1)] in the first place. We hope the classification and open challenges identified here [Tables 2 and 5] provide a concrete agenda for both the RL and HRI communities.