Bio-integrated systems for silent speech recognition: from advanced bioplatforms to machine learning-assisted biosignal decoding

EMG-based SSIs

EMG-based SSIs capture biopotentials of articulatory muscles during speech production [Figure 1A]. Figure 3A illustrates the working principle of EMG-based SSIs. Skin-mounted EMG electrodes acquire biopotential signals induced by facial and neck muscular activities associated with silent speech^[43,65]. In typical bipolar electrode configurations^[66], two recording electrodes (one positive and one negative) are placed on the target muscles to measure the voltage difference between them. A third electrode, fixed to a nearby bony area such as the collarbone, serves as a reference ground and provides a stable baseline^[22]. Alternatively, monopolar electrode configurations^[66] employ a single active (positive) electrode placed on the target muscle and a negative electrode on a nearby bony area. This setup occupies less space on the skin surface, allowing more channels to be placed. To understand EMG-based SSI implementation, it is essential to first examine the nature of the signals being captured. Each muscle fiber produces a brief, spike-shaped action potential. The measured EMG signal is a summation of signals from all muscle fibers within the area beneath the active electrode. During silent speech, these signals oscillate around a zero baseline with a maximum amplitude generally below 1 mV and a frequency band under 500 Hz^[67,68], providing rich signal patterns that can be decoded into speech content. The acquired raw EMG signals are generally amplified by a differential amplifier. Subsequently, these signals are processed through a bandpass (anti-aliasing) filter, typically spanning 20 to 500 Hz^[67,68]. This stage is critical for removing high-frequency noise and ensuring that the signal bandwidth conforms to the Nyquist-Shannon sampling theorem before digitization, thereby preventing aliasing. An analog-to-digital (A/D) converter finally converts the signals to digital output for further processing.

Figure 3. EMG-based SSIs. (A) Working principle of EMG-based SSIs; (B) Dry electrode made by gold thin films for SSIs. Adapted from Ref. [63], under CC BY 4.0 license; (C). Dry electrode based on AgNWs for SSIs. Adapted with permission from Ref. [5] Copyright 2023, Wiley-VCH; (D) Wet electrode based on ion gels for SSIs. Adapted with permission from Ref. [64] Copyright 2023, Wiley-VCH. EMG: Electromyography; SSIs: silent speech interfaces; AgNWs: silver nanowires; Difff. Amp.: differential amplifier; PET: polyethylene terephthalate; SEM: scanning electron microscope.

The effectiveness of EMG-based SSIs is fundamentally limited by the quality of the electrode-skin interface, driving continuous innovation in electrode materials and designs. Traditional approaches have relied on commercially available electrodes, including gold-plated cup dry electrodes, Ag/AgCl dry electrodes, and pre-gelled wet electrodes^[69-71]. Among these options, pre-gelled electrodes often provide superior signal quality due to the low electrode-skin impedance. However, they suffer from several limitations for long-term use, including signal degradation caused by gel dehydration, skin irritation triggered by the conductive gel, and bulkiness and visibility on the skin surface^[72]. To address these concerns, researchers have developed soft gel-free dry electrodes that prioritize both long-term performance and user comfort^[73-76]. To ensure mechanical compatibility with biological tissues, these systems employ materials with properties that closely match those of skin or neural tissues, including elastomers such as PDMS (Young’s modulus ~1-3 MPa^[77]). In addition, soft materials, such as hydrogels, are frequently selected for their high stretchability (up to 1,000%) and low bending radii. These characteristics minimize mechanical constraints on natural movements^[78]. The developed electrodes can seamlessly conform to the morphology of the skin, even under skin deformations during speech. A representative example is shown in Figure 3B, where thin gold film electrodes have been used for EMG sensing^[63]. The thickness of the gold thin film is 1.2 µm, and the film is supported by a 47-µm Tegaderm patch [Figure 3B, Structure]. The electrode array can achieve conformal contact with the skin texture, as confirmed in the SEM images [Figure 3B, Interface SEM]. Conformal contact is essential for high-fidelity EMG signal acquisition, as this can minimize the interface gap between the electrode and skin, reduce electrode-skin impedance, and alleviate interference from motion artifacts^[5,79].

Although gold thin-film electrodes improve user comfort, they remain opaque and visibly noticeable on facial skin. This visibility challenge has motivated the development of EMG electrodes with high transmittance^[80]. Specifically, electrodes based on silver nanowires (AgNWs) provide good transmittance, in addition to high conductivity and excellent stretchability^[5]. Figure 3C illustrates the unobtrusive appearance of an AgNW-based electrode on human skin. Owing to the large aspect ratio of nanowires, a low-density AgNW network can maintain conductive pathways via overlapping nanowires, which allows most visible light to pass through [Figure 3C, AgNW SEM]^[81]. Notably, when placed around the mouth and neck muscles, AgNW electrodes are able to detect subtle muscle activities during silent speech. Assisted by ML-based speech recognition models, the system can differentiate between words with similar visemes, such as “ship” and “sheep”^[5], by capturing fine articulatory differences that are visually indistinguishable but electrically distinct. Ionogel-based electrodes represent another approach to achieving unobtrusive EMG sensing. The soft and transparent gel forms a continuous ionic pathway and remains transparent in the visible spectrum^[82]. Figure 3D presents an example of SSIs enabled by ionogel electrodes. The iongel can establish conformable contact with the skin, ensuring low impedance and high signal quality^[64]. Practically, the electrode array connects to a compact circuit board that can be conveniently placed in a chest pocket for everyday use. Instead of relying on skin patches, electrodes can also be built directly into everyday accessories. For example, recent designs hide graphene-coated textile electrodes inside headphone earmuffs to read muscle signals completely unnoticed^[83].

EMG electrodes described above must be paired with appropriate data acquisition systems to realize their full potential. The performance of EMG-based SSIs also depends on the data acquisition device. For research and clinical settings, bench-top systems such as ADInstruments PowerLab with built-in filters offers high precision acquisition of EMG signals, with sampling frequencies up to several tens of kilohertz^[84]. Portable devices are also available for daily applications. For instance, the OpenBCI Cyton streams data at 250 Hz via Bluetooth and up to 1 kHz via Wi-Fi, with a lightweight, pocket-sized form factor^[85]. Another portable system, the Delsys Trigno Quattro, comprises built-in filters, four EMG channels, and wireless transmission, with sampling frequencies up to 2.2 kHz^[86]. Sampling rate is the key to capturing precise information from EMG signals. The informative frequency band of speech-relevant EMG signals is generally below 500 Hz. According to the Nyquist-Shannon sampling theorem, to accurately capture these signals without distortion, the sampling rate must be at least twice the highest frequency present^[87,88]. Therefore, a minimum sampling rate of 1 kHz is necessary to prevent aliasing and ensure accurate signal reconstruction. This requirement is crucial when selecting data acquisition devices for EMG applications, as inadequate sampling rates can lead to the loss of critical information and misinterpretation of muscle activity. Therefore, choosing a device often involves balancing precision and portability. While traditional setups rely on bench-top instruments or portable rigid-printed PCB devices to digitize signals, these rigid components heavily restrict overall wearability and create a severe mechanical mismatch with soft epidermal sensors. To overcome this limitation, recent academic research has shifted toward deploying fully integrated, flexible PCB (fPCB) data acquisition (DAQ) systems^[89]. By matching the mechanical compliance of human skin and eliminating the need for bulky hardware, this structural integration ensures long-term attachment stability and minimizes motion artifacts typically induced by rigid components^[90].

EMG-based SSIs are highly promising for daily use due to their noninvasive nature, compatibility with soft, skin-conformal electrodes, and ability to operate reliably in various conditions, including low-light and noisy environments. Advances in dry stretchable electrodes (such as those enabled by serpentine-patterned gold films, AgNWs, and ionogels) have greatly improved the wearability, user acceptance, and signal quality of EMG electrodes^[5,63,64]. Despite these advances, several technical challenges should be addressed to enhance the long-term deployment. Future research should focus on long-term robustness to tackle problems related to sweat accumulation, instabilities caused by repeated removal and reattachment, and signal variations caused by motion artifacts. From a system integration perspective, efforts should also be devoted to minimizing the size of data acquisition components, improving the flexibility of integrated systems, and reducing power consumption. The path forward requires coordinated advances across multiple domains. Advancements in low-power amplifiers, wireless transmission techniques, data compression methods, and optimized compact designs, can help minimize power consumption and device size without compromising signal quality. These hardware improvements, combined with continued electrode innovation and signal processing advances, position EMG-based SSIs as one of the most promising approaches for practical SSR systems.

EEG-based SSIs

Please note that we limit our discussion of EEG-based SSIs to the assessment of EEG signals alone as a sensing modality. We acknowledge that strategies such as multimodal fusion and mixed training paradigms can enhance overall brain-computer interface (BCI) performance and system-level accuracy. As illustrated in Figure 4A, EEG-based SSIs measure brain electrical activities in the conceptual preparation stage of the speech generation process [Figure 1A]. Similar to EMG, EEG captures biopotential signals using either bipolar or monopolar configurations. EEG electrodes are placed on the scalp to monitor brain activity rather than on muscles. A common placement follows the international 10-20 system, which standardizes electrode positions over key brain regions^[95]. The neural signals originate from cortical neurons and spread through brain tissues via volume conduction. Subsequently, the signals travel through structures such as the skull and scalp to the scalp electrode via capacitive conduction^[96]. These weak signals, typically ranging from 1 to 100 µV in adults^[97], are then amplified by a differential amplifier, filtered through a band-pass filter, and digitized by an A/D converter. This signal conditioning process is similar to that used for EMG but is adjusted for lower amplitudes and narrower bandwidths. More details on EEG data acquisition devices can be found in the previous review paper^[98]. The resulting EEG signals are generally decomposed into five frequency bands: delta (< 4 Hz), theta (4-8 Hz), alpha (8-12 Hz), beta (12-30 Hz), and gamma (> 30 Hz)^[99]. In SSI research, delta and theta bands are typically analyzed because their slower rhythms are better aligned with the pacing of phrases and syllables during speech, offering stronger links to speech intention. Some studies also use gamma band activities to capture finer details and improve recognition accuracy^[100-102].

Figure 4. EEG-based SSIs. (A) Working principle of EEG-based SSIs. The simplified EEG interface in the middle part of (A): Adapted from Ref. [91], under CC BY 4.0 license. EEG signals in the right part of (A): Adapted from Ref. [92], under CC BY 4.0 license; (B) SSIs enabled by scalp EEG and around-ear EEG. Scalp EEG: Adapted from Ref. [93], under CC BY 4.0 license. Around-ear EEG: Adapted from Ref. [94], under CC BY 4.0 license. EEG: Electroencephalography; SSIs: silent speech interfaces.

While EEG sensors offer a non-invasive approach, these signals typically undergo severe attenuation and reduced spatial resolution due to volume conduction through the skull. Despite these physical limitations, recent studies demonstrate emerging potential for EEG-based SSR. Early work successfully decoded vowel imagery using EEG and extreme learning machines^[93]. Figure 4B shows the topographical distribution of gamma-wave activity in one subject during vowel imagery. Enhanced activity was observed in both temporal regions when the subject imagined vowels. More recently, researchers achieved silent speech decoding utilizing around-ear EEG combined with large-scale training data^[94], as shown in Figure 4B. Furthermore, this domain is advanced by pretraining large brain language models specifically for active EEG-based BCIs^[103]. Capturing these highly attenuated signals requires robust sensor interfaces.

To obtain high-fidelity EEG signals, a typical setup requires the participant to wear a cap [Figure 4A] on the head. Electrodes are mounted on the cap to measure biopotentials from the scalp^[104-107]. A chin rest is often necessary since it can help reduce the motion artifacts in the signals. Similar to EMG, the electrode design in EEG is crucial. Commercial pre-gelled electrodes used in EMG can often be applied to EEG. Additionally, snap electrodes with a comb structure are also widely used for EEG, since they can form better contact with the scalp, especially with the interference of hair^[108,109]. However, the limited contact area of comb tips can result in higher electrode-skin impedance, potentially reducing signal quality. Recent developments have explored bio-gels to paint or print electrodes directly onto the skin, offering an effective solution for maintaining electrode-skin contact in hairy regions^[110,111]. The bio-gel offers several beneficial features, including strong adhesion, rapid curing, and easy removal, while also being skin-friendly and environmentally sustainable.

Apart from gel electrodes, gel-free dry electrodes have been explored to address concerns with conductive gel in commercial pre-gelled electrodes. One system based on dry electrodes demonstrates basic command-based control, such as ‘stop’ and ‘forward,’ for wheelchair users^[112]. Another example utilizes a similar design but with different electrode attachment positions^[113]. This system comprises a wireless earbud-like EEG device with three electrodes (working (positive), reference (negative), and ground) placed on the facial skin. Compared to commercial EEG devices, the earbud-like EEG device shows better signal quality with higher SNRs^[113]. Although the above two systems have not yet been directly applied to silent speech analysis, they share the same goal as SSIs - that is, to interpret people’s attempts and intentions. However, such interfaces lack sufficient electrodes and signal channels for achieving a more comprehensive SSR.

EEG-based SSIs face significant challenges in both comfort and signal quality. Traditional headsets are often bulky and uncomfortable, making them difficult to wear for long periods. While reducing the number of electrodes and the coverage area can enhance comfort, this approach typically compromises performance, given the fact that the speech recognition accuracy is limited even with high-density electrode configurations. EEG signals are highly vulnerable to interference, particularly interference from non-speech-relevant brain activities, such as cognitive distractions and mental wandering. These fundamental limitations, combined with substantial inter-subject variability and session-to-session variations, create significant barriers to achieving the stable and practical performance required for continuous use.

IMU-based SSIs

IMU-based SSIs track changes in articulator movements (e.g., speech-induced jaw and skin movements) during the speech generation process [Figure 1A and Figure 5A]. One IMU unit, typically in the form of a small circuit board, can provide up to 9 degrees of freedom (9DOF) by combining a tri-axis accelerometer, gyroscope, and magnetometer^[116]. The accelerometer and gyroscope are often used together for SSIs, while the magnetometer is usually used in a different setup because it requires external permanent magnets. For the accelerometer and gyroscope-based SSIs, the IMU is typically attached to the neck, jaw, or facial skin using medical tape or a soft adhesive elastomer^[47]. This allows the IMU to move together with the skin, transferring skin movements during silent speech through the IMU-skin interface. The accelerometer records subtle changes in acceleration, and the gyroscope detects changes in angular velocity. These signals are converted into digital signals that reflect articulatory movements. Unlike the accelerometer and gyroscope-based approach, in a magnetometer-based SSI, the sensor is placed on a bony area with negligible skin movements during speech, and a small permanent magnet is attached to the skin where deformation occurs^[115]. The magnetometer detects changes in the magnetic field caused by the magnet’s movement. These approaches capture low-frequency (about 1-20 Hz^[117]) deformations related to silent speech. The signals are easily affected by head movements and changes in the geomagnetic field.

Figure 5. IMU-based SSIs. (A) Working principle of IMU-based SSIs. The left and right parts of (A): Adapted from Ref. [114], under CC BY 4.0 license; (B) IMU-based SSIs with calibration of head movements. Adapted from Ref. [114], under CC BY 4.0 license; (C) SSIs based on the soft magnetic skin and magnetometers. Adapted with permission from Ref. [115] Copyright 2023, RSC. IMU: Inertial measurement unit; SSIs: silent speech interfaces; DF: differential feature.

Extensive research has been conducted on IMU-based SSIs^{[3,46,47,117]}. Researchers have developed SSI using two IMUs attached below the jaw and on the neck to detect subtle skin deformations during speech^[47]. Such a system applies deep learning to interpret silent commands from motion signals. When the users silently read on-screen text, the system roughly pairs sensor data with phoneme sequences generated from the text using the timestamp. Connectionist temporal classification (CTC)^[118] is then used to establish the alignment between the motion data and the target phonemes, enabling silent speech-to-text recognition^[47]. Figure 5B shows a similar IMU-based interface, but incorporates an additional sensor mounted on the head^[114]. This extra sensor enables real-time calibration that eliminates motion artifacts caused by head and body movements, ensuring that only lip and chin movements are captured. A data fusion algorithm combines signals from the three IMUs to reconstruct high-fidelity lip motion, and a speech movement reconstruction strategy is used to generate fluent sentence data from a small set of word-level samples [Figure 5B, Data Fusion]. The calibration process for removing motion artifacts is crucial, particularly for mobile or daily applications, as it significantly improves signal quality, long-term stability, and the resulting speech recognition accuracy.

Apart from accelerometer and gyroscope-based SSIs, Figure 5C highlights an interface using a single piece of soft magnetic skin and two magnetometers (one working and one reference) to monitor deformations from the TMJ^[115]. The soft magnetic skin, which is a stretchable composite of magnetic particles and a silicone matrix, was attached to the facial skin behind the ramus. As the user silently articulates, skin and jaw movements in the TMJ region alter the magnetic field from the soft magnetic skin, which can be detected by a nearby working magnetometer placed behind the ear. To reduce noise from non-speech body motions or geomagnetic fields, a reference magnetometer is placed on the opposite side of the head [Figure 5C, Sensor Positions]. To optimize sensor placement, digital image correlation (DIC) is adopted to map skin strain and displacement during silent speech. The ramus area near the ear is identified as the optimal sensing location, considering both signal strength and wearability [Figure 5C, DIC Measurement]. Such a system is very similar to the PMA system (discussed in Section 3.2), with the primary difference being the placement locations of the magnetic components.

IMU-based SSIs are well-suited for daily use due to their compact design, low power consumption, and built-in wireless data transfer capabilities. However, these systems often suffer from poor skin conformability because rigid circuit boards do not adapt well to curved and moving surfaces. Development of flexible and stretchable IMUs is essential for enhancing comfort and ensuring more precise, stable tracking of natural facial and jaw movements. In addition, motion artifacts and ambient interference remain significant challenges in IMU systems. Importantly, the motion artifacts in IMU systems are fundamentally different from those in biopotential signals, such as EEG and EMG. Motion artifacts in EEG and EMG mainly arise from disturbances at the electrode-skin interface. In contrast, motion artifacts in IMU-based systems are typically induced by body movements. Additional units acting as calibration references are necessary to eliminate the influence of body movements or geomagnetic fields.

Strain sensor-based SSIs

Strain sensor-based SSIs measure changes in articulator movements [Figure 1A] by detecting facial skin deformations associated with speech articulation using strain sensors. Figure 6A depicts the working principle of SSIs based on strain sensors. To accurately capture facial movements, the strain sensors must be highly stretchable and conformal to the skin^[119]. This conformability allows the sensors to deform alongside the skin surface during articulation, ensuring good mechanical contact without restricting natural facial movements. This approach differs from IMU-based methods in two key aspects: (1) IMUs acquire motion signals at discrete points or acquire averaged signals across a broader area, whereas strain sensors provide continuous multi-location deformation measurements; (2) Commercial IMU devices are typically rigid and cannot achieve conformal skin contact. Fully conformal IMU systems could potentially offer superior performance, but they have not yet been developed for SSI applications.

Figure 6. Facial skin-mounted strain sensors for SSIs. (A) Working principle of strain sensor-based SSIs. The left part of (A): Adapted from Ref. [119], under CC BY 4.0 license. Signals in the right part of (A): Adapted from Ref. [120], under CC BY 4.0 license; (B) SSIs enabled by resistive strain sensors for monitoring facial strain and DIC techniques for selecting sensing locations based on strain and displacement. Adapted with permission from Ref. [121], Copyright 2022, ACS; (C) SSIs enabled by biaxial resistive strain sensors for monitoring facial strain. Adapted from Ref. [120], under CC BY 4.0 license; (D) SSIs enabled by triboelectric sensors for monitoring facial movements. Adapted from Ref. [11], under CC BY 4.0 license. SSIs: Silent speech interfaces; DIC: digital image correlation.

Strain sensors can operate based on various principles, such as resistive, piezoelectric, and triboelectric effects^{[11,120,122-127]}. For instance, resistive strain sensors generate highly sensitive signals (i.e., resistance changes) in response to tiny skin movements around the lips, which can be acquired using simple data acquisition methods^[120]. These signals typically occur within a low-frequency range below 20 Hz, effectively capturing the subtle mechanical motions of speech articulation.

Most strain sensors developed for SSIs utilize resistive sensing mechanisms. A representative single-axis resistive strain sensor based on AgNWs is shown in Figure 6B^[121]. This sensor incorporates a strain-modulating structure to increase the sensitivity [Figure 6B, Structure]. The strain-modulating structure features periodic Ag-Ni particle regions in a PDMS matrix that locally increase stiffness, which leads to concentrated strain in adjacent softer PDMS regions and thus enhanced sensitivity. This study adopts DIC analysis combined with ML algorithms to identify optimal sensor placement locations [Figure 6B, DIC Measurement]. The ML model ensures that sensors are positioned not only in regions of maximum strain but also at locations rich in discriminative information for distinguishing speech-related facial movements^[121].

Beyond single-axial strain sensors, biaxial resistive sensors have been developed to capture facial skin deformations in two orthogonal directions [Figure 6C]^[120]. In this approach, four sensors were placed around the mouth to monitor dynamic skin deformations during silent speech [Figure 6C, Sensor Position]. Each sensor comprises two orthogonally aligned silicon nanomembrane (SiNM) strain gauges [Figure 6C, Structure]. Due to directional sensitivity and mechanical decoupling achieved through the fractal serpentine design, strain along one axis induces minimal response in the orthogonal gauge, effectively minimizing crosstalk [Figure 6C, Biaxial Sensing]. The fractal serpentine design also enhances the sensor stretchability by allowing the intrinsically stiff silicon material to accommodate large deformations through its geometrically engineered layout. With a total thickness under 8 μm, the self-standing sensor conforms seamlessly to facial skin during speech, enabling accurate and repeatable signal acquisition [Figure 6C, Conform to Skin]. By minimizing bending stiffness, this reduced thickness allows the intrinsically stiff and brittle single-crystalline silicon to accommodate large deformations without fracturing.

Besides the resistive sensor, triboelectric sensors represent another approach for SSIs [Figure 6D]^[11]. Triboelectric sensors were positioned at similar facial locations as resistive strain sensors [Figure 6D, Sensor Position]. Each triboelectric sensor consists of flexible polyvinyl chloride (PVC) and nylon layers separated by a sponge spacer and encapsulated in a polyimide film [Figure 6D, Structure]. During speech articulation, skin deformation causes mechanical contact and separation between the layers, which generates triboelectric charges and leads to measurable electrical signals [Figure 6D, Charge Transfer]. While these electrical signals do not directly represent strain values, they correlate strongly with skin strain caused by lip motions^[11]. Piezoelectric strain sensors, such as the conformable AlN-based conformable facial code extrapolation sensor (cFaCES)^[122], have shown strong potential for decoding facial deformations by converting localized skin strains into electrical signals. Although these sensors offer high sensitivity and rapid response to dynamic strain, they have not yet been implemented in SSIs, despite their proven ability to capture facial motions.

Strain sensor-based SSIs offer excellent skin conformability and wearing comfort, through the use of soft, stretchable materials and deformable structural designs (e.g., fractal patterns). However, one of the major challenges lies in system integration. Current designs often lack compact, high-density, multi-channel arrays capable of supporting accurate spatial mapping of articulatory movements. Additionally, repeated attachment and removal cycles can lead to mechanical degradation or signal drift, compromising long-term reliability and usability. The development of fully integrated systems incorporating wireless data transmission, robust interconnections, and enhanced reusability remains a critical research focus for the practical deployment of strain sensor-based SSI technology.

Throat-mounted mechanical sensor-based SSIs

Throat-mounted mechanical sensor-based SSIs capture mechanical deformations associated with articulator movements and articulation effects [Figure 1A]. Figure 7A summarizes the sensing of throat movements. Unlike facial strain sensors that primarily detect low-frequency skin deformations on the cheeks and lips, throat-mounted sensors capture a broader spectrum of physiological signals with distinct frequency characteristics^[1]. These sensors, typically implemented as ultrasoft patches positioned over the larynx, detect both slow muscle motions (mechanical deformation rather than biopotentials) and high-frequency residual vibrations with frequencies extending up to 2 kHz. The captured signals exhibit distinct frequency bands corresponding to different physiological processes: Muscle motions around the throat are typically confined to relatively low frequencies, while esophageal voice signals extend to mid-frequency ranges, and remaining laryngeal or acoustic energy can appear at higher frequencies^[1]. Importantly, throat movement signals persist during silent speech, although the high-frequency content may be significantly attenuated or absent. Because the sensor is positioned directly at the source of these signals, it captures richer mechanical and acoustic information and remains less susceptible to interference from head movements or background noise.

Figure 7. Throat-mounted mechanical sensors for SSIs. (A) Signals at different frequencies from the throat. The right part of (A): Reproduced with permission from Ref. [1] Copyright 2023, Springer Nature; (B) SSIs enabled by soft magnetic composites for monitoring throat movements. Adapted from Ref. [54], under CC BY 4.0 license; (C) SSIs enabled by ion gel-based sensors for monitoring pressure at the skin-sensor interface on the throat. Adapted from Ref. [128], under CC BY 4.0 license. SSIs: Silent speech interfaces; PDMS: polydimethylsiloxane; MI: magnetic induction; MC: magnetomechanical coupling.

The development of throat monitoring for SSIs is concentrated around strain, pressure, and vibrational sensing. As an example of the strain-based devices, Figure 7B presents a throat monitoring system based on magnetoelasticity and electromagnetic induction for detecting laryngeal muscle movements^[54]. It should be noted that the motion sensor shown in Figure 7B utilizes a magnetic sensing mechanism. The articulatory movements alter the geometry of the soft magnetic film and therefore change the magnetic flux density. The key sensing layer is the magneto-mechanical coupling (MC) layer, composed of a PDMS matrix embedded with magnetic microparticles [Figure 7B, Structure]. During silent speech articulation, laryngeal muscles expand and contract, causing the MC layer to undergo corresponding deformations [Figure 7B, MC Layer Deformation]. These deformations alter the magnetic flux within the surrounding serpentine copper coils, consequently modifying the induced current. The kirigami structure amplifies both horizontal and vertical strain responses, ensuring that subtle throat muscle motions can be effectively converted into electrical signals.

Pressure sensors represent another approach for detecting throat movements. For instance, Figure 7C shows a throat-mounted device using a pressure sensor based on a zwitterionic hydrogel^[128]. This structure features seamless integration of a hydrogel sensing element, a polymer support frame, and a wireless data acquisition circuit in a compact, skin-conformal design [Figure 7C, Structure]. The device monitors the pressure between the throat skin and the sensor. The sensing mechanism relies on force-induced ion generation: When throat pressure is applied, the distance between zwitterionic groups decreases, triggering water dissociation and generating mobile OH⁻ ions. The ion generation produces measurable ionic currents, leading to resistive changes under pressure [Figure 7C, Morse Code Signal]. The ion-based conduction enables high sensitivity even under very low pressures. Rather than attempting full word recognition, this system demonstrates its capability in Morse code recognition. The user controls the contraction of their muscles around the throat, and the system converts the silent cues into short and long signals that represent dots and dashes. While this method enables silent communication without audible speech or visible movement, reliance on Morse code limits its efficiency for rapid and natural communication.

As for vibrational sensing, an artificial throat is a device that detects muscle movements or residual vibrations in the throat and converts them into sound or speech^[53,129]. This technology has attracted significant research interest due to its potential to assist individuals who have lost their voice box in regaining the ability to communicate^[1,130]. The artificial throat integrates both sensing and sound-emitting functions within a single patch attached to the laryngeal skin^[1]. Its structure includes a perception layer that is made using laser-induced graphene on a flexible polyimide substrate, whose electrical resistance changes under subtle strain. The perception layer detects low-frequency throat muscle motions and mid-to-high-frequency vocal vibrations. A separate vocalization layer converts the collected electrical signals into audible sound. This system collects motion signals, converts them into spectrograms, and applies deep learning to recognize speech content. The recognized text is then synthesized into audio output by the artificial throat, thereby restoring speech capability to users without vocal cords. Newer acoustic sensors utilize piezoelectric micromachined transducers to achieve a broader bandwidth and higher sensitivity^[131]. This allows for accurate speech recognition even in noisy environments or while the user is wearing a mask.

Throat-mounted SSIs offer a simple wearable solution for capturing laryngeal movements. Their compact design enhances user comfort and facilitates continuous monitoring, making them suitable for daily use. However, throat-mounted systems are often limited to single channels and may not provide sufficient information to support full natural language recognition^[128], especially when signals corresponding to laryngeal movements are weak.

Comparisons of epidermal systems for SSIs

Representative epidermal systems for SSIs are summarized in Table 1, as detailed in Sections 3.1.1 to 3.1.5. The research landscape encompasses two primary development trajectories: (1) Hardware-focused studies that prioritize signal quality enhancement and improved wearability (reflected in advances in stretchability and unobtrusiveness); and (2) Algorithm-centric approaches that emphasize signal processing innovations, including phoneme-level recognition, speech synthesis, and large-scale dataset utilization. These systems fall into four main hardware modalities: EMG, EEG, IMU, and deformation sensors (including facial or throat-mounted variants). A fundamental distinction exists between signal types: EMG and EEG capture biopotential signals, while IMU and deformation sensors (e.g., strain and pressure sensors) detect mechanical motions. This fundamental signal difference underlies the core trade-off: Biopotential signals are closer to the neuromuscular source, providing direct access to neural activation patterns, whereas mechanical signals reflect articulator movements in a more straightforward manner.

In terms of hardware, EMG and strain-sensor systems have demonstrated significant progress in wearing comfort, unobtrusiveness, and social acceptance. These improvements have been facilitated by the adoption of flexible, stretchable, and often transparent materials that conform to the skin, thereby enhancing physical comfort and social acceptance. In contrast, EEG systems are bulky and require a large number of channels. Existing systems often rely on rigid and opaque components. Due to these limitations, EEG and IMU systems are considered less comfortable and socially acceptable for extended daily deployment.

Skin-mounted sensors face significant interference from motion artifacts during daily activities. These artifacts primarily arise from interface impedance instability, overlapping biopotentials, and mechanical disturbances. To ensure signal integrity, a multi-layered approach is required^[138]. (1) At the hardware level, achieving a stable bioelectronic-tissue interface is critical. Using low-modulus materials ensures mechanical matching with the skin, preventing detachment and subsequent impedance fluctuations. Other approaches to enhance conformal contact include reducing device thickness or using deformation-tolerant structures such as serpentines or kirigami patterns. Additionally, covalent chemical bonding or bio-adhesives provide robust integration in dynamic environments^[139-141]; (2) At the sensor level, specialized sensor designs, such as material-enabled dampers or strain isolators, can absorb or block mechanical vibrations before they reach the sensing area^[142]; (3) At the circuit level, signal compensation channels can be deployed to capture and subtract common-mode noise^[143,144]; (4) At the software level, algorithmic intervention remains a pivotal post-processing strategy. Adaptive filtering and transforms (e.g., wavelet or fast Fourier) selectively remove unwanted frequency components^[145,146]. For more complex, non-linear artifact patterns, ML and deep learning models can be trained to extract target physiological features from noisy datasets^[146,147].

Powering flexible SSIs remains a primary challenge for continuous, real-world deployment. Conventional bulky and rigid batteries severely compromise the skin-conformability of bio-integrated systems. Future interfaces may integrate flexible power sources or advanced energy harvesting modules, such as biomechanical or thermal harvesters, to enable self-sustained, battery-free systems^[148]. Beyond hardware improvements, deploying AI-assisted power management strategies is equally critical. By intelligently and dynamically balancing sensor sampling rates with model computation loads, these systems can possibly reduce overall energy consumption without sacrificing decoding accuracy^[149].

Regarding software development and SSR performance, recent studies have demonstrated that EMG- and IMU-based SSIs can achieve high recognition accuracy at both the word and phoneme levels^[46,132]. In contrast, EEG and deformation-sensor-based systems generally achieve lower accuracy^[121,133]. Notably, the recognition accuracy for EEG systems sometimes falls below 50% for certain recognition tasks, as shown in Table 1. Although direct performance comparisons remain challenging due to variations in datasets and evaluation metrics, collective findings across multiple modalities reveal the relative strengths and limitations inherent to each method. A detailed discussion of ML-assisted signal processing of silent speech cues is provided in Section 4.

EPG-based SSIs

EPG-based SSIs capture cues from articulator movements (i.e., tongue movements) in the speech generation process [Figure 1A]. As shown in Figure 8A, an EPG system employs a custom-made palate plate placed in the user’s mouth, sitting against the hard palate^[152]. This plate contains an array of small electrodes connected to an external control unit^[153]. A low-level alternating current is applied between each electrode and a reference electrode placed on the skin, such as on the cheek or wrist^[154]. When the tongue makes contact with an electrode, the impedance changes, allowing the system to detect the contact through changes in the electrical signal. If there is no contact, the signal remains unchanged. Each electrode records a binary signal that reflects the presence or absence of tongue contact at each moment^[155]. It is worth noting that EPG does not measure biopotential signals as EMG does. Instead, it senses direct physical contact rather than measuring voltage changes stemming from ion flows in the body. By sequencing the on/off patterns from all electrodes over time, EPG provides a real-time contact map. These spatial-temporal patterns from the map reflect the tongue’s position relative to the hard palate during silent speech, making EPG useful for SSR.

Figure 8. EPG or EOS-based SSIs. (A) Working principle of EPG-based SSIs. The left part of (A): Adapted from Ref. [150], under CC BY 4.0 license; (B) EPG systems that can measure the contact between the tongue and the sensor. Adapted with permission from Ref. [52], Copyright 2022, ACM; (C) EOS systems for measuring contact and distance between the tongue and the sensor. Adapted from Ref. [151], under CC BY 4.0 license. EPG: Electropalatography; EOS: electro-optical stomatography; SSIs: silent speech interfaces.

One example of an EPG-based SSI is illustrated in Figure 8B. The user wore a custom-made dental retainer (palate plate) embedded with 124 electrodes that detect tongue-palate contact at a sampling rate of 100 Hz [Figure 8B, EPG User]^[150]. The palate plate must be custom-fit through a dental impression process^[156] to ensure accurate electrode placement. Signals are transmitted through a wired cable to an external processing device, which currently limits both social acceptability and portability [Figure 8B, Custom Palate Plate]. The system can visualize contact patterns, known as palatograms^[157], to capture tongue positions during silent speech. An example of palatograms for the silent letter “s” is provided in Figure 8B, EPG Signals. The palatograms show clear electrode activation when the tongue makes contact with the palate plate. In contrast, vowels such as “i” often involve little or no contact with the palate [Figure 8B, EPG Signals], resulting in sparse or absent signals.

EPG-based SSIs offer detailed signals about tongue-palate contact during speech, providing high spatial and temporal resolution beneficial for speech therapy and research. However, their practical application faces several challenges. The requirement for a custom-fit artificial palate with embedded electrodes can cause discomfort, affecting wearability and user compliance. This setup is also sensitive to factors such as saliva-induced electrode degradation and saliva bridging between adjacent electrodes^[155], which can compromise signal quality and robustness over time. Moreover, EPG’s reliance on physical contact limits its effectiveness in capturing certain phonemes, especially those involving minimal tongue-palate interaction^[151]. These limitations affect EPG’s standalone efficiency as an SSI, indicating a need for integration with complementary sensing methods to enhance performance and user experience.

EOS-based SSIs

Intraoral systems for SSIs are designed to monitor articulatory movements, capturing movement patterns of the tongue relative to the hard palate. While traditional contact sensors (EPG) are excellent for characterizing direct tongue-to-palate contact when pronouncing consonants (e.g., /t/, /d/, and /n/), they fail to capture the tongue kinematics during the production of vowels (e.g. /o/, /e/), where the tongue lowers to create an open space, or during certain fricatives (e.g., /s/, /sh/) that require near-palate constrictions with zero applied pressure. Proximity sensing is therefore essential to bridge this gap, as it maps the continuous trajectory of the tongue even during these non-contact gestures. For example, systems like EOS extend traditional EPG by integrating optical modules acting as proximity sensors to capture both contact and non-contact information. Furthermore, given the highly limited space inside the oral cavity for mounting sensors, developing compact, multi-modal sensor units capable of both proximity and pressure sensing in a planar layout is highly meaningful for the practical advancement of intraoral SSIs.

The EOS system combines EPG and optical distance sensing to overcome EPG’s limitations [Figure 8C, Assembled Sensor Unit]^[151]. The contact sensing component of the EOS system includes 32 gold-plated pads arranged to detect the tongue-palate contact during silent speech. The body clock contact establishes electrical contact with the user’s body to serve as a reference node [Figure 8C, Circuit]. When the tongue touches any sensor pads, a voltage drop is detected, indicating contact between the tongue and the plate. This body block contact ensures a consistent voltage baseline across the array and enables accurate, stable contact sensing.

In addition to the EPG electrodes, the EOS system includes several pairs of light emitters and detectors embedded in the custom-fit palate plate [Figure 8C, Circuit]^[151]. The tongue acts as the reflective surface. Each infrared emitter sends light toward the tongue, and the nearby detector senses the amount of light reflected [Figure 8C, Optical Distance Sensing Principle]. When the tongue is close to the palate, more light is reflected, resulting in a higher detector voltage. As the tongue moves further away, less light is reflected, and the voltage decreases. After calibration, the system converts these voltage changes into real-time measurements of the distance between the tongue and each optical distance sensor.

By combining binary contact data with distance measurements, EOS-based SSIs can track the tongue’s position relative to the hard palate even without direct contact, addressing a key limitation of traditional EPG. However, like EPG, EOS systems suffer from saliva-related issues that can interfere with sensor function. Additionally, similar to EPG systems, the need for a wired connection to an external control unit reduces their wearability and limits social acceptance due to the visible and cumbersome cabling.

PMA-based SSIs

Like other intraoral systems, PMA-based SSIs track tongue positions during the speech generation process [Figure 1A]. Figure 9A illustrates the working principle of PMA-based SSIs. Similar to the IMU-based method discussed in Section 3.1.3, PMA employs magnetometers, but differs in the magnet placement. In PMA, some magnets are attached to the tongue, making it an intraoral system^[158]. In contrast, in the IMU-based SSIs, magnets are placed on the facial skin, making it an epidermal system^[115]. In PMA, external magnetometers, typically integrated into a headset, detect changes in the magnetic field caused by the movement of the magnets and, consequently, the tongue^[158,161]. However, PMA cannot identify the movement path of each magnet because every magnetometer senses the combined magnetic field. The final signal is the sum of all magnetic flux densities from all magnets^[162]. Nevertheless, different silent speech movements produce distinct magnetic patterns, which can be recognized using appropriate models. The output of the system is a time-series of tri-axis magnetic flux density values. These signals can be used directly for SSR or transformed into approximate movement paths^[158].

Figure 9. PMA or EMA-based SSIs. (A) Working principle of PMA-based SSIs. The left part of (A): Adapted from Ref. [4], under CC BY 4.0 license; (B) PMA system for SSIs. The PMA headset and one magnet in the left part of (B): Adapted from Ref. [158], under CC BY 4.0 license. The portable device and multiple magnets in the right part of (B): Adapted from Ref. [4], under CC BY 4.0 license; (C) Working principle of EMA-based SSIs. The left part of (C): Adapted from Ref. [159], under CC BY 4.0 license. Signals in the right part of (C): Adapted from Ref. [160], under CC BY 4.0 license; (D) EMA system for SSIs. The wave system in the left part of (D): Adapted from Ref. [158], under CC BY 4.0 license. The rest of (D): Adapted from Ref. [159], under CC BY 4.0 license. PMA: Permanent magnetic articulography; EMA: electromagnetic articulography; SSIs: silent speech interfaces.

Figure 9B presents one PMA system that integrates a camera for lip motion capture, a microphone for audio recording, and an array of 24 three-axis magnetometers for tracking tongue movements^[158]. In this system, only one small permanent magnet (3 mm in diameter) was attached approximately 1 cm from the tongue tip [Figure 9B, One Magnet]. During silent speech, the magnetometers positioned around the cheeks record changes in the magnetic field caused by tongue motions. These obtained magnetic signals are projected into a 2D lateral path, providing trajectories that correspond to specific silent speech phrases. Despite high accuracy and rich information from multiple magnetometers, the headset^[158] is not portable and is fixed in place during experiments to minimize head motion and ensure accurate tracking. The system captures multimodal data, including magnetic signals, lip images, and audio, simultaneously. However, this study did not perform full data fusion. Instead, only the magnetic signals and audio signals were used for tongue-movement-to-speech synthesis. Lip image data from the camera were collected but not used in this analysis.

Besides the system introduced above, a portable PMA system has also been developed using a lightweight headset [Figure 9B, Portable Device]^[4,161,162]. The headset integrates three working magnetometers and one reference magnetometer. A total of six small permanent magnets were attached using tissue adhesive: four on the lips, one on the tongue tip, and one on the tongue blade [Figure 9B, Multiple Magnets]. The working magnetometers track changes in the magnetic field caused by articulator motions, while the reference sensor isolates the geomagnetic field and ambient interference. This setup reduces environmental noise, allowing reliable signal extraction. In addition, this design offers greater portability and reduces hardware complexity. The trade-off exists between sensor density and portability. While the first system is non-portable due to its large array of magnetometers and the need for head stabilization, it provides higher spatial resolution and can convert raw magnetic signals into tongue trajectories. In contrast, the portable device sacrifices spatial fidelity and uses raw magnetic signals directly without spatial reconstruction for SSR.

Overall, PMA offers a wireless solution for SSIs by tracking articulator movements through magnetic field variations. This wireless design enhances wearability, eliminating the need for intrusive wires, which is typically necessary for EPG, EOS, and EMA. However, PMA faces several challenges in terms of robustness and precision. The system relies heavily on magnetic field measurements but lacks explicit positional tracking of specific points. This can create difficulties in capturing specific regions’ positions, potentially affecting speech recognition accuracy. Additionally, PMA systems are susceptible to interference from geomagnetic fields and user head movements, which necessitate additional calibration processes.

EMA-based SSIs

EMA-based SSIs are another approach to monitor tongue position during the speech generation process [Figure 1A and Figure 9C]. EMA requires an external device composed of several transmitter coils that are fixed around the head and driven by distinct alternating currents. These coils create overlapping magnetic fields with known strength and phase at each moment^[163]. Small receiver coils are attached to the tongue, lips, jaws, or palate and are connected to the control unit by cables. As each receiver coil moves with the articulator, its position changes within the external magnetic field, causing a change in the voltage induced in the receiver coil due to Faraday’s law^[164]. This induced voltage varies with the receiver coil’s distance and orientation relative to the transmitter coils. By sampling the voltage signal from each coil, the system processes these signals and calculates their three-dimensional positions and thus the tongue movement.

In a representative work, the EMA system utilized the commercially available NDI wave system [Figure 9D, Wave System] to capture tongue movements using the external electromagnetic fields^[163]. Different studies may attach the wired sensor coils to the tongue at different positions [Figure 9D, EMA Sensors], but a reference sensor is usually placed on the forehead to account for head movements and enable motion correction [Figure 9D, Calibration Sensor]^[159,165]. The sensor coils are compact, typically measuring just a few millimeters in size [Figure 9D, Coil Structure]^[166]. They are lightweight and designed to minimize interference with natural speech production. Since such an EMA system is a commercial device, it can directly output the trajectories of coils using available integrated algorithms for processing raw data.

Compared to PMA systems, EMA offers higher spatial accuracy and more stable performance, making it well-suited for precise articulatory tracking^[167,168]. This is because EMA tracks each sensor coil individually, allowing it to reconstruct the full movement path of every attachment point^[167,168], which is not possible with PMA. However, EMA is less portable due to its reliance on wired sensors and external magnetic field generators^[163], which can reduce user comfort and limit movement. EMA is also sensitive to nearby metal objects. Moreover, EMA outputs raw induced voltage signals that can be converted into 2D or 3D motion paths, providing physically interpretable trajectories^[158]. whereas portable PMA systems often use raw magnetic signals directly in ML models without spatial reconstruction.

Comparisons of intraoral systems for SSIs

In summary, the intraoral systems, as detailed in Sections 3.2.1-3.2.4 and summarized in Table 2, can be divided into hard palate-mounted systems (EPG and EOS) and tongue-mounted systems (PMA and EMA). From a hardware development perspective, intraoral systems often face challenges in wearability and user comfort. EPG, EOS, and EMA designs require wired connections to external acquisition systems (as indicated by WDI = Yes in Table 2), making them unsuitable for daily use. Conversely, PMA shows advantages in device compactness, but the tongue-mounted permanent magnet (as indicated by TMI = Yes in Table 2) can reduce user comfort as it has not been optimized for the oral environment. Efforts have been made to improve the compactness of the device and wireless transmission in intraoral systems, such as the tactile oral pad^[174], but this oral pad has not yet been applied to SSIs. From a software perspective, intraoral systems have demonstrated promising recognition performance. For instance, both EOS and EPG have achieved great word recognition rates^[52,170]. PMA and EMA systems have been applied to both word-level and phoneme-level tasks, often using deep learning models such as RNNs to decode rich spatial-temporal tongue motion data. Signal-to-audio synthesis tasks using intraoral data have also been explored, although many results are not yet benchmarked with standardized accuracy metrics. A detailed discussion of algorithms can be found in Section 4.

ECoG-based SSIs

ECoG systems have been applied to assist a participant whose speech ability was affected by a severe stroke^[37]. In severe cases, patients suffer extensive damage to the left hemisphere, including brainstem atrophy resulting from the stroke. Brainstem atrophy can impair speech abilities because it is responsible for critical neural pathways that coordinate the muscles involved in speech production^[182]. Damage to the brainstem area disrupts these pathways, leading to speech disorders such as dysarthria^[183]. Despite the injury and disrupted neural pathways, the motor cortex remained viable, allowing placement of a high-density ECoG array to capture neural signals. The implemented array consists of 253 electrodes positioned over regions of the sensorimotor cortex^[37]. The sensorimotor cortex is a relatively large area that includes the primary motor cortex and primary somatosensory cortex, covering speech production areas associated with the movement of the face, tongue, jaw, and larynx^[184]. This placement allows researchers to capture cortical signals corresponding to articulatory intentions, even though the participant could not physically produce speech.

The sample neural signals were recorded from these electrodes when the participant attempted different articulator movements beyond pure imagined speech^[37]. These signals reflect the electrical activity of the brain during attempted speech and exhibit distinct temporal patterns for different activities. The participant used a visual cue system (monitor) to prompt attempted speech, and the resulting ECoG signals were decoded in real time using an RNN. The design and placement of the ECoG array, combined with advanced decoding algorithms, achieve a median WER of 25% on a vocabulary set of 9,655 sentences^[37]. Recent developments in specialized, high-resolution ECoG electrode arrays have significantly improved speech decoding^[178]. The flexible polymer ECoG electrode array in Figure 10B is equipped with either 128 or 256 micro-electrodes. These tiny sensors are compared against much larger, traditional macro-ECoG setups [Figure 10B, array comparison]. This massive upgrade highlights a jump to 34-57 electrodes per square centimetre [Figure 10B, electrode density]. Such precise placement maps exactly how these high-resolution arrays are implanted directly over the speech motor cortex [Figure 10B, electrode position]. Recent work also extends this technology to continuously streaming neuroprostheses that synthesize speech in 80-ms increments^[185]. These models can also personalize the audio output to match a participant’s pre-injury voice. Beyond stroke patients, researchers have successfully applied ECoG-based speech synthesis to individuals with ALS^[186]. These systems decode neural activity into intelligible keywords while allowing the user to speak at their own pace. In comparison to the MEA method, ECoG adopts a flat grid placed over a large area of the sensorimotor cortex, enabling broad coverage of speech-related regions. ECoG is less invasive than MEA but lacks single-neuron resolution and cannot access deeper brain structures.

MEA-based SSIs

Similar to ECoG, MEA has been used to develop a high-performance SSI for an ALS patient^[38]. The participant is severely dysarthric and cannot produce clear speech, though the partial facial movements are retained. Four MEAs, each with 64 channels, were implanted in the cortex [Figure 10C, Embedded Devices, Sensing Positions], targeting two key regions: Two arrays were positioned in the area of the ventral premotor cortex (area 6v)^[187], and the other two arrays were placed in the Broca’s area (area 44)^[188] to capture signals associated with facial movements and language planning. The study found that area 6v contains a rich representation of speech articulators. Each array measures 3.2 mm × 3.2 mm and was manufactured by Blackrock Microsystems, featuring 1.5 mm-long silicon microelectrodes coated with iridium oxide. These penetrating electrodes enable single-neuron-level recordings of spiking activities. The implanted MEAs are connected to external recording hardware through a percutaneous pedestal with a wire bundle containing 256 leads [Figure 10C, Component]. Although 256 electrodes were used, neural decoding is primarily driven by the 128 electrodes in area 6v due to their stronger tuning to speech articulators. Data from each electrode is processed in real time using digital filters and algorithms, extracting features including threshold crossings and spike power. These features are fed into an RNN model trained to decode phonemes in real time for reconstructing intended sentences. The MEA system achieves a WER of 9.1% for a 50-word vocabulary and 23.8% for a 125,000-word vocabulary^[38]. Compared to ECoG, the MEA system offers much higher spatial resolution and the ability to detect single-neuron activities^[189], critical for fine articulatory decoding. Recent studies have expanded the capabilities of MEA-based speech systems by exploring new cortical regions and decoding techniques. MEA devices implanted in the supramarginal gyrus have successfully decoded internal or imagined speech at the single-neuron level^[190]. Researchers have also achieved instantaneous voice synthesis by recording neural activity from 256 microelectrodes in the ventral precentral gyrus of a participant with ALS. This closed-loop neuroprosthesis translates cortical signals into audio in real time, allowing the user to consciously modulate their intonation^[191]. Furthermore, this high-density setup calibrates rapidly, enabling self-paced conversations at approximately 32 words per minute^[192]. However, MEA implantation is more invasive, covers a limited cortical surface area, and may experience signal instability over time.

SEEG-based SSIs

SEEG, as a minimally invasive surgical procedure, is primarily used to identify deep areas in the brain where difficult-to-treat epileptic seizures originate^[193]. Researchers have applied SEEG to map speech areas in the brain^[194]. During the surgery, surgeons drill small holes in the patient’s skull and use MRI images to guide thin leads to reach the selected cortical and subcortical (white matter) areas. Each SEEG lead comprises 10-18 platinum contacts, with 0.8 mm in diameter, 2 mm contact length, and 1.5 mm inter-contact spacing. Patients performed simple language tasks like naming pictures or saying action words, while SEEG recorded their brain signals. SEEG signals are recorded in bipolar configuration at 1,200 Hz sampling frequency. These signal changes can reveal active speech regions. Notably, this study involved speech region mapping but has not yet applied SEEG to speech reconstruction. Regarding similarities and differences of SEEG to ECoG/MEA, SEEG records LFPs^[195], similar to ECoG. Unlike SEEG and ECoG, MEA can also capture single-neuron spikes^[189]. However, SEEG’s depth electrodes can monitor brain activities throughout deeper regions, not just on the cortical or subcortical surfaces. SEEG relies on stereotactic implantation through small burr holes, avoiding the need for a large craniotomy. From a surgical access perspective, this skull-level exposure is less invasive than subdural grid placement^[196,197]. However, the procedure still requires advancing depth electrodes directly into the gray and white matter, carrying an inherent risk of local neural tissue damage^[198]. Therefore, labeling SEEG as “less invasive” strictly refers to the cranial opening. This physical disruption of internal tissue should be accounted for when evaluating the total system safety. Consequently, SEEG offers a favorable balance between invasiveness, coverage, and functional mapping capabilities, though its potential for speech reconstruction has not been explored as thoroughly as ECoG and MEA. To address the historical lack of SEEG data, researchers have recently introduced comprehensive datasets capturing vocalized, mimed, and imagined speech modes^[199]. This includes expanding SEEG research to tonal languages like Mandarin Chinese, where variations in pitch necessitate highly nuanced decoding algorithms. Beyond discrete word recognition, recent studies demonstrate that the continuous acoustic speech envelope can be successfully reconstructed from SEEG signals^[200].

Comparisons of surgically embedded systems for SSIs

As detailed above and in Table 3, the surgically embedded system represents the most invasive but also the most information-rich modality of SSIs. From a hardware standpoint, ECoG and MEA require craniotomy, with MEA being more invasive due to its penetration through cortical gray matter (see columns “Location” and “Surgery Invasiveness” in Table 3). SEEG offers a less invasive approach because it requires only burr holes for inserting electrode shafts. All three methods can provide hundreds of channels of signal for speech recognition (see column “Channel Amount” in Table 3). MEA can provide neural signals (spikes) with the highest resolution among the three methods, albeit at the expense of increased invasiveness. From the software perspective, ECoG and MEA have shown promising performance in decoding speech at the phoneme and word levels. ECoG has also been applied to signal-to-audio synthesis^[39]. SEEG, while less extensively studied compared to the other two methods, has shown moderate recognition performance in word-level recognition and signal-to-audio synthesis^[204,205]. A more detailed discussion of word-level, phoneme-level, and audio synthesis can be found in Section 4.

Noise reduction

Preprocessing in SSR typically begins with modality-specific noise reduction techniques, forming the first step toward reliable signal representation. The biosignals used in SSR, such as sEMG and EEG signals, are inherently susceptible to various sources of noise, including motion artifacts, sensor drift, and powerline interference. To address this, tailored denoising strategies are applied. In sEMG, bandpass filtering (e.g., 20-500 Hz^[67,68]) is commonly used to eliminate low-frequency motion artifacts and high-frequency noises, while the notch filter (50/60 Hz)^[23] is used to mitigate the impact of powerline interference. For EEG, methods like common spatial patterns (CSP)^[214] and independent component analysis (ICA)^[215] can help isolate task-relevant neural signals from background activity. Additionally, the wavelet transform^[216] is broadly employed across modalities to suppress transient disturbances while preserving essential signal characteristics. These noise reduction steps not only improve signal quality but also serve as a prerequisite for accurate recognition and robust cross-condition generalization.

Normalization and synchronization

The variability introduced by differences in subjects, sessions, and sensor setups poses a key challenge for SSR, which can be addressed through effective normalization and synchronization. Normalization ensures that signal amplitudes and distributions are consistent across conditions, enabling models to focus on task-relevant variations rather than irrelevant fluctuations. Different normalization strategies are applied depending on the structure of the data and the model architecture. Feature normalization methods, such as z-score normalization, min-max scaling, and channel-wise standardization, are commonly used in the preprocessing stage to align signal scales across users or sessions^[217]. These approaches are particularly effective for biosignals like sEMG, where different channels may exhibit distinct baseline characteristics. In deep learning-based SSR systems, normalization may also occur within the model. Batch normalization^[218], which normalizes inputs across the batch dimension during training, helps stabilize learning and accelerates convergence, especially in convolutional architectures. However, in sequential models such as RNNs or Transformers, where batch statistics vary over time, alternative strategies are often preferred. Temporal normalization, for instance, normalizes signal features along the time axis and is useful in handling signal drift and session-specific shifts^[219]. Similarly, instance normalization and layer normalization^[220,221], which operate at the sample or layer level, are beneficial in real-time SSR scenarios where batch sizes may be small or inconsistent. Synchronization complements normalization by ensuring temporal alignment across different modalities. This is especially critical in multimodal SSR systems that combine, for example, sEMG and lip imagery^[222]. Methods such as cross-correlation, energy-based alignment, and the use of external triggers help maintain semantic consistency across input streams. Together, normalization and synchronization significantly improve recognition accuracy, enable effective multimodal integration, and support the development of robust, generalizable SSR systems.

Signal segmentation

Segmentation transforms continuous biosignals into modeling-ready units, such as phonemes, words, or sentences, and is a crucial step in SSR preprocessing. By converting raw, unstructured signals into discrete chunks, segmentation enables models to better capture the temporal and linguistic structure of speech. Traditionally, fixed-size sliding windows, typically 30 to 50 milliseconds and having overlaps between windows, are used, as they roughly match the duration of phonemes^[67,223]. These overlapping windows help capture local articulatory dynamics and provide uniform inputs for downstream processing. However, static segmentation schemes struggle with natural variability in speech rate and articulation. In fast or slow speech, fixed windows may misalign with actual phoneme boundaries, leading to degraded performance. To overcome this limitation, adaptive segmentation methods have been introduced. For instance, dynamic time warping (DTW) aligns utterances by minimizing temporal distortion across articulatory patterns^[224]. In higher-level tasks, such as sentence-level SSR, hierarchical segmentation is often applied. This involves first detecting word-level pauses or silences and then segmenting those regions further into phonemic units^[12]. Such multi-scale segmentation improves linguistic alignment and model interpretability. Segmentation also impacts computational complexity. Smaller windows offer higher temporal resolution but result in longer sequences and increased memory usage, while larger or adaptive segments reduce computational load but may sacrifice detail. Therefore, effective segmentation balances linguistic precision with efficiency. Ultimately, segmentation supports both accurate modeling and practical deployment, making it a foundational component of robust SSR systems.

Handcrafted features in time and frequency domain

Handcrafted features form the foundation of early SSR systems and remain widely used due to their interpretability and simplicity. In the time domain, statistical features such as root mean square (RMS), zero crossing rate (ZCR), mean absolute value (MAV), and integrated EMG (IEMG) are commonly extracted to characterize signal energy, muscle activation, and waveform complexity^[225]. These features are particularly informative in EMG and EEG-based SSR tasks, where temporal fluctuations correspond to articulatory muscle activities or cortical responses. In the frequency domain, spectral analysis techniques help capture phoneme-level acoustic or articulatory characteristics. Power Spectral Density (PSD) and Short-Time Fourier Transform (STFT) reveal frequency-specific energy distributions, which are useful in distinguishing different speech gestures^[226,227]. Time-frequency representations, such as Continuous Wavelet Transform (CWT)^[228], further improve robustness by capturing localized spectral variations over time, especially important for nonstationary signals like ultrasound. Among the most widely used handcrafted descriptors are Mel-Frequency Cepstral Coefficients (MFCCs), originally developed for acoustic speech recognition^[229]. In SSR, MFCCs have been adapted by computing them on transformed biosignals, such as EMG spectrograms or lip imagery treated as pseudo-spectrograms. While this allows for the reuse of well-understood features, MFCCs rely on perceptual auditory scaling, which may not always align with the physical properties of biosignals like EMG or EEG, limiting their interpretability in non-acoustic modalities.

Learned features from deep models

To overcome the limitations of manually designed features, modern SSR systems increasingly rely on learned features derived from data-driven models. Deep learning, particularly using CNNs, has become the dominant framework for feature extraction in recent SSR research. CNNs are especially well-suited for 2D and 3D inputs such as spectrograms, lip images, or ultrasound frames. These networks automatically learn hierarchical representations, capturing both low-level textures and high-level articulatory patterns without manual engineering. In high-density sEMG (HD-sEMG) applications, CNNs are often used to extract spatiotemporal activation maps, which encode channel-wise information across time and frequency^[213]. These learned features have been shown to improve both phoneme and word-level classification performance. For image-based modalities such as lip or tongue motion, CNNs can be applied directly to raw image frames or processed representations such as motion fields^[230]. In 3D CNN architectures, temporal sequences of images are input as volumetric data, allowing the model to simultaneously capture spatial and temporal correlations. To further improve spatial sensitivity, attention mechanisms can be integrated into CNNs, allowing the network to focus on informative regions while ignoring background noise or irrelevant areas^[231]. Some SSR systems employ hybrid approaches, feeding handcrafted features such as MFCCs or CWT coefficients into CNNs. This allows the model to benefit from the domain knowledge embedded in handcrafted features while gaining the adaptability of learned representations^[232]. Importantly, learned features offer greater flexibility and generalization compared to handcrafted ones. They can adapt to varying signal modalities, speaker-specific traits, and environmental conditions, making them particularly valuable in large-scale or cross-subject SSR applications. Moreover, the ability to fine-tune learned representations using end-to-end training pipelines facilitates joint optimization of feature extraction and classification. In summary, both handcrafted and learned features play complementary roles in SSR, and the choice or combination of feature extraction methods directly influences system performance, adaptability, and scalability.

Data augmentation is particularly important for SSIs due to the scarcity and high acquisition cost of physiological signals. Recent work has explored augmentation strategies directly on articulatory time-series, extending concepts from SpecAugment to non-acoustic modalities^[233]. These include time masking and articulatory-dimension masking to simulate temporal dropout and sensor corruption, as well as signal-level perturbations (such as noise injection and temporal scaling) to model variability in sensor noise and speaking rate^[234]. In contrast to conventional audio augmentation, these approaches preserve underlying articulatory dynamics while introducing controlled variability, which improves model robustness in low-resource SSI settings.

Traditional sequence models

Before the adoption of deep learning techniques, SSR systems largely relied on statistical models such as HMMs and GMMs^[27,28]. These approaches followed a structured pipeline comprising sequential stages: signal segmentation, handcrafted feature extraction, and statistical classification. Among these, HMM-GMM frameworks were particularly prominent for modeling the temporal dynamics of biosignals, especially in sEMG-based SSR. An HMM represents a sequence of observed data as a series of hidden states, where each state has a probability distribution over the possible observations. Transitions between states follow a Markov process, meaning the current state depends only on the previous state. In the context of SSR, each state is typically associated with a subunit of speech, such as a phoneme or syllable, and observations correspond to extracted features from the biosignals, such as MFCCs or RMS energy. GMMs are often used to model the emission probabilities, capturing the variability of feature vectors within each state^{[4,19,26,50,235-238]}. For example, in the EMG-PIT corpus^[239], sEMG signals are segmented into fixed-size windows and transformed into MFCC or RMS features. These are then used to train an HMM where each word or phrase is represented by a sequence of states with learned transition and output probabilities. This framework allows the system to capture local temporal dependencies and perform sequence decoding via algorithms such as the Viterbi algorithm^[240]. In visual modalities such as ultrasound or lip video, traditional classifiers including Support Vector Machines (SVMs) and Random Forests (RFs) were commonly employed, often with frame-level features^[52,241,242]. While these methods are computationally efficient and relatively interpretable, they have several limitations. They require extensive feature engineering and lack the ability to model long-range dependencies. Moreover, their generalization across speakers, sessions, and noise conditions is often poor, which restricts their scalability in real-world SSR applications.

Deep learning models

The introduction of RNNs marked a major shift in SSR, moving away from rigid, handcrafted pipelines toward data-driven, end-to-end learning. RNNs are specifically designed to handle sequential data by maintaining an internal memory of past inputs, making them well-suited for modeling the temporal dynamics of biosignals such as EMG, EEG, or lip movement sequences^{[234,243-248]}. Unlike HMMs, which rely on predefined state transitions and assume conditional independence between observations, RNNs are capable of learning complex, long-range temporal dependencies directly from data without strong assumptions about sequence structure.

Among RNN variants, Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) have become the most widely used in SSR tasks due to their ability to avoid the vanishing gradient problem inherent in standard RNNs. These gated architectures maintain memory over longer time horizons and selectively update internal states based on input relevance. This enables more accurate modeling of articulatory patterns that span multiple frames, which is particularly important in biosignals that exhibit temporal variability and nonstationarity. Bidirectional RNNs (BiRNNs) further enhance the modeling capability by incorporating both past and future context into the hidden state representation^[249-251]. This dual perspective is especially valuable in SSR, where signal noise, delays, or articulation ambiguity may obscure the correct interpretation of a phoneme or word when viewed in isolation.

These architectures have been successfully deployed in SSR systems such as EarSSR^[248], which uses sEMG signals, and LipWatch^[24], which uses lip video inputs. Both systems combine RNN encoders with CTC decoders, allowing the model to learn alignments between input features and output sequences without needing frame-level labels. The CTC objective function introduces a blank symbol and multiple alignment paths, making it robust to uncertain or variable-length input-output correspondences. A common SSR architecture today is the hybrid CNN-RNN-CTC pipeline. In this design, CNNs serve as front-end feature extractors for spatial or spatiotemporal representations, such as spectrograms or lip image sequences, while RNNs model the temporal evolution of these features^[252-254]. CTC decoding then enables flexible output prediction, even in the presence of inconsistent alignment across training samples. Compared to traditional HMMs, which require explicit state definitions and emission models, this pipeline learns both representation and alignment jointly, resulting in more adaptive and scalable systems.

In addition to improved performance, deep learning models offer better generalization across subjects and recording conditions, especially when combined with regularization strategies, data augmentation, or domain adaptation. As such, RNN-based architectures represent a foundational step in the transition from handcrafted SSR pipelines to fully trainable, end-to-end learning systems.

In recent years, Transformer-based architectures have emerged as a powerful alternative to RNNs for modeling sequential data^[59]. Transformers use self-attention mechanisms to capture global dependencies in the input without relying on recurrence. This allows for parallel computation, significantly improving training efficiency and scalability for long sequences. Transformers compute attention scores between all input positions, enabling the model to directly relate distant time steps - an advantage particularly relevant for SSR, where articulatory cues or EMG patterns at one time may depend on earlier or later contexts^{[30,44,255-260]}. Recent adaptations such as Lightweight Transformers or Conformer architectures, which combine convolution and attention, have also been proposed to reduce computational complexity and latency, making real-time or on-device SSR applications more feasible^[253]. Deploying SSR on wearable interfaces imposes strict constraints on inference latency and computational load. While heavy architectures, such as standard Transformers, can achieve high accuracy, they incur memory and power costs that far exceed the capabilities of edge microcontrollers. Consequently, on-device implementation requires aggressive architectural optimization to maintain real-time responsiveness while minimizing battery drain^[261]. To achieve this, standard compression strategies such as model pruning, knowledge distillation, and low-bit weight quantization are widely explored^[262]. Additionally, replacing computationally expensive self-attention mechanisms with lightweight, convolution-dominant temporal networks has recently enabled low inference latencies on edge-AI platforms^[263]. In addition, pretrained Transformer models, when fine-tuned on SSR data, show promise for transfer learning and cross-subject generalization, particularly in low-data settings^[256]. Overall, the introduction of Transformer-based models marks a new phase in SSR development, offering flexible, scalable, and highly expressive modeling capabilities that outperform RNNs in many settings. As SSR datasets grow in size and modality diversity, Transformer architectures are expected to play an increasingly central role in future research and deployment.

End-to-end pipelines and cross-modal fusions

End-to-end SSR systems aim to directly map raw or minimally processed signals to linguistic outputs. A key enabler is CTC, which removes the need for manual alignment^[241,264]. CTC-based decoders, when paired with CNN or RNN encoders, allow flexible training from unsegmented data, making them especially suitable for SSR tasks with weak supervision. In multimodal fusion settings, systems like mSilent^[255] or EarCommand^[265] incorporate inputs from different sensor streams. Here, multi-stream CNNs or attention-based fusion layers integrate modality-specific features before sequence modeling. This enables richer representations and improves robustness. Residual CNNs, depthwise separable convolutions, and lightweight architectures further support real-time deployment on wearable devices. Recent work in SSR adopts modular architectures that separate signal encoding from language modeling. Transformer-based encoders extract structured representations from non-acoustic inputs (e.g., lip movements or EMG), while large language models (LLMs) serve as semantic decoders that resolve ambiguity and improve linguistic consistency^[266,267]. Integration of LLMs with perceptual front ends significantly reduces WER by leveraging strong language priors, either through direct conditioning (e.g., prompt-based decoding) or post-hoc correction of predicted sequences. In addition, Transformer-based cross-attention supports multimodal fusion, allowing alignment of complementary signals such as EMG and visual features within a unified representation^[268]. These approaches improve robustness, particularly in low-resource and cross-subject SSR scenarios^[268]. Recent efforts also explore adversarial training or domain adaptation (e.g., Mordo2^[254]) to enhance generalization across subjects. Inter-subject variability in physiological signals motivates the use of domain adaptation techniques. Adversarial approaches such as adaptive domain adversarial neural network (TADANN)^[269] and its variants^[42] learn domain-invariant representations through discriminator to align feature distributions across subjects. Discrepancy-based methods, particularly maximum mean discrepancy (MMD), explicitly minimize the distribution differences in feature space and are often combined with adversarial training. Such hybrid approaches reduce cross-subject performance degradation in EMG-based SSR, especially under limited-data settings^[270]. Collectively, these deep learning frameworks represent the most scalable and effective SSR modeling paradigm.

Signal-to-text vs. signal-to-audio

SSR systems typically aim for one of two types of outputs: signal-to-text or signal-to-audio. The majority of existing research focuses on signal-to-text systems, which translate biosignals into discrete linguistic tokens such as phonemes, words, or full sentences. These systems are well aligned with traditional ASR frameworks and are commonly evaluated using standard metrics such as WER^[271]. Their popularity stems from the relative ease of supervision, availability of text-labeled datasets, and the clear interpretability of symbolic outputs. In contrast, signal-to-audio systems seek to reconstruct acoustic speech waveforms from articulatory or physiological inputs. These systems typically employ regression-based architectures and are evaluated using perceptual quality metrics, such as Perceptual Evaluation of Speech Quality (PESQ) or intelligibility scores^[272]. Unlike classification-based signal-to-text systems, signal-to-audio pipelines must learn precise, fine-grained mappings from articulatory dynamics to acoustic features, making the task significantly more challenging. Temporal alignment and prosodic control further increase the modeling complexity. Due to the difficulty of acquiring high-quality parallel bio signal-speech waveform data, signal-to-audio SSR remains a relatively underexplored area. A limited number of systems (e.g., articulatory-to-waveform synthesis models^[30]) have attempted this task, but they are often constrained by data sparsity, modality noise, and high computational demands. Despite these challenges, signal-to-audio SSR holds promise for applications in speech prostheses or silent communication interfaces where natural-sounding output is desired. Continued progress in generative modeling, such as diffusion models or neural vocoders, may eventually help close the gap between articulatory signals and intelligible speech reconstruction. Recent work also considers edge-oriented deployment strategies for wearable SSI systems, including model compression and knowledge distillation to reduce computational and memory requirement. Knowledge distillation allows compact models to retain performance by transferring information from larger teacher networks, which supports real-time on-device inference^[273].