Decoding Movement Intention: Building a Real-Time Brain-Computer Interface with PyTorch and Unity

How we built a complete BCI pipeline using LSL synchronization, MNE-Python, and EEGNet to control a virtual wheelchair with brain waves.

Controlling machines with thoughts was once the exclusive domain of science fiction. Today, Brain-Computer Interfaces (BCIs) are a reality, offering pathways to restore independence to individuals with severe motor impairments.

But building a BCI is one of the most complex interdisciplinary engineering tasks in computer science. It sits at the intersection of neuroscience, signal processing, real-time communications, deep learning, and 3D simulation.

Our graduation team at Çankaya University set out to build NeuroMotion: a complete BCI system that captures microvolt-level electrical potentials from the scalp, filters out massive muscular noise, decodes motor imagery intentions using convolutional neural networks, and streams commands to a virtual wheelchair simulator in real-time.

Here is the technical blueprint of how we designed, trained, and deployed the system.

The BCI Architecture: From Scalp to Simulation

To control an object smoothly in a 3D environment, the system must operate under a strict latency budget. In virtual reality or active simulations, any delay between user intent and object actuation longer than 500 milliseconds causes sensory mismatch, resulting in severe motion sickness.

To meet this ceiling, we designed a decoupled, multi-threaded pipeline:

  ┌─────────────────────────────────────────────────────────────┐
  │                        LSL Streams                          │
  └──────────────────────────────┬──────────────────────────────┘
                                 │ EEG Data (500 Hz)
                                 ▼
                     Asynchronous Queue Buffer
                                 │
                                 ▼
                    SciPy Preprocessing Pipeline
                     (1-50 Hz Band-pass + Notch)
                                 │
                                 ▼
                      PyTorch EEGNet Classifier
                     (Left, Right, Foot, Idle)
                                 │
                                 ▼
                      UDP JSON Command Packets
                                 │
                                 ▼
                     Unity 3D Wheelchair Controller

1. Real-Time Signal Processing (MNE-Python & SciPy)

EEG electrodes capture electrical potentials in the microvolt ( $\mu V$ ) range. These signals are constantly contaminated by environmental electromagnetic noise (power line interference at 50/60 Hz) and physiological noise, such as eye blinks (EOG) or muscle contraction (EMG).

To isolate motor intent, we built an optimized, real-time preprocessing pipeline using MNE-Python and SciPy:

Lab Streaming Layer (LSL): We used the LSL protocol to capture streams from a 32-channel Mitsar-202 EEG device. LSL handles sub-millisecond synchronization across threads, aligning raw signals with event triggers.
Causal Filtering: We applied a digital causal IIR band-pass filter (1–50 Hz) and a notch filter at 50 Hz. We chose a causal filter because, unlike zero-phase filters, it does not require future data points, keeping processing latency under 5ms.
Sliding Windows: The preprocessed data was chunked into a sliding window of 250 samples at a 250 Hz sampling rate, updating every 200ms for continuous prediction.

2. Deep Learning Core (EEGNet vs. ShallowConvNet)

Once we had a clean window of spatial-temporal signals, we needed to classify the user's mental state into four classes: Left Hand, Right Hand, Foot, or Idle.

We evaluated two main deep learning architectures implemented via PyTorch:

EEGNet

EEGNet is a compact convolutional neural network designed specifically for BCIs. It utilizes depthwise and separable convolutions to extract spatial and temporal features without triggering parameter explosion:

# Simplified representative concept of an EEGNet block
class EEGNetBlock(nn.Module):
    def __init__(self, channels, samples):
        super().__init__()
        # Temporal Convolution to learn frequency filters
        self.temporal = nn.Conv2d(1, 8, (1, 64), padding='same', bias=False)
        # Depthwise Spatial Convolution to map channel relationships
        self.spatial = nn.Conv2d(8, 16, (channels, 1), groups=8, bias=False)
        self.separable = nn.Conv2d(16, 16, (1, 16), groups=16, bias=False)
        
    def forward(self, x):
        return self.separable(self.spatial(self.temporal(x)))

By keeping the parameter count low (~1.6K params), EEGNet trained effectively on limited data, achieving an average balanced accuracy of 72.3% on motor imagery tasks, and up to 86.5% on clean subjects.

ShallowConvNet

Inspired by Filter Bank Common Spatial Patterns (FBCSP), ShallowConvNet uses a larger parameter footprint (~29K params) and temporal pooling to decode band-power features. It proved highly effective for physical grip execution, achieving a peak accuracy of 83.3%.

3. Deployment: Model Serialization and Sidecar Architecture

Running PyTorch deep learning models directly inside a game engine like Unity is challenging because C# lacks mature tensor frameworks, and running inference on CPU threads inside the main rendering loop causes frame drops.

To solve this, we used a Python Sidecar Architecture. The deep learning model is hosted in an isolated Python process. We serialize the trained PyTorch weights to an ONNX (Open Neural Network Exchange) format to optimize inference speeds:

import torch

# Export the trained EEGNet model to ONNX format
dummy_input = torch.randn(1, 1, 32, 250) # Batch size 1, 1 channel, 32 electrodes, 250 samples
torch.onnx.export(
    model, 
    dummy_input, 
    "eegnet_model.onnx",
    input_names=["eeg_data"],
    output_names=["prediction_logits"],
    dynamic_axes={"eeg_data": {0: "batch_size"}}
)

The sidecar process loads the model via onnxruntime, pulls preprocessed EEG windows from our asynchronous queues, runs inference, and packages command predictions into JSON structures, streaming them via UDP to Unity under 500ms.

4. Solving Clinical Challenges: Voluntary vs. Involuntary Actions

A major concern for clinical clinicians using BCIs is artifact false-positives. If a patient experiences a muscle spasm, a standard BCI might misinterpret the high voltage as a command, sending the wheelchair forward.

To prevent this, our clinical dashboard application (NeuroDiag)—built with Electron, React, and a high-performance PixiJS WebGL waveform renderer—implements an Intentionality Algorithmic Gate.

The system continuously monitors the pre-motor cortex channels (C3 and C4) on the 32-channel layout. If a significant muscle artifact spike occurs without preceding pre-motor activity, the system tags the event as "Involuntary/Spasm" and blocks wheelchair actuation.

Engineering Takeaways

Decouple acquisition from classification: Real-time bio-signal processing requires dedicated threads. One thread should pull raw data from the device at a constant rate, while a separate worker thread runs preprocessing and neural network inference.
Causal filters are mandatory for real-time control: Never use forward-backward (zero-phase) filters in a live loop; they introduce look-ahead leakage and processing delays. Causal IIR filters are the correct choice for low-latency pipelines.
Use UDP for real-time streaming: When building control bridges to game engines or simulations, TCP's packet loss recovery creates latency spikes. UDP ensures the lower-latency controller always acts on the freshest signal.