--- title: "Decoding Movement Intention: Building a Real-Time Brain-Computer Interface with PyTorch and Unity" description: "A technical walkthrough of a real-time BCI pipeline using EEG preprocessing, PyTorch classification, and Unity simulation." date: "2026-06-30" tags: [BCI, PyTorch, Unity, EEG, Real-Time Systems] keywords: [brain computer interface, EEG, PyTorch, Unity 3D, EEGNet, real-time signal processing, motor imagery] image: "/My.jpeg" imageAlt: "Murat Tut portfolio image" aiSummary: "This article explains the architecture of a real-time BCI system, from EEG acquisition and preprocessing to deep learning classification, low-latency communication, and Unity-based simulation control." --- *How we built a complete BCI pipeline using LSL synchronization, MNE-Python, and EEGNet to control a virtual wheelchair with brain waves.* Controlling machines with thoughts was once the exclusive domain of science fiction. Today, Brain-Computer Interfaces (BCIs) are a reality, offering pathways to restore independence to individuals with severe motor impairments. But building a BCI is one of the most complex interdisciplinary engineering tasks in computer science. It sits at the intersection of neuroscience, signal processing, real-time communications, deep learning, and 3D simulation. Our graduation team at Çankaya University set out to build **NeuroMotion**: a complete BCI system that captures microvolt-level electrical potentials from the scalp, filters out massive muscular noise, decodes motor imagery intentions using convolutional neural networks, and streams commands to a virtual wheelchair simulator in real-time. Here is the technical blueprint of how we designed, trained, and deployed the system. --- ## The BCI Architecture: From Scalp to Simulation To control an object smoothly in a 3D environment, the system must operate under a strict latency budget. In virtual reality or active simulations, any delay between user intent and object actuation longer than **500 milliseconds** causes sensory mismatch, resulting in severe motion sickness. To meet this ceiling, we designed a decoupled, multi-threaded pipeline: ``` ┌─────────────────────────────────────────────────────────────┐ │ LSL Streams │ └──────────────────────────────┬──────────────────────────────┘ │ EEG Data (500 Hz) ▼ Asynchronous Queue Buffer │ ▼ SciPy Preprocessing Pipeline (1-50 Hz Band-pass + Notch) │ ▼ PyTorch EEGNet Classifier (Left, Right, Foot, Idle) │ ▼ UDP JSON Command Packets │ ▼ Unity 3D Wheelchair Controller ``` --- ## 1. Real-Time Signal Processing (MNE-Python & SciPy) EEG electrodes capture electrical potentials in the microvolt ($\mu V$) range. These signals are constantly contaminated by environmental electromagnetic noise (power line interference at 50/60 Hz) and physiological noise, such as eye blinks (EOG) or muscle contraction (EMG). To isolate motor intent, we built an optimized, real-time preprocessing pipeline using **MNE-Python** and **SciPy**: 1. **Lab Streaming Layer (LSL)**: We used the LSL protocol to capture streams from a 32-channel Mitsar-202 EEG device. LSL handles sub-millisecond synchronization across threads, aligning raw signals with event triggers. 2. **Causal Filtering**: We applied a digital causal IIR band-pass filter (1–50 Hz) and a notch filter at 50 Hz. We chose a causal filter because, unlike zero-phase filters, it does not require future data points, keeping processing latency under 5ms. 3. **Sliding Windows**: The preprocessed data was chunked into a sliding window of 250 samples at a 250 Hz sampling rate, updating every 200ms for continuous prediction. --- ## 2. Deep Learning Core (EEGNet vs. ShallowConvNet) Once we had a clean window of spatial-temporal signals, we needed to classify the user's mental state into four classes: **Left Hand**, **Right Hand**, **Foot**, or **Idle**. We evaluated two main deep learning architectures implemented via PyTorch: ### EEGNet EEGNet is a compact convolutional neural network designed specifically for BCIs. It utilizes **depthwise and separable convolutions** to extract spatial and temporal features without triggering parameter explosion: ```python # Simplified representative concept of an EEGNet block class EEGNetBlock(nn.Module): def __init__(self, channels, samples): super().__init__() # Temporal Convolution to learn frequency filters self.temporal = nn.Conv2d(1, 8, (1, 64), padding='same', bias=False) # Depthwise Spatial Convolution to map channel relationships self.spatial = nn.Conv2d(8, 16, (channels, 1), groups=8, bias=False) self.separable = nn.Conv2d(16, 16, (1, 16), groups=16, bias=False) def forward(self, x): return self.separable(self.spatial(self.temporal(x))) ``` By keeping the parameter count low (~1.6K params), EEGNet trained effectively on limited data, achieving an average balanced accuracy of **72.3%** on motor imagery tasks, and up to **86.5%** on clean subjects. ### ShallowConvNet Inspired by Filter Bank Common Spatial Patterns (FBCSP), ShallowConvNet uses a larger parameter footprint (~29K params) and temporal pooling to decode band-power features. It proved highly effective for physical grip execution, achieving a peak accuracy of **83.3%**. --- ## 3. Deployment: Model Serialization and Sidecar Architecture Running PyTorch deep learning models directly inside a game engine like Unity is challenging because C# lacks mature tensor frameworks, and running inference on CPU threads inside the main rendering loop causes frame drops. To solve this, we used a **Python Sidecar Architecture**. The deep learning model is hosted in an isolated Python process. We serialize the trained PyTorch weights to an **ONNX** (Open Neural Network Exchange) format to optimize inference speeds: ```python import torch # Export the trained EEGNet model to ONNX format dummy_input = torch.randn(1, 1, 32, 250) # Batch size 1, 1 channel, 32 electrodes, 250 samples torch.onnx.export( model, dummy_input, "eegnet_model.onnx", input_names=["eeg_data"], output_names=["prediction_logits"], dynamic_axes={"eeg_data": {0: "batch_size"}} ) ``` The sidecar process loads the model via `onnxruntime`, pulls preprocessed EEG windows from our asynchronous queues, runs inference, and packages command predictions into JSON structures, streaming them via UDP to Unity under 500ms. --- ## 4. Solving Clinical Challenges: Voluntary vs. Involuntary Actions A major concern for clinical clinicians using BCIs is artifact false-positives. If a patient experiences a muscle spasm, a standard BCI might misinterpret the high voltage as a command, sending the wheelchair forward. To prevent this, our clinical dashboard application (**NeuroDiag**)—built with Electron, React, and a high-performance PixiJS WebGL waveform renderer—implements an **Intentionality Algorithmic Gate**. The system continuously monitors the pre-motor cortex channels (C3 and C4) on the 32-channel layout. If a significant muscle artifact spike occurs without preceding pre-motor activity, the system tags the event as "Involuntary/Spasm" and blocks wheelchair actuation. --- ## Engineering Takeaways 1. **Decouple acquisition from classification**: Real-time bio-signal processing requires dedicated threads. One thread should pull raw data from the device at a constant rate, while a separate worker thread runs preprocessing and neural network inference. 2. **Causal filters are mandatory for real-time control**: Never use forward-backward (zero-phase) filters in a live loop; they introduce look-ahead leakage and processing delays. Causal IIR filters are the correct choice for low-latency pipelines. 3. **Use UDP for real-time streaming**: When building control bridges to game engines or simulations, TCP's packet loss recovery creates latency spikes. UDP ensures the lower-latency controller always acts on the freshest signal.