Scientists have long wanted to understand human thoughts by reading brain signals. Recent advances in artificial intelligence (AI) and machine learning have made it possible to turn brain signals into images.
These signals come from functional magnetic resonance imaging (fMRI), a method that measures brain activity by detecting blood flow changes. However, earlier methods could only recreate what someone sees or imagines without letting them change those images. Connecting brain signals to language is hard because brain activity is complex and unclear. Changing specific parts of a mental image while keeping others the same is also a challenge.
Researchers from universities in Japan, China, Saudi Arabia, and the UAE created DreamConnect, a system that reads brain signals and edits mental images using simple language instructions.
DreamConnect uses a dual-stream diffusion framework, a technique that processes brain signals in two steps. The first step turns brain signals into rough images, and the second step refines those images based on language prompts, like asking to change a horse into a unicorn. To make this work smoothly, the system uses an asynchronous diffusion strategy, meaning it first creates a basic image outline before applying specific changes. Large language models (LLMs) help focus edits on the right parts of the image without affecting others.
The researchers tested DreamConnect with a large fMRI dataset called the Natural Scenes Dataset and used AI to create extra training data.
Advancing imagination and technology
Tests showed DreamConnect can recreate images as well as the best existing systems and is better at editing images based on instructions. Unlike other methods, it works directly with brain signals, making the process faster and more precise. This technology could help designers visualize ideas quickly, assist patients in expressing thoughts visually for therapy, or support creative learning. DreamConnect shows how human imagination and AI can work together, opening new possibilities for technology and creativity.
The researchers have described the methods and results of this study in a paper published in Visual Intelligence.