Meta has unveiled a new open-source AI model called ImageBind, which has the ability to link multiple streams of data including text, audio, visual data, temperature, and movement readings. While the model is currently a research project and has no immediate consumer or practical applications, it demonstrates the potential of generative AI systems that can create immersive, multisensory experiences.

The core concept of the research is linking together multiple types of data into a single multidimensional index, which is known as an “embedding space” in the world of AI. The recent boom in generative AI has relied on systems that link together text and images during the training stage, looking for patterns in visual data while connecting that information to descriptions of the images. The same is true of many AI tools that generate video or audio.

Meta’s ImageBind is the first model to combine six types of data into a single embedding space, including visual, thermal, text, audio, depth information, and movement readings generated by an inertial measuring unit (IMU). The model has the potential to create virtual reality experiences that not only generate audio and visual input but also environmental information and movement on a physical stage.

The research is also interesting because Meta is open-sourcing the underlying model, which is becoming an increasingly scrutinized practice in the world of AI. Advocates argue that open-sourcing allows third parties to scrutinize the systems for faults and ameliorate some of their failings, while it also provides a commercial benefit as it allows companies to recruit third-party developers as unpaid workers to improve their work.

Meta’s open-sourcing of its AI model comes at a time when rivals like OpenAI and Google have become increasingly secretive, but it remains to be seen whether the model will have any immediate commercial applications. The company’s lack of commercial achievement in AI has enabled this approach, and for the time being, it is continuing with its open-source strategy.

Meta has unveiled a new open-source AI model called ImageBind, which has the ability to link multiple streams of data including text, audio, visual data, temperature, and movement readings. While the model is currently a research project and has no immediate consumer or practical applications, it demonstrates the potential of generative AI systems that can create immersive, multisensory experiences.

The core concept of the research is linking together multiple types of data into a single multidimensional index, which is known as an “embedding space” in the world of AI. The recent boom in generative AI has relied on systems that link together text and images during the training stage, looking for patterns in visual data while connecting that information to descriptions of the images. The same is true of many AI tools that generate video or audio.

Meta’s ImageBind is the first model to combine six types of data into a single embedding space, including visual, thermal, text, audio, depth information, and movement readings generated by an inertial measuring unit (IMU). The model has the potential to create virtual reality experiences that not only generate audio and visual input but also environmental information and movement on a physical stage.

The research is also interesting because Meta is open-sourcing the underlying model, which is becoming an increasingly scrutinized practice in the world of AI. Advocates argue that open-sourcing allows third parties to scrutinize the systems for faults and ameliorate some of their failings, while it also provides a commercial benefit as it allows companies to recruit third-party developers as unpaid workers to improve their work.

Meta’s open-sourcing of its AI model comes at a time when rivals like OpenAI and Google have become increasingly secretive, but it remains to be seen whether the model will have any immediate commercial applications. The company’s lack of commercial achievement in AI has enabled this approach, and for the time being, it is continuing with its open-source strategy.