← Back to Research Radar
Academic Publication Academic Publication

Deep Multimodal Data Fusion

327
Citations
October 31, 2024
Published Date

Research Abstract & Technology Focus

Multimodal Artificial Intelligence (Multimodal AI), in general, involves various types of data (e.g., images, texts, or data collected from different sensors), feature engineering (e.g., extraction, combination/fusion), and decision-making (e.g., majority vote). As architectures become more and more sophisticated, multimodal neural networks can integrate feature extraction, feature fusion, and decision-making processes into one single model. The boundaries between those processes are increasingly blurred. The conventional multimodal data fusion taxonomy (e.g., early/late fusion), based on which the fusion occurs in, is no longer suitable for the modern deep learning era. Therefore, based on the main-stream techniques used, we propose a new fine-grained taxonomy grouping the state-of-the-art (SOTA) models into five classes: Encoder-Decoder methods, Attention Mechanism methods, Graph Neural Network methods, Generative Neural Network methods, and other Constraint-based methods. Most existing surveys on multimodal data fusion are only focused on one specific task with a combination of two specific modalities. Unlike those, this survey covers a broader combination of modalities, including Vision + Language (e.g., videos, texts), Vision + Sensors (e.g., images, LiDAR), and so on, and their corresponding tasks (e.g., video captioning, object detection). Moreover, a comparison among these methods is provided, as well as challenges and future directions in this area.
multimodal fusion
Read Full Literature

AI Semantic Synergy Context

Connecting this academic literature to real-world market discussions and products.

crossref.org › academic paper
0%

Deep Multimodal Data Fusion

Multimodal Artificial Intelligence (Multimodal AI), in general, involves various types of data (e.g., images, texts, or data collected from different sensors), feature engineering (e.g., extraction...
crossref.org › academic paper
0%

A technical review of multi-omics data integration methods: from classical statistical to deep generative approaches

Abstract The rapid advancement of high-throughput sequencing and other assay technologies has resulted in the generation of large and complex multi-omics datasets, offering unprecede...
openalex.org › research concept
0%

An improved hypergraph convolutional network based on multi-channel fusion signals for semi-supervised fault diagnosis of autonomous underwater vehicle thrusters

Abstract Autonomous underwater vehicle (AUV), as a highly efficient tool for ocean exploration, relies on thrusters whose fault diagnosis is a key aspect to ensure safe navigation. However, single-...
crossref.org › academic paper
0%

A survey on multimodal large language models

ABSTRACT Recently, the multimodal large language model (MLLM) represented by GPT-4V has been a new rising research hotspot, which uses powerful large language models (LLMs) as a brai...
news.ycombinator.com › AI insight
0%

Show HN: DeepTable – an API that converts messy Excel files into structured data

DeepTable addresses a pervasive and costly data ingestion problem for enterprises: transforming complex, unstructured Excel data into usable, structured formats. The explicit mention of LLMs failin...

Frequently Asked Questions (FAQ)

Curated market intelligence mapped to this research.

What is the core focus of the research titled 'Deep Multimodal Data Fusion'?

This literature focuses on: Multimodal Artificial Intelligence (Multimodal AI), in general, involves various types of data (e.g., images, texts, or data collected from different sensors), feature engineering (e.g., extraction, combination/fusion), and decision-making (e.g., ...

Are there open-source GitHub repositories related to Deep Multimodal Data Fusion?

Yes, open-source projects like fikrikarim/parlor (On-device, real-time multimodal AI. Have natural voice and vision conversations with an AI that runs entirely on your machine. Powered by Gemma 4 E...) are actively building upon these concepts.

Which startups are commercializing the technology behind Deep Multimodal Data Fusion?

Products like Qwen3.6-Plus are bringing this to market. Their focus is: Multimodal AI optimized for real-world coding agents.

What other academic literature is closely related to 'Deep Multimodal Data Fusion'?

Yes, highly correlated activity was mapped. An entry titled 'Deep Multimodal Data Fusion' discusses this: Multimodal Artificial Intelligence (Multimodal AI), in general, involves various types of data (e.g., images, texts, or data collected from differe...

How is the concept of 'Deep Multimodal Data Fusion' being discussed by engineers on Hacker News?

Yes, highly correlated activity was mapped. An entry titled 'Show HN: DeepTable – an API that converts messy Excel files into structured data' discusses this: DeepTable addresses a pervasive and costly data ingestion problem for enterprises: transforming complex, unstructured Excel data into usable, struc...

Cite this Market Intelligence Report

Reference our AI-mapped synergy between this research and the commercial market to instantly build authority.

Commercial Realization

Startups and Open Source tools heavily associated with the concepts explored in this paper.

Associated Media Narrative