MOSAIC: A Multimodal Semantic-Oriented Alignment with Integrated Contrastive Learning for Multimodal Knowledge Graph Construction from Scientific Documents
Abstract
Multimodal Knowledge Graphs (MMKGs) offer a promising paradigm for integrating heterogeneous sources into a unified, queryable, semantically structured representation. However, existing MMKG construction pipelines remain predominantly text-centric, extracting information from textual passages while leaving much of the visual and structural knowledge in scientific papers unrepresented. This produces fragmented graphs with disconnected components, isolated singleton nodes, and weak cross-modal connectivity, reducing the effectiveness of retrieval-augmented generation (RAG) systems that rely on interconnected graph traversal for multi-hop reasoning and evidence aggregation. We propose Multimodal Semantic-Oriented Alignment with Integrated Contrastive Learning (MOSAIC), an end-to-end framework for constructing semantically coherent MMKGs from heterogeneous scientific documents. MOSAIC introduces four complementary contributions: (i) adaptive clustering via HDBSCAN, which infers data-driven entity boundaries without the brittle ε hyperparameters of conventional DBSCAN; (ii) a confidence-aware cross-modal alignment mechanism applying cosine-similarity gating to selectively invoke Large Language Models (LLMs), reducing spurious alignments and computational overhead; (iii) post-fusion semantic bridging, which links semantically related but structurally disconnected components through cosine-similarity-weighted bridge edges; and (iv) self-supervised contrastive embedding fine-tuning…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.