← Back to Research Radar
Academic Publication Academic Publication

Benchmarking Large Language Models in Retrieval-Augmented Generation

315
Citations
March 24, 2024
Published Date

Research Abstract & Technology Focus

Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented generation on different large language models, which make it challenging to identify the potential bottlenecks in the capabilities of RAG for different LLMs. In this paper, we systematically investigate the impact of Retrieval-Augmented Generation on large language models. We analyze the performance of different large language models in 4 fundamental abilities required for RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness. To this end, we establish Retrieval-Augmented Generation Benchmark (RGB), a new corpus for RAG evaluation in both English and Chinese. RGB divides the instances within the benchmark into 4 separate testbeds based on the aforementioned fundamental abilities required to resolve the case. Then we evaluate 6 representative LLMs on RGB to diagnose the challenges of current LLMs when applying RAG. Evaluation reveals that while LLMs exhibit a certain degree of noise robustness, they still struggle significantly in terms of negative rejection, information integration, and dealing with false information. The aforementioned assessment outcomes indicate that there is still a considerable journey ahead to effectively apply RAG to LLMs.
benchmarking large language models retrieval-augmented generation
Read Full Literature

AI Semantic Synergy Context

Connecting this academic literature to real-world market discussions and products.

crossref.org › academic paper
64%
🔥

Benchmarking Large Language Models in Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of ...
crossref.org › academic paper
0%

Improving medical reasoning through retrieval and self-reflection with retrieval-augmented large language models

Abstract Summary Recent proprietary large language models (LLMs), such as GPT-4, have achieved a milestone in tackling diverse challenges in the ...
crossref.org › academic paper
0%

Biomedical knowledge graph-optimized prompt generation for large language models

Abstract Motivation Large language models (LLMs) are being adopted at an unprecedented rate, yet still face challenges in knowledge-intensive dom...
crossref.org › academic paper
0%

When large language models meet personalization: perspectives of challenges and opportunities

AbstractThe advent of large language models marks a revolutionary breakthrough in artificial intelligence. With the unprecedented scale of training and model parameters, the capability of large lan...
crossref.org › academic paper
0%

Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond

This article presents a comprehensive and practical guide for practitioners and end-users working with Large Language Models (LLMs) in their downstream Natural Language Processing (NLP) tasks. We p...

Frequently Asked Questions (FAQ)

Curated market intelligence mapped to this research.

What is the core focus of the research titled 'Benchmarking Large Language Models in Retrieval-Augmented Generation'?

This literature focuses on: Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented generation on different large ...

Are there open-source GitHub repositories related to Benchmarking Large Language Models in Retrieval-Augmented Generation?

Yes, open-source projects like FreedomIntelligence/OpenClaw-Medical-Skills (The largest open-source medical AI skills library for OpenClaw🦞.) are actively building upon these concepts.

Which startups are commercializing the technology behind Benchmarking Large Language Models in Retrieval-Augmented Generation?

Products like Ollang DX are bringing this to market. Their focus is: The AI Language Execution Layer for Enterprise.

What other academic literature is closely related to 'Benchmarking Large Language Models in Retrieval-Augmented Generation'?

Yes, highly correlated activity was mapped. An entry titled 'Benchmarking Large Language Models in Retrieval-Augmented Generation' discusses this: Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing re...

Cite this Market Intelligence Report

Reference our AI-mapped synergy between this research and the commercial market to instantly build authority.

Commercial Realization

Startups and Open Source tools heavily associated with the concepts explored in this paper.

Associated Media Narrative