← Back to Research Radar
Academic Publication Academic Publication

A framework for human evaluation of large language models in healthcare derived from literature review

249
Citations
September 28, 2024
Published Date

Research Abstract & Technology Focus

AbstractWith generative artificial intelligence (GenAI), particularly large language models (LLMs), continuing to make inroads in healthcare, assessing LLMs with human evaluations is essential to assuring safety and effectiveness. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare across various medical specialties and addresses factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Our literature review of 142 studies shows gaps in reliability, generalizability, and applicability of current human evaluation practices. To overcome such significant obstacles to healthcare LLM developments and deployments, we propose QUEST, a comprehensive and practical framework for human evaluation of LLMs covering three phases of workflow: Planning, Implementation and Adjudication, and Scoring and Review. QUEST is designed with five proposed evaluation principles: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence.
Read Full Literature

Correlated Market Trend: .net Framework

Bridging academia to market: The 60-day public search velocity mapping directly to the core technology of this paper. Dashed line represents 7-day moving average.

AI Semantic Synergy Context

Connecting this academic literature to real-world market discussions and products.

crossref.org › academic paper
0%

A Survey on Evaluation of Large Language Models

Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role...

crossref.org › academic paper
0%

Large Language Models in Healthcare and Medical Domain: A Review

The deployment of large language models (LLMs) within the healthcare sector has sparked both enthusiasm and apprehension. These models exhibit the remarkable ability to provide proficient responses...

crossref.org › academic paper
0%

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Abstract Clinical decision-making is one of the most impactful parts of a physician’s responsibilities and stands to benefit greatly from artificial intelligence solutions and lar...

crossref.org › academic paper
0%

Testing and Evaluation of Health Care Applications of Large Language Models

ImportanceLarge language models (LLMs) can assist in various health care activities, but current evaluation approaches may not adequately identify the most useful application areas.ObjectiveTo summ...

crossref.org › academic paper
0%

Large Language Model Influence on Diagnostic Reasoning

ImportanceLarge language models (LLMs) have shown promise in their performance on both multiple-choice and open-ended medical reasoning examinations, but it remains unknown whether the use of such ...

Frequently Asked Questions (FAQ)

Curated market intelligence mapped to this research.

What is the core focus of the research titled 'A framework for human evaluation of large language models in healthcare derived from literature review'?

This literature focuses on: AbstractWith generative artificial intelligence (GenAI), particularly large language models (LLMs), continuing to make inroads in healthcare, assessing LLMs with human evaluations is essential to assuring safety and effectiveness. This study revie...

Are there open-source GitHub repositories related to A framework for human evaluation of large language models in healthcare derived from literature review?

Yes, open-source projects like slowmist/openclaw-security-practice-guide (This guide is designed for OpenClaw itself (Agent-facing), not as a traditional human-only hardening checklist.) are actively building upon these concepts.

Which startups are commercializing the technology behind A framework for human evaluation of large language models in healthcare derived from literature review?

Products like Offsite are bringing this to market. Their focus is: Build teams of humans and agents, watch them work..

What other academic literature is closely related to 'A framework for human evaluation of large language models in healthcare derived from literature review'?

Yes, highly correlated activity was mapped. An entry titled 'A Survey on Evaluation of Large Language Models' discusses this: Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various a...

Cite this Market Intelligence Report

Reference our AI-mapped synergy between this research and the commercial market to instantly build authority.

Commercial Realization

Startups and Open Source tools heavily associated with the concepts explored in this paper.

Associated Media Narrative