Teaching LLMs the Nuances of Hospital Funding Instruments

Recent advancements in large language models (LLMs) have sparked significant interest across a broad spectrum of use cases and business domains [2, 5, 12]. The objective of this paper is to investigate whether large language models (LLMs) can effectively understand and interpret the complex logic behind hospital funding instruments, such as the NordDRG system [3]. The integration of Large Language Models (LLMs) with the NordDRG system enhances AI-driven healthcare management, particularly in CaseMix systems. Traditionally developed by a small group of experts, the complex logic of NordDRG could be made more accessible to a wider range of stakeholders through LLMs, enabling natural language interaction and broader engagement. This paper evaluates the effectiveness of three distinct AI design patterns - Custom GPT, Retrieval-Augmented Generation (RAG), and Multi-Agent Systems (MAS) - in developing an AI architecture capable of understanding the NordDRG specification. Our findings suggest that most current general LLMs, such as GPT-4, struggle to effectively handle the NordDRG logic. However, when LLMs are enhanced with Retrieval Augmented Generation (RAG) [6], the enhanced NordDRG AI agent is capable of engaging in natural language conversations with various stakeholders. Also, the introduction of the most advanced models, such as GPT-4o, reaches the same level of results as previous generation models with RAG. This paper provides a comparative analysis of three AI design patterns. Our research contributes by establishing a CaseMix benchmark dataset for evaluating large language model (LLM) capabilities using the NordDRG system. To the best of our knowledge, this is the first benchmark in the hospital funding sector. Our findings highlight the potential of LLMs to transform CaseMix system development by providing robust, scalable, and context-aware AI solutions.