Identifying nursing sensitive indicators from electronic health records in acute cardiac care―Towards intelligent automated assessment of care quality

Abstract Aim The aim of this study is to explore the potential of using electronic health records for assessment of nursing care quality through nursing‐sensitive indicators in acute cardiac care. Background Nursing care quality is a multifaceted phenomenon, making a holistic assessment of it difficult. Quality assessment systems in acute cardiac care units could benefit from big data‐based solutions that automatically extract and help interpret data from electronic health records. Methods This is a deductive descriptive study that followed the theory of value‐added analysis. A random sample from electronic health records of 230 patients was analysed for selected indicators. The data included documentation in structured and free‐text format. Results One thousand six hundred seventy‐six expressions were extracted and divided into (1) established and (2) unestablished expressions, providing positive, neutral and negative descriptions related to care quality. Conclusions Electronic health records provide a potential source of information for information systems to support assessment of care quality. More research is warranted to develop, test and evaluate the effectiveness of such tools in practice. Implications for Nursing Management Knowledge‐based health care management would benefit from the development and implementation of advanced information systems, which use continuously generated already available real‐time big data for improved data access and interpretation to better support nursing management in quality assessment.


| BACKGROUND
Providing high-quality health services is key in managing cardiovascular diseases and improving patient outcomes (Thomas et al., 2018).
Despite the efforts in achieving a downward trend of disease prevalence in high-income countries (Amini et al., 2017), cardiovascular diseases are still an important cause of mortality globally, causing over 38% of premature deaths in 2019 (WHO, 2021). Improving service quality has been estimated to prevent up to 2.5 million of these deaths annually (WHO, 2020a). Acquiring good quality is a continuous cycle of planning, implementing and evaluating quality improvement activities in all levels of a health care system (WHO, 2020b). On the clinical level, systematic quality assessment of cardiac care using illness-specific quality metrics has the potential to improve care outcomes, but to be effective, the results should be connected to patient-related outcomes, such as patient experience or mortality (Chatterjee & Joynt, 2014). The use of nursing-sensitive indicators could provide a more holistic perspective, focusing on the assessment of nursing care quality (Heslop et al., 2014). However, previous literature shows little evidence of the application of these indicators in the assessment of cardiac care.
The use of nursing-sensitive indicators can help build a foundation to quantify, measure and improve dynamic nursing care quality within all domains of nursing (Afanef et al., 2021). In fact, systematic assessment of quality based on nursing-sensitive indicators have shown to improve the quality of nursing care (Elgseer et al., 2021). For example, nurse-to-patient ratios in acute care have been proven to influence a variety of patient outcomes, such as mortality, providing valuable information when constructing optimal staffing models (Driscoll et al., 2018). In cardiac care, the use of indicators in monitoring the quality of highly specialized procedures, such as cardiac catheterization, has an impact on their quality and safety (Shen et al., 2021). However, issues regarding the selection of appropriate indicators to use, report and embed in clinical practice hinder their introduction in continuous quality assessment and improvement initiatives (Burston et al., 2013). A recent study indicated that barriers to quality improvement for nurse managers included a lack of timely data presented in a usable and easyto-access manner (Alexander et al., 2022). Additionally, information needs vary between different actors and settings in health care, such as professionals, units and work shifts, creating a need for tailored and dynamic information systems, which respond to different users' individual needs (Peltonen et al., 2019). From a nurse's standpoint, providing requisite data entries for quality assessment generally increases documentation time and results in a need to make duplicate entries to differing documentation systems (Elgseer et al., 2021).
The potential of using electronic health records (EHRs) as a secondary data source for quality assessment has been recognized since their introduction. From early on, concerns regarding the accurate portrayal of the complexity of care and data requirements for comprehensive quality assessment as well as accuracy, comparability and timeliness of extracted data have been presented (Roth et al., 2009).
These issues are as relevant today, as the use of single nursingsensitive indicators extracted from EHRs to assess nursing care quality is on the increase. One example is the automatic detection of medication errors by comparing medication prescribing and documentation as extracted from EHRs (Kirkendal et al., 2020).
Research utilizing multiple nursing-sensitive indicators simultaneously to form a holistic picture of care quality in acute cardiac care is lacking to our knowledge. In the intensive care setting, efforts to use EHR data when manually extracting information on care quality are showing encouraging results. A study by Seaman et al. (2017) showed that combining structured and free text entries describing selected quality outcome measures provided accurate information on quality. The included measures were heavy sedation, use of physical restraints, presence and intensity of pain, unplanned extubation and pressure injuries. The used manual extraction tool in the study was considered labour-intensive, and the use of free text notes prohibited the use of automated extraction methods available to the researchers (Seaman et al., 2017).
Compared with manual extraction methods, automated methods to extract quality information from EHRs show advantages in data integrity, reliability and accuracy (Brundin-Mather et al., 2018). Introducing artificial intelligence (AI), a range of intelligent technologies present the best possible solutions in automated problem detection or prediction to improve safe patient care (Sensmeier, 2017). Methods utilizing AI have proven their applicability. Machine learning approaches, for example, hold the potential to be used in identifying inpatient fall risks from EHR's and administrative data (Lindberg et al., 2020). Natural language processing (NLP), in turn, has been applied as a novel way to process and present information gathered from free text EHR notes efficiently (Juhn & Liu, 2020;Koleck et al., 2019). All in all, the use of AI in nursing-relevant tasks is on the rise, with over 50% of said technologies using EHRs as the data source and approximately 10% of these technologies focusing on NLP .
Previous research findings indicate that quality assessment initiatives could benefit from the rich data EHRs have to offer and the use of carefully selected nursing-sensitive indicators. This study addresses the gap of knowledge in using nursing-sensitive indicators in assessing care quality in acute cardiac care units using EHRs as a data source.
The results support the application of NLP methods on free text narratives, which could have a potential impact on effective and automated real time care quality assessment. The aim of this study was to examine the potential of using EHRs in assessing care quality through selected nursing-sensitive indicators in acute cardiac care. The study question was: What expressions indicating nursing care quality can be identified and extracted from structured and free-text notes in patients' EHRs? 2 | METHODS

| Research design
This retrospective descriptive study was guided by the theory of value-adding analysis described by Eakin and Gladstone (2020).
Value-adding analysis is a form of qualitative research, in which conventional qualitative analysis methods are complemented by a process of analysis aiming to construct concepts on a more abstract level. The analysis held four interrelated features: interpretation, contextualization, "creative presence of the researcher" and critical inquiry. The qualitative methods used included deductive and inductive (Elo & Kyngäs, 2008) as well as summative content analysis (Hsieh & Shannon, 2005) aiming to describe and quantify the phenomena of interest. Reporting was conducted according to the COREQ-checklist for qualitative studies (Tong et al., 2007).

| Setting
The EHR data used in this study were collected from a cardiac centre of one out of 21 hospital districts in Finland. This cardiac centre is part of highly specialized medical care offered at public hospitals. The centre operates in a university hospital, performing acute care including cardiac and lung surgeries and day surgeries, as well as outpatient follow-ups. In 2021, the centre had 20,562 outpatient visits and a total of 4963 in-hospital care episodes.
The EHR system used in the cardiac centre is a multi-professional system developed for documenting all relevant perspectives related to a patient's clinical care, including, for example, nursing and physician notes, prescriptions and treatment charts, as well as laboratory examinations and results from each service event, where the provider has interacted with a patient for care or medical treatment. Following the Finnish decree on patient records (298/2009) and the decree on patient status and rights (758/1992), all necessary and comprehensive information regarding the arrangement, planning, execution and monitoring of good patient care need to be documented in this EHR system. This information includes preliminary information regarding the patient's health status, illnesses or injuries, observations made by the caretakers, laboratory and imaging results as well as care measures to restore or maintain the patient's health or alleviate suffering. Entries made in the EHRs must be clear and understandable, containing only generally established concepts and abbreviations.
The documentation in the EHRs follows a title structure where predefined classifications are complemented using narrative free text. All health care professionals make entries to the system following codes and standards specific to their own professional groups. For nurses, the system allows to document patient care systematically by phase of nursing process. Structured data elements, such as nursing diagnosis, interventions, outcomes, intensity and discharge summary, follow the Finnish Care Classification (FinCC) system as presented by the Finnish institute for health and welfare (Kinnunen et al., 2021).
The documentation entries by different health care professionals form together an entity describing the patient's health situation and clinical pathway in a holistic and multi-professional way. Hence, when evaluating patient care, all entries made in the EHR-system are essential when seeking for an understanding of the overall picture.

| Participants
The data were collected from patients (n = 1852) admitted and treated at the acute cardiac care units of the cardiac centre during January 2020. The data collection point was selected to ensure the most recent data but to avert the possible skewness caused by the global outbreak of COVID-19. The data included all clinical care entries made in the centre, such as free text nursing and physician notes as well as structured clinical measurements, patient evaluations and laboratory results from all units. Only operating room reports, intensive care nursing narratives and radiology images from the cardiac centre were excluded.
The sample consisted of the EHRs of 230 patients. To ensure as wide, versatile and reliable picture as possible regarding patient care, all available entries made in the EHR were included in this study. Only including one source of the documentation, such as the nursing narratives alone, would not provide a comprehensive picture of the care provided, as the documentation in the EHR is done in a multiprofessional way with all different perspectives complementing each other. The included records held 540 care episodes lasting from one to 13 days. They contained 6867 notes, including but not limited to nursing notes (n = 2673) and measurements (n = 2336), physician notes (n = 527), diagnostic notes (n = 377) and laboratory results (n = 347).

| Data collection
The data extraction from the EHR archives was performed by a computer scientist. The data were randomly organized by patient text files. Using a systematic random sampling method with a periodic interval, every fifth text file was selected until no relevant new discoveries were made (Grove, 2017). The point of saturation was determined by following a deductive codebook approach (Kerr et al., 2010), and saturation was reached after 180 patients. The saturation was verified by continuing the analysis with an additional 60 patients' records, and no new expressions were identified.

| Data analysis
The structure of the deductive analysis was operationalized by combining selected and well-established nursing-sensitive indicators used for assessing care quality suitable for the acute cardiac care unit environment. These patient outcome indicators included the National Database of Nursing Quality Indicators (Montalvo, 2007) and nursingsensitive indicators applied to the context of acute care (Heslop et al., 2014) and served as a codebook to guide the deductive analysis process.
Expressions related to nursing care quality, such as words and word pairs, were extracted from the EHRs to a spreadsheet containing a codebook presented as a structured categorization matrix. The expressions were collected and coded by one researcher, with the results analysed and discussed together with another researcher during and after the whole data analysis process. The expressions inside the codebook categories were further divided into subcategories.
They were also quantified in attempts to contextualize the context as well as to further examine the use of the expressions (Hsieh & Shannon, 2005).

| Ethical considerations
This study utilized EHRs, which are classified as personal data. In

| RESULTS
The sample of patients consisted of 98 (43%) women and 132 (57%) men born between 1923 and 1995, ages ranging from 25 to 97 years (standard deviation 41 years). In total, 1676 expressions related to care quality were extracted of which 1235 (73.7%) originated from free text and 441 (26.3%) from structured notes.
The value-adding analysis was concretized as an interpretative inductive analysis. The expressions were divided into two categories: established expressions related to nursing care quality and unestablished expressions related to nursing care quality. Established expressions provided straightforward information directly related to nursing care quality that could be utilized as is or with minor adjustments in a tool assessing care quality. Unestablished expressions, in turn, provided information indirectly related to nursing care quality of which more research is still needed before its application in care quality assessment. Table 1 illustrates quantities of expressions related to nursing care quality extracted from free text and structured notes.  3.1 | Established expressions related to nursing care quality Established expressions related to nursing care quality described experiences of pain, adverse events and patient satisfaction, as described in Figure 1. The subcategory regarding adverse events (n = 59, 4.0% of all) was mainly described in free text (n = 58, 4.7% of free text entries).
Patient falls and fall related injuries described both the occurrence of the fall and details regarding the injury. All these expressions were extracted from free text notes (n = 54, 4.4% of free text entries).
Expressions describing the occurrence (n = 36) were negative verbs indicating the patient falling or slipping during hospitalization. Patient fall related injuries contained information on both the obtaining (n = 11) but also the avoidance (n = 7) of bruises, dints, tears or unde- The subcategory of expressions describing patients' functional health (n = 56, 3.0% of all) were related to a patient's general status and ability to cope at home. Expressions describing the general status of patients were mainly perceived as good with positive descriptions (n = 40). In contrast, all descriptions (n = 9) regarding patients' abilities to cope at home were negative. All expressions describing patients' functional health originated from free text notes (4.5% of all free text entries).

| DISCUSSION
The results present an overview of EHR-based information related to nursing care quality from three differing standpoints that need to be

F I G U R E 2 Unestablished expressions related to nursing care quality
The first standpoint indicates the degree of readiness of nursingsensitive indicators to be used in care quality assessment. Previous research presents ample evidence of the use of nursing-sensitive indicators corresponding to the established expressions found in our study, such as experiences of pain or prevalence of pressure injuries (Seaman et al., 2017). These examples also include research focusing on extracting information related to pain management, ranging from risk identification to the evaluation of analgesic effect or pain prevalence (Nomura et al., 2021). In contrast, before their utilization in systems that help evaluate care quality, more research is needed on unestablished expressions, which are more complicated to interpret.
Careful consideration is warranted on how to use these expressions in the assessment care quality. An ample example is the sub-category describing a patient's physical health containing information on signs and presence of infection. During manual data extraction, single expressions were identified describing hospital acquired infections, a widely accepted nursing quality indicator as presented by Montalvo (2007). Instead of investigating single expressions related to infections as indicating a hospital acquired infection, the timeline and the prevalence of the symptoms must be taken into consideration (Warner et al., 2013). Assessing care quality is perhaps not merely about following single indicators, but more about merging several indicators and examining trends or patterns.
These issues become even more relevant when considering the transition from manual to automated extraction methods. In a study by Ehrentraut et al. (2018), using machine learning methods, the detection of hospital-acquired infections from structured and free text EHR's showed promising results when comparing records from patients diagnosed with a hospital-acquired infection to records from undiagnosed patients. These results indicate the potential of textclassification techniques in real-world applications, reducing the manual data entry labour of health professionals (Ehrentraut et al., 2018.) The second standpoint was the presentation of information extracted through their informational value, as illustrated by the presentation of positive, neutral and negative expressions related to nursing care quality. This approach was selected to highlight that nursing care quality is not just the prevalence of absence of negative adverse events. However, in addition to reflecting the nursing care quality, the expressions also reflect the quality of nursing documentation. In a study investigating nurse prioritization leading to unfinished care, documentation was rated among the middle-ranking tasks , indicating that during an intense shift some interventions or observations face the danger of being left undocumented.
This can lead to issues regarding data accuracy, completeness, and consistency as well as credibility and timeliness (Feder, 2018). When using EHRs as a data source for real-time care quality assessment, these deficiencies must be understood and taken into consideration from the development phase to end user implementation and beyond.
Decision making based on information derived from EHRs requires knowledge not only on the quality indicators used, but also on how the information should be interpreted. Nurse managers' competence requirements should therefore not only address quality assessment and improvement, but also comprehensive expertise on information management. In this study, efforts were made to meet the specific information needs of nurse managers in cardiac care units. However, information on care quality would benefit not only the day-to-day management, but also management on other decision-making levels in health care organizations, providing important bottom-up information to support upper management as well. From a patient's perspective, care is a continuum of contacts with varying professionals in different settings in the health care system. To form an overall picture, it would be beneficial to acquire quality information from all points of the patient's care pathway (Hanefeld et al., 2017). Expanding future research to explore the needs in other health care settings would benefit both nursing management and patient care.
The importance of promoting guidelines for high quality nursing documentation is evident. An ill-fitted documentation system may result in inconsistent descriptions regarding patient care, disabling holistic care quality assessment (Rossi et al., 2022).  (Seaman et al., 2017). Automated data extraction methods would therefore highly benefit from the introduction of NLP to support the secondary use of EHR data.
So far, NLP has been used to identify single nursing-sensitive indicators such as patient falls from EHRs (Tohira et al., 2021), indicating a possibility for a holistic care quality assessment of using multiple nursing-sensitive indicators simultaneously, providing reliable and real-time information to support nurse managers in quality evaluation and improvement initiatives. However, reliable evidence on how these tools function in practice warrants still more research , stressing the importance of not only developing and testing these tools, but also implementing and evaluating them in practice.
The results of our study indicate that developing an NLP algorithm for care quality assessment could be used to identify information related to a patient's experience of pain, perceived health, psychosocial health and physical health from EHRs. More research is still needed on how to acquire comprehensive information on adverse events, patient satisfaction or functional health. One question to be resolved is whether the data gap could be narrowed merely by increasing nurse education on documentation, or if additional data sources are needed to provide this information.
The study's limitations are related to the manual data extraction method used in this study, as the expressions were extracted by only one researcher. To increase reliability, the results were regularly reviewed together with another researcher and their evaluation and interpretation discussed. Other limitations include secondary use of EHR's as a data source, as the accuracy and quality of the data are highly dependent on clinical evaluation as well as resources available for documentation.

| CONCLUSIONS
EHRs are a potential data source for assessing care quality, but more research is needed about interpreting and using this information in quality improvement. Issues regarding quality of nursing documentation would benefit from guidelines that promote high-quality nursing documentation and user-driven systems, but also from improving the information technology competence of the nurses and nurse management. Entries made in both structured and free text notes increase the risk of double entries, with free text providing a more holistic view on nursing care quality. Assessment care quality could benefit from the introduction of NLP in free text notes. More research is still needed to develop and test such tools in clinical practice.

| IMPLICATIONS FOR NURSING MANAGEMENT
Nurse managers have an essential role in assessing care quality and quality improvement initiatives. The health system generates massive amounts of data continuously, but the tools to effectively utilize this big data for improved data access and interpretation to better support nursing management are lacking. The development, implementation and evaluation of advanced information systems based on user needs would highly benefit knowledge-based management and support nursing management in quality assessment, by proxy benefiting patient and staff outcomes. They could also provide valuable real time information of the impact of transformational periods, such as the implementation of new technologies, educational interventions and clinical processes. Additionally, using analytical techniques to support nursing tasks is proven to have an impact on the multifaceted and difficult problem of making nursing practise visible (Macieira et al., 2018). In addition to making nursing more visible and providing information to advance the day-to-day operations management, systematic automated quality assessment methods could be beneficial to long-term management as well, presenting information on quality variations over time and revealing trends on a larger scale. This information may be used for benchmarking nursing care quality to monitor performance when compared to others, which could help attract and retain the workforce needed.