Impacts of Covid-19 on HCAHPS

In this new challenge for Maven Healthcare, I worked as a data analyst for the American Hospital Association (AHA), a national organization that represents hospitals and their patients, and serves as a source of information on health issues and trends.

As part of my role, I have been asked to review the results of the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) from the last 9 years. One of the three goals of the HCAHPS program, which aims to provide standardized research tools to measure patient perceptions of hospital care, is "to create incentives for hospitals to improve the quality of care."

My main task was to assess whether the HCAHPS program was successful in achieving this goal by answering the following questions:

Have hospitals' HCAHPS scores improved over the past 9 years?

Are there any specific areas where hospitals have made more progress than others?

Are there any major areas of opportunity remaining?

What recommendations can you make to hospitals to help them further improve the patient experience?

In order to answer the questions proposed for analysis of the results of the HCAHPS survey, a methodology consisting of 5 steps was used: Business understanding; Data understanding; Data preparation; Analysis and Interpretation of the results.

1. Business understanding

The American Hospital Association developed the HCAHPS with the purpose of measuring the quality of services and the satisfaction of patients in American hospitals that support public health insurance, the Medicare and Medicaid.

In addition, the main focus of the research is to allow the comparability between hospitals that use HCAHPS in order to improve the quality of services provided.

The HCAHPS is organized into 3 categories: composite measures, individual assessment and general items. It evaluates aspects such as:

COMMUNICATION WITH NURSES

COMMUNICATION WITH DOCTORS

RESPONSIVENESS OF HOSPITAL STAFF

COMMUNICATION ABOUT MEDICINES

DISCHARGE INFORMATION

CARE TRANSITIONS

CLEANLINESS OF HOSPITAL ENVIRONMENT

QUIETNESS OF HOSPITAL

OVERALL HOSPITAL RATING

WILLINGNESS TO RECOMMEND THE HOSPITAL

Each of the categories is assessed by at least one of up to three objective multiple-choice questions. Usually three options are available for each question, but some have only two options. The options are organized in the following order: Bottom-box, corresponding to the least positive response; Middle-Box, corresponding to neutral answer and Top-box, corresponds to the most positive answer to the questioning. For questions with only two options, the Middle-Box is abstracted.

The HCAHPS seeks to assess whether the points identified in the patient's journey are satisfactory or not and shows an overview of the degree of satisfaction of those attended by the hospital.

Despite being a research that can generate powerful insights, it must be sent in the right context and knowing how to apply the HCAHPS will directly reflect on the results obtained.

The HCAHPS research should only be applied in general hospitals that have a complete hospitalization process, which means that it should not be used in clinics, for example. This characteristic of how to apply the HCAHPS happens because the research is formulated based on aspects that are only found in hospitals that provide full hospitalization.

Another important point is that the HCAHPS should not be used to compare hospital sectors, only between institutions.

Placing the patient as the focus of care is paramount for the success of any health institution, which is why it is so important to measure the experience of the people assisted by the hospital.

By carrying out this type of monitoring, it is possible to identify friction points and improvements throughout the patient's journey, positive issues that must be maintained and new possibilities for growth and investment in the institution.

Knowing how to apply HCAHPS is the first step in understanding how this strategy works, as giving voice to the patient is the right path for hospitals that want to strengthen their name in the market and ensure their financial health.

The correct way to apply the HCAHPS is to send it to a portion of patients who were discharged or moved to another institution, it is recommended that this must be done within a period between 48 hours and 6 weeks after discharge.

Currently, the ideal is for the survey to be sent through an online channel such as e-mail, but originally the survey was developed to be applied via physical forms or over the phone (IVR and/or customer service).

2. Data understanding

For the analysis of the results, a mass of data was released about the collections carried out between 01/10/2013 and 30/09/2022. The reports are normally published in July of each year, presenting results referring to the collection carried out between October and September of previous years. For example, the report published in July 2019 is based on data collected between October 2017 and September 2018.

This mass of data was released in CSV file format, consisting of eight data files and a file referring to the data dictionary. The complete description of the other data files and their attributes was found in the data dictionary.

Regarding the quality of the data, three “traps” were found. The first in the “responses.csv” file in the “State” attribute where 56 different values were found. Since there are only 51 States, correctly listed in the “states.csv” file, we had five values for the “State” attribute in the “responses.csv” file without correspondence in the “states.csv” file, which are: AS; GU; MP; PR; VI. Since there are no States represented by these five acronyms, presuming the occurrence of errors in data collection, we had as a result in the graphs 17,105 “Completed Survey”, 65 Hospitals and 42 Response Rate (%) results without the correct identification of the corresponding State .

The second trap was also found in the “responses.csv” file in the “Response Rate (%)” attribute where 10,785 lines had a value that could not be converted to an integer. For example on the fourth line, the value “Between 100 and 299”.

The last, but not least, trap was found in the “questions.csv” file in the “Bottom-box Answer” and “Middle-box Answer” attributes of the first twelve lines of the file. Considering that the Bottom-box value should be the least positive answer and the Middle-box value the neutral answer, apparently the values found in the “questions.csv” file: “Usually” for Bottom-box and “Sometimes or never” for Middle-Box they appeared to be inverted. When checking the original form, the inversion of values was proven for some unknown reason. This trap was addressed in step three.

Analyzing the two result files: “national_results.csv” and “state_results.csv” it was immediately identified that the results of the first file could be obtained through the second. Therefore, the “national_results.csv” file was disregarded for making the data model in step 3.

The other files, “measures_original.csv”, “reports.csv” and “states.csv” became self-explanatory from the description in the data dictionary and did not present any anomaly.

In this step, we will not address any issue related to the Primary Key, Cardinality and Granularity, thus avoiding redundancies of information already contained in the data dictionary.

3. Data preparation

The files released as a data source were stored in a Sharepoint folder and extracted in Power Query using the Csv.Document and Web.Contents functions.

In the table obtained from the “measures.csv” file, the columns "Measure ID" were renamed to "MEASURE_ID", "Measure" to "MEASURE_DESC" and "Type" to "MEASURE_TYPE". Next, a column was added for sorting and another with the name of the measures in capital letters.

From the “questions.csv” file, the table of the same name was obtained, where the columns were renamed as follows: "Question Num" to "QUESTION_NUM", "Measure ID" to "MEASURE_ID", "Question" to "QUESTION_DESC" . Afterwards, a sorting column was added. Also from the “questions.csv” file, after a sequence of transformations, the “questions_answer” table was obtained.

In the table obtained from the “reports.csv” file, the columns "Release Period" were renamed to "RELEASE_PERIOD", "Start Date" to "RELEASE_START_DATE", "End Date" to "RELEASE_END_DATE". Next, a RELEASE_DATE column was added to compose the relationship with the calendar dimension.

The most refined transformations took place in the “states_results.csv” file. Initially, the columns "State" were renamed to "STATE_ID", "Measure ID" to "MEASURE_ID", "Release Period" to "RELEASE_PERIOD". The columns “Bottom-box Percentage”, “Middle-box Percentage” and “Top-box Percentage” were transformed into lines, obtaining the columns “BOX_DESC” and “BOX_PERCENTAGE”. It was later merged with the “questions_answer” table. Finally, the “RELEASE_DATE” column was added to compose the relationship with the calendar dimension.

From the “states.csv” file, the table of the same name was obtained, where the columns "State" were renamed to "STATE_ID", "State Name" to "STATE_NAME", "Region" to "STATE_REGION". Next, a column was added with the extraction of the URL referring to the SVG file of the flag of each State from Wikipedia. The “STATE_NAME_FULL” column has also been added for use in the map.

Once the necessary transformations in the data sources were completed, a data model as close as possible to the star schema model was built from the tables obtained. This model was composed of two fact tables: “f_RESPONSES” and “f_STATES_RESULTS”. And five dimension tables: “d_CALENDARIO”, “d_REPORTS”, “d_MEASURES”, “d_QUESTIONS” and “d_STATES”.

Once the extraction, transformation, loading and modeling are finished, we begin the process of constructing the measurements using the DAX language. With the first DAX measures created, it was possible to measure the performance of the data model, which was satisfactory for analysis.

4. ANALYSIS

In this phase, the construction of the visual layer of the report was started. We initially opted for a multi-page layout: a cover page for the project; a page with an overview of the results; a page with the result of the applied analysis.

In the construction of wallpapers and background Figma was used. The icons were obtained from the Flaticon website and the images directly from Internet searches.

The cover page basically consisted of a visual layer seeking a visual identity aligned with the project. On the second overview page, a Z-Layout was used, with summary cards at the top and five graphics distributed just below the page.

For the first chart on the overview page, a donut chart design was chosen, where the measurement categories were arranged. The donut design was chosen due to the reduced number of measurement categories (three). The main purpose of this chart is to create context for understanding the nature of the data. In it, it is possible to have an initial view of the set of existing measures, as well as how they are grouped and also how they are analyzed. This visual has a tooltip where you can check how many questions and answer options compose each measure.

In the second graph, a visual dispersion was used, with the measurements on the X axis and the average of the [BOX_PERCENTAGE] attribute of the “f_STATE_RESULTS” fact table on the Y axis. As a caption, an emoticon related to each of the values of the [BOX_DESC] attribute of the same table was used. For the size of the marker, the average attribute [BOX_PERCENTAGE] was used again. For the vertical segmentation obtained, the green color and circle shape were used for the Top-box values (more positive responses), for the Middle-box values (intermediate responses) the diamond shape and yellow color were used and for the Botton-box values the square shape in reddish color.

Also in the second graph, a Tooltip was released, which is revealed when hovering the mouse over each of the markers. In this Tooltip we have a line graph with the evolution over the months/year of publication of the reports the score values obtained for each response option of the referred measure. In the subtitle it is possible to visualize the variation within the period for the answer option. Just to the right, in a thread chart, is the average response rate. Below, four horizontal bar graphs show the three states with the greatest growth in the measure response option, the three regions with the greatest growth in the measure response option, the three states with the greatest decline in the measure response option, and the three regions with a greater drop in the response option of the measure. In these four charts, for the Top-box (more positive) response option, growth was considered an improvement and a decrease a worsening. On the other hand, for the Botton-box answer option (less positive) the growth was considered a worsening and the decrease an improvement. Finally, questions applied to the selected measure were listed.

The third chart distributes the number of complete surveys answered by regions in a visual form of horizontal bars.

The fourth visual shows a map of the states where the number of hospitals that participated in the survey is defined by the intensity of the blue color. In the subtitle it is possible to identify the state with the largest number of participating hospitals and the state with the smallest number of participating hospitals. In this, the tooltip presents the identification of states through the image of the flag and its name, followed by the number of respondents (obtained from the number of complete responses and percentage of responses), and also followed by the number of complete responses, as well as response percentage and number of hospitals. Just below, a table listing the measures and the variation of each of the respective response options between the smallest and largest year of the selected period. The last column of this table evaluates the performance of the measure, based on the variation in the response rate of the set of options. For example: for the total period, between 2015 and 2023, the RESPONSIVENESS OF HOSPITAL STAFF measure, in the State of Montana, had a decrease of -1.00% in the Botton-box response option (less positive), a decrease of -4.00% in the Middle-box answer option (intermediate) and a +5.00 growth in the Top-box answer option (more positive). Therefore, considering a decrease of -1.00% in the least positive response rate, a decrease of -4.00% in the intermediate response rate and an increase of +5.00% in the most positive response rate, it can be concluded that this measure improved its score in the period. In the upper right part of this tooltip, a thread chart was added with the number of measurements that improved and the number of measurements that worsened.

In the last graph, with lines, we have the evolutionary percentage of response over the months/year of publication of the reports.

5. INTERPRETATION OF THE RESULTS

Have hospitals' HCAHPS scores improved over the last 9 years?

The direct answer to the first question is NO. At the national level none of the measures achieved an improvement in the last 9 years. For a more positive response, from the top-box, the drop between the reports published in July/2015 and July/2023 was from -0.08 for the QUIETNESS OF HOSPITAL measure to -3.43 for the COMMUNICATION ABOUT MEDICINES measure. For even more positive response, for all measures, the last report's response rate is lower than the average for all years. The least positive response, from the bottom-box, increased from +0.51 for the COMMUNICATION WITH NURSES measure to +2.39 for the COMMUNICATION ABOUT MEDICINES measure.

However, this generalized deterioration in the performance of the measures does not result from a recurrent drop in the evaluation of the measures over the 9 years. Instead, the evaluation of the measures showed a recurring improvement until the report published in July/2021. As of the report published in July/2022, an abrupt drop in top-box ratings begins. We can see that the data collection period (Oct/2020-Sep/2021) of the report published in July/2022 coincides with the worst moment of the Covid-19 pandemic. We can assume that this atypical event severely affected the evaluation of the measures.

On the other hand, in the state results, 21 states had worsening in all measures. 8 States had worsening in 9 out of 10 measures, and only in 3 of these a measure improved, in the other 6 states a measure was stagnant. In 43 States more than 50% of the measures got worse.

Even in this negative scenario, there were some highlights. Alaska, Montana, and North Dakota improved on 9 out of 10 measures, thus demonstrating an early recovery in positive HCAHPS ratings. Geographically, at the top of the border between the Mountain and West North Central regions are most of the top performing states.

Are there specific areas where hospitals have made more progress than others?

As all measures deteriorated in the last published report, to identify those where the greatest progress has been made, we chose to look at the highest performance and the average of the top-box results over the 9 years.

Considering these parameters, the DISCHARGE INFORMATION measure achieved the highest top-box performance (87.08%) in the report published in July/2021, averaging 86.76% over the 9 years.

Although HCAHPS does not have the main objective of comparing sectors, due to the very nature of the questions applied, different amounts and types of answers (yes or no; scales from 1 to 10; agreement), the second best performance would be with the COMMUNICATION WITH DOCTORS measure with a maximum top-box performance of 81.86% in July/2021 and an average of 81.38% in the 9 years.

Are there any major areas of opportunity remaining?

In the same sense as the previous item, the CARE TRASITION measure had the lowest maximum performance of all measures of 53.84% in the report published in July/2021, thus becoming the most suitable area for improvements.

What recommendations can you make to hospitals to help them further improve the patient experience?

HCAHPS is a tool that can be used to help organizations improve the patient experience as well as capture the effect related to clinical quality. Such content displays how HCAHPS data should be used in context alongside other organizational performance information.

Hospital leaders must use the survey as a tool to strengthen relationships with patients and improve service. However, as with any tool, data must be applied wisely. Research shouldn’t be the only way for organizations to gain information about the patient experience. To get the most value out of the data, it should be considered in conjunction with other relevant organizational metrics. Not just the patient's experience, but also the staff's experience. To establish relevance for clinicians, survey data must be an integral part of hospital quality and safety improvement efforts, not just a measure of customer service.

Understanding HCAHPS data requires assuming more than an organization's current performance. In publicly released HCAHPS metrics, there is a wealth of information. Leaders need to understand them and use them to guide improvement efforts. In addition to current performance, special attention should be paid to trends, benchmarking and unit analysis. In addition, it is also necessary to observe the representation of the lower box, the least positive response category in the HCAHPS survey, it must be verified whether the organization's percentage is higher than the national score; this will help set priorities.

The next step is to identify priorities for improvement. An improvement-focused approach will help coordinate efforts toward success. Priorities should be integrated and aligned with other organizational priorities and developed in conjunction with team inputs.

After identifying priority areas for improving the patient experience, organizations must determine performance-enhancing interventions. This is possible by reviewing successful and unsuccessful processes and common characteristics of hospitals that have already improved HCAHPS performance. Simply deciding to adopt a practice is not enough. Close attention must be paid to how to do this consistently and effectively in each organization. Also not forgetting to actively explore the knowledge of your own hospital; internal review is essential.

Effective use of data is not an intuitive skill for everyone, so providing appropriate tools and training is essential. Understanding what motivates individual team members is critical to success. It is important to communicate both the objective and the strategic vision behind the objective. Each team member must know what is expected of them. Involving the team is another step to be taken in applying the results.

Each improvement cycle must include continuous measurement and monitoring for success. Team metrics can provide valuable insight into which aspects of patient experience improvement initiatives are working and which aspects should be refined or abandoned. Organizations are famous for planning and implementing performance improvements and forgetting to follow through after the initial implementation. And in parallel to this idea, it is worth saying: sometimes knowing what to stop doing is as important as knowing what to implement.

Not all intervention attempts will succeed... It is important for leaders to recognize success as well as effort. Teams that have invested time and energy into actions should be recognized for the work they have done. Even if the goal has not yet been achieved, simple progress should be celebrated.

As previously mentioned here, the states of Alaska, Montana and North Dakota, which improved in 9 out of 10 measures, are examples of resumption of growth that deserve recognition.