INTRODUCTION

There are currently more than 180 000 patients on haemodialysis (HD) in Europe, receiving treatment at 4000 centres.1,2 Researchers have observed a significant variability in the results from treatment between different centres, related to heterogeneous quality of the care provided to patients.3-6 In addition, HD costs in Spain range between 30 000 and 47 000 Euros per patient per year.7,8 However, despite the great social, health, and economic importance of renal replacement therapy, we currently do not have access to an adequate system for evaluating dialysis centres.

The Quality control Working Group of the Spanish Society of Nephrology (WG) developed this study for the purpose of evaluating HD centres reflecting the values and preferences of stakeholders. The objective is to create a system for weighting the results from each HD centre from a global health perspective, taking into account: clinical results, mortality, health-related quality of life, patient satisfaction, and costs, and thus facilitating the creation of an adequate compound indicator for comparing results between centres acceptable among dialysis stakeholders.

METHOD

This study forms part of the Global Haemodialysis Centre Evaluation Study developed by the WG, whose final objective is the effective evaluation of these centres by analysing the corresponding values produced.

WG first established essential requirements for an “appropriate” HD centre evaluation system; plain, transparent, comprehensive, reproducible, acceptable, benchmarking, useful, and focused on centre improvement.

The WG, composed exclusively of clinicians and based on previous studies,9,10 established a first group of indicators for the results of HD treatment referred to as preselected indicators. Later, a Focus Group (FG) was established, independent of the WG, which was composed of nine individuals who represented the primary actors in the HD process; three patients, three clinicians, and three clinical managers.

The requirements for the different constituents of the FG were: patients must have been on HD for at least three years and must have served as coordinators for renal patient organisations; the clinicians must have been of recognised prestige and ample experience, two of these nephrologists and one an internal medicine specialist; the clinical managers were selected for three different specialties: economic, medical, and health services researcher.

In order for the FG to carry out its work, we established a standardised methodology to be carried out in different phases.11,12 Once the members of the FG accepted participation, and before any previous contact had been made between them, each received an email with a document outlining the purpose of the study and the preselected indicators. Two weeks after receiving the document, all members of the FG met to receive more information regarding the study framework, the objectives of the study, and the preselected indicators. This was followed by a discussion to evaluate the incorporation, elimination, or modification of several of the preselected indicators in order to obtain selected indicators. This in turn was followed by three rounds of individual weighting of the selected indicators, each of which was separated by a session of debate and reflection. As such, two periods of discussion were held, one between the first and second round of individual weighting, and the second between the second and third rounds. The individual weighting process consisted of allotting each member of the FG with 100 points to assign to each of the selected indicators based on their personal values and preferences, taking into account the insights gained in the discussion periods. The debate carried out during the discussion periods was based on contrasting the different weighting results that varied between different individuals and actors. The second round of individual weighting could result in the approval of a metric in the case of unanimity among all members. If the decisions were not unanimous after the second round, a third round of individual weighting continued, whose mean result was determinately approved. The meetings were directed by a member of the WG, recorded, and evaluated a posteriori by two independent clinicians.

We established the requirements that the clinical results should comply with: routine use in HD centres, therapeutic resources capable of modification, and a direct relationship with morbidity/mortality.13-15 The clinical results were weighted by group evaluation, although individual comments on each result from each actor were encouraged and documented. Later, in the WG, we considered the importance of each different indicator that was unanimously approved.

RESULTS

The preselected indicators were established by the WG in November 2007, and are summarised in Table 1. During December 2010, once the members of the FG received information regarding the study characteristics and preselected indicators, the group was formed in agreement with the established requirements. During the FG meeting, in the evaluation phase of the preselected indicators, a new indicator was proposed in the form of the number of days of hospitalization per patient per year. This indicator was suggested by the clinicians, and was unanimously approved for inclusion. The FG carried out the first and second rounds of individual weighting of the selected indicators, and when no unanimous decisions were made in the second round, a third was required as well. The lack of unanimity after the second round of debate was due to several members claiming that mortality was not given sufficient weight. The weightings of the different variables in all three rounds are shown in Table 2. To simplify, the results of the weighting are presented as the mean value for each facet of the focus group: patients, clinicians, and managers.

During the second discussion period, and by initiative of the patients group, the importance of certain clinical indicators was argued over others. From their perspective, the type of vascular access was a particularly relevant variable due to the greater level of incommodity and decreased quality of life associated with using a central venous catheter as compared to an arteriovenous fistula. Patients requested that this consideration be taken into account when weighting clinical results. For their part, clinicians seconded the proposal of considering the type of vascular access as a critical point in obtaining adequate results on HD. The meeting lasted a total of 210 minutes, separated into two periods on the same day.

With successive rounds of weighting, clinical results and annual mortality rates tended to gain more importance as compared to initial values, whereas other results had an inverse tendency. The third weighting session received the unanimous approval of all members of the FG. The weighted variables, presented as a mean of all components of the FG were: clinical results: 38.9; annual mortality: 25.0; patient satisfaction with the HD centre: 12.2; health-related quality of life: 15.6; and economic cost: 8.3 (total: 100).

DISCUSSION

Evidence-based medicine and the evaluation of health technologies are primarily focused on improving the benefits to the patient. However, the indicators used tend to be chosen by researchers who do not take into account the opinions of patients or other actors in the provision of health care.16 This frequently leads to indifference or even rejection of evaluation systems by patients, clinicians, and managers. An incorporation of the opinions of different actors, especially patients, and integration of their points of view should be a fundamental aspect of evaluating the results in any healthcare process. Democratisation of evaluation systems is an essential element for facilitating their acceptance and achieving satisfactory results.

The models used for certifying and accrediting dialysis centres assume compliance with certain requirements and standards without considering the opinions of the affected actors in the healthcare process. Additionally, these models are inadequate for establishing comparisons between different centres. On the other hand, the model proposed by the European Foundation for Quality Management (EFQM) assigns a disproportionate value to process indicators (called agents), to the detriment of variables reflecting clinical results. The process indicators should be considered as one tactic for the achievement of a primary objective, which comes in the form of the final results, and as being proportionate to the value produced.17

We believe that the creation of a compound indicator can facilitate comparisons between different centres (benchmarking) and aid in the effective evaluation of the value produced by each of them. An indicator of these characteristics can maximise the stimulus for improvement in these centres. Such a model can also allow for desegregating the different components of the indicator, thus avoiding a loss of information which accompanies the use of a single indicator. When the information is obtained on an individual basis, we can also adjust for comorbidity between centres. As such, the model would combine the stimulus produced by a single indicator with the availability of disaggregated information, focused on centre improvement. Such a model is comprehensive because it considers all variables relevant to the process of HD. Associated with a previously established automatic classification system for the results at each centre, the model is also simple and capable of providing transparency and reproducibility in the evaluation of HD centres.

The methodology used by the FG was chosen because it provides a useful tool for obtaining and processing information based on complex and qualitative sources, with ample detail and diverse backgrounds. Our methodology necessitated a capacity for integrating different points of view, beliefs, and experiences, which is a distinct possibility when using a FG. In order to achieve the objective of weighting the different variables, the primary focus of the study, we did not consider it necessary to perform a quantitative analysis of the results; however, we did use a standardised methodology.

As regards the evolution of the individual weighting process, clinical results and annual mortality progressively gained more importance in the second and third weighting sessions as compared to the first, whereas patient satisfaction and health-related quality of life lost importance. This progression could suggest a tendency to assign a greater value to objective sources of evaluation as the comprehension of these variables increases. One notable aspect that upholds this interpretation is the fact that the initial weightings, which came before the first discussion and debate, were not radically different between the different actors, but rather quite comparable, and with a maximum difference between groups of 10 points out of 100. In addition, the results of the second and third weighting sessions were similar between the three groups of actors, which demonstrates the level of consensus reached through discussion and agreement of opinion following debate. This progression suggests that the points of view held by different actors are not so disparate once all of their members deeply understand the significance of the results and can reflect upon them.

However, the study also had various limitations. Firstly, the representativeness of each group may be insufficient for establishing the actors’ opinions. However, the differences between the final intra-actor weights (patients, clinicians, and managers) were minimal (the results are not shown for simplification). The difference reached 12 points in one variable, 10 in three, and less than 10 in the rest, which indicates substantial internal validity. On the other hand, since all members of the FG came from a single cultural context, we cannot extrapolate the results to other areas with different sets of values.

To conclude, the weighting structure of the HD results appears appropriate for evaluating HD centres, since it takes into account the relevant results from a comprehensive perspective and assigns weight in accordance with the values and preferences of the actors involved in the process. An evaluation of these characteristics may result more acceptable for patients, clinicians, and managers, as well as providing greater legitimacy for effective implementation and dissemination, while also contributing to the evaluation of the value produced in each centre and improving their results.

Conflicts of interest

The authors state that they have no potential conflicts of interest related to the contents of this article.

Table 1. Indicators preselected by the Quality Management Working Group

Table 2. Weighting of the selected indicators, performed by patients, clinicians, and clinical managers in the first, second, and third rounds

Table expressing the homogeneity and contingency among different intra-actor weightings; the third also includes the results for all three actors together