As mentioned above, quality assessment programs are mostly based on clinical databases. However, the reliability of such clinical databases is largely unknown. Hence, a study was designed to audit such a clinical outcome database using a specially trained study nurse [16]. The main purpose of this study was to assess the reliability of residents in tracking complications after surgery. The study was divided into two periods of 3 months duration each. In the first period, a specially trained study nurse, not being involved in the primary care process of the patients, audited all outcome data as recorded by residents in an undisclosed manner. All complications were recorded and graded according to the Clavien–Dindo classification [15] as well as all comorbidities using the Charlson Risk Index. Inconsistencies between recorded and audited data were evaluated.

A total of 305 patients were included during the first period and 447 patients during the second period. The study population of each period was homogenous in terms of types of operation, age, gender, ASA, body mass index, and length of hospitalization. During this period, a total of 206 complications occurred and 80 % of these complications were not recorded. Of grade I complications (without need for further treatment), 94 % were not recorded, of grade II complications (requiring drugs) 54 %, and of grade III complications (requiring surgical, endoscopic, or radiologic intervention) 71 %. Grade IV (requiring intensive care; n = 1) and grade V complications (death of the patient, n = 1) were both not documented in the database. In the second period, 347 complications occurred. Surprisingly, quality recording did not significantly improve. Of the complications, 79 % were still not or not correctly assessed; 89 % of grade I complications were not or not properly recorded, 59 % of grade II complications, 47 % of grade III complications, and one-fourth of the grade IV complications. All grade V complications were recorded. Focusing on clinically relevant complications (grade II and higher), there was a marginal improvement in the second period with 52 % of the complications that were missed compared with 61 % in the first period. However, this difference did not reach statistical significance.

This study has highlighted that data collection by residents is not suitable for quality control [16]. Strikingly, the reliability of the collected data did not improve despite of teaching, and despite of the disclosure of the audit. The reason for this enormous underreporting of complications is multifactorial. First, recording of outcome data is time-consuming and might therefore be disregarded by the residents. The restriction of the weekly working hours may also significantly impede reliable data collection: Restriction of working time leads to many transitions of care causing loss of information. Second, lack of incentives may also explain insufficient data collection by residents; comprehensive data collection is not rewarded and, unreliable data collection has no negative consequences. This lack of motivation also holds for the hospital as a whole, because there is no apparent monetary benefit for the institution to collect such data. In contrast, payers do not reimburse the additional workload for the data collection in most health care systems. And thirdly, surgeons in general are keener on focusing on their core business, the work in the operating room, than on administrative duties, which may unveil data possibly pointing out poor quality.

The assessment of crude morbidity and mortality rates as done in most of the surgical studies do not reflect a surgical performance as the population treated may differ widely in terms of its preoperative risk [17]. Therefore, an appropriate adjustment for the case mix is required for valid comparisons. But risk- adjusted outcome data alone are of little relevance unless there is a consensus on how to report surgical outcome. Additionally, the severity of postoperative complications must be taken into account for quality control since the severity of a complication has been shown to correlate with prolonged hospital stay [18], higher costs [19], and patient dissatisfaction [20].

The risk of a patient to develop postoperative complications may be assessed on an intuitive basis (e.g., expressed by grades as proposed by the American Society of Anesthesiologists [ASA] or with assistance of a Visual Analogue Scale [VAS]) or by objective scoring systems. The value of subjective prediction of postoperative complications has been recognized since the introduction of the ASA grading system. This subjective interpretation of the patients’ health and risk status has gained wide acceptance despite the lack of objective evaluation criteria. The disadvantage of this classification is that the intrinsic risk of the planned surgical intervention is not taken into account since the risk profile of a patient is highly reliant on the procedure. It is quite obvious that we may not expect the same risk to develop postoperative complications for a patient after cholecystectomy and after gastric resection. Woodfield et al. published a risk scoring system based on a VAS [10]. Using such an approach, the planned procedure of a patient is intuitively taken into account, thus correcting the limitation of the ASA scoring system. However, such an approach has different weaknesses. First, intuitive risk assessment relies on experience limiting intuition as a good risk predictor for less experienced surgeons. Secondly, there is the danger of an inflated risk assessment since the higher the estimated risk the better the risk-adjusted outcome will look like [21]. Taking these issues into account, a more objective strategy to predict the risk of a patient is necessary such as risk-scores.

Risk scores in surgery are used to estimate the risk for complications of one individual patient or a selected patient population after surgery in a standardized way. Several risk scores have been defined for surgery over the past years. These scores may be classed into three categories: First, there are general scoring systems assessing the operative risk such as the Physiological and Operative Severity Score for the enUmeration of Mortality and morbidity (POSSUM) [22]. Then, there are scores that cover a specific kind of morbidity such as the Goldman and Detsky indices for cardiac complications [23, 24]. And finally, there are scoring systems being related to a specific condition or disease such as the Acute Physiology And Chronic Health Evaluation II score (APACHE II score) [25] or the Ranson criteria for acute pancreatitis [26]. Despite of these different scoring systems, surgical performance is generally evaluated without such classifications. This circumstance might be explained by the complexity of such scores or by its specificity for a given patient population hampering the broad introduction of such systems in clinical practice.