The Automatable Activity–Based Approach to Complexity Unit Scoring as a task-specific model approach to monetizing outcomes of pathology artificial intelligence solutions
Article information
Abstract
Background
Cost-containment policies are increasingly affecting decision-making in healthcare. In this context, the need for monetization of digital health interventions has been recently emphasized. Previous studies have attempted to extrapolate cost containment in conjunction with the implementation of digital pathology solutions mostly on the basis of operational cost savings or diagnostic error reduction. However, no study has attempted to link a wider spectrum of potential diagnostic tasks performed by artificial intelligence algorithms to financial figures.
Methods
Herein, we employ a workload measurement tool for the purpose of monetizing particular outcomes associated with the implementation of a pathology artificial intelligence solution. A hundred and thirty-two prostate core biopsy samples were encoded for workload using the Automatable Activity–Based Approach to Complexity Unit Scoring. Subsequently, avoided workload, full-time equivalent gains, and corresponding cost savings were calculated assuming full clinical deployment of a well-developed prostate cancer screening tool.
Results
For a fixed percentage of negative cores and a steady yearly workload of prostate core biopsies, the estimated total avoided workload amounted to 4,291 complexity units per year, with an average avoidance of 16.25 complexity units per ascension number. The calculated full-time equivalent gains were 0.12, whereas projected cost savings were as high as €2,402.34 per year or €0.55 per complexity unit, which in turn would yield an average of €8.93 per ascension number.
Conclusions
The Automatable Activity–Based Approach to Complexity Unit Scoring appears to be a suitable economic evaluation tool for assessing the possible implementation of task-specific artificial intelligence solutions in a given histopathology laboratory or group of laboratories, considering it is a task-specific workload measurement tool per design.
INTRODUCTION
Digital transformation is considered the next evolutionary step of diagnostic histopathology laboratories’ workflow [1]. One of the endpoints of said transformation is the integration of artificial intelligence (AI) pathology solutions in routine clinical practice, first and foremost in the form of companion diagnostic tools able to increase the quality of pathology reporting while simultaneously decreasing (or, at least, not increasing) turnaround time. Of note, as is true for all forms of healthcare interventions nowadays, pathology digitization is subjected to cost-containment policies worldwide. This is particularly accurate for low- and middle-income countries (LMICs), where limited workforce capacity combined with insufficient funding and lack of appropriate infrastructure and technical expertise result in considerable challenges when it comes to initiating meaningful interventions in pathology services [2]. Although more practicable schemes for the implementation of AI-enabled digital pathology in LMICs have been suggested [3], one must keep in mind that long-term strategic planning is required in order to achieve peak resource allocation efficiency and optimal intervention outcomes [4]. In support of the latter statement, the need for monetization of the outcomes of digital health interventions has been pointed out, all in the wider context of the need for economic evaluation of said interventions [5].
Numerous strategies have been devised for the purpose of linking digital transformation in pathology to actual financial figures. Some studies have focused on cost reductions as a direct result of full or partial digitization of pathology pre-analytical, analytical and post-analytical processes, be it in the form of reduced shipping expenses [6], avoided costs from archival and retrieval of cases [7], and decreased overall functional expenses resulting from the consolidation of multiple pathology labs [8]. Others have extrapolated potential savings by forgoing molecular pathology techniques like next-generation sequencing in favor of AI solutions combined with immunohistochemistry or polymerase chain reaction [9], or by omitting unnecessary immunohistochemistry investigations altogether [7]. One private laboratory has presented financial metrics demonstrating return on investment (ROI) following the digitization of diagnostic pathology workflows, tied to service reimbursement [10]. Another model has investigated the potential cost savings across an entire healthcare system as a result of reductions in analytic phase errors (and thus cost reduction from avoided overtreatment or undertreatment of certain groups of oncology patients) within a digitalized histopathology laboratory compared to a non-digitalized setting [8].
Recently, the Digital Pathology Association has introduced an online ROI calculator [11], enabling decision-makers in healthcare institutions to assess costs and outcomes when building a business case on the digitalization of their histopathology lab workflows. Despite these advancements, no study so far has led to the development of a novel tool or the assessment of an existing tool for the sole purpose of monetizing the outcomes of a given pathology AI solution implemented within an already digitized environment.
In this study, we have attempted to use a pre-existing workload assessment system, specifically the Automatable Activity–Based Approach to Complexity Unit Scoring (AABACUS) [12], with the aim of associating the outcomes of a pathology AI solution with meaningful financial figures. AABACUS was selected since it allocates complexity units (CU) to distinct diagnostic tasks for any given biopsy or resection specimen, including activities like surgical margin evaluation, quantitative interpretation of immunohistochemistry assays, and similar procedures, making it a frankly task-specific workload measurement tool thus suitable to evaluate the outcomes of different AI solutions, regardless of the specific task they perform in the diagnostic process. Notably AABACUS was previously incorporated in a study reporting the advantages of a multisite anatomic pathology informatics system, the most important being its enabling of a remote sub-specialty sign-out model of workflow [13]. The authors speculated that their suggested workflow model would increase efficiency and, in turn, reduce turnaround time and costs related to unnecessary testing and case transportation without, however, mentioning specific financial figures. Utilizing this tool, we opted to assess the prostate cancer screening tool developed by Campanella et al. [14]. The latter selection was made due to the software’s distinct impact, namely the reduction in workload for a pathologist by screening out hematoxylin and eosin (H&E) slides of prostate core biopsies not invaded by prostate adenocarcinoma.
We anticipate that the abovementioned assessment as well as the financial data generated from the present study will provide further support for informed decision-making by laboratory managers and healthcare policy-makers alike, specifically regarding cost-effectiveness of AI solutions’ integration in routine histopathology diagnostic procedures.
MATERIALS AND METHODS
Assumptions
For the current study, several assumptions were established. These included operating within a fully digitalized pathology laboratory and the Conformité Européenne (CE) accreditation of the AI solution developed by Campanella et al. [14]. Moreover, in accordance with the above authors’ remark in their original study, the prostate screening software should be able to screen out a substantial portion of prostate core biopsy slides—up to 75% of the total workload that would otherwise be assigned to prostate core biopsy sign-out—without compromising diagnostic sensitivity or specificity [14]. As the present study attempts a simulation of use rather than an actual use of the abovementioned software, AI performance was assumed sufficient in the context of it having received CE-accreditation, per the first assumption of the study. Additionally, it was presumed that the AI tool would be fully integrated into routine clinical practice, eliminating the respective workload entirely rather than serving as a companion diagnostic tool. In other words, slides flagged as negative for malignancy by the AI solution would not have to be reviewed by a consultant pathologist at all as a result of our assumed operational model.
As for the financial framework, the cost basis for the acquisition of a given AI solution under a subscription model was estimated to range between $1,000–1,500 per month, as outlined in the Digital Pathology Association's White Paper [15], or approximately €916–1,373 per month with current conversion rates. Considering net profit computation, solely the cost reductions stemming from workload avoidance were factored in as revenue generation. Consequently, potential cost savings related to analytical phase error reduction or alternative resource allocation resources post the integration of the AI are beyond the scope of the current study.
Data extraction
For data extraction, we carried out a database search through the LIS of Evaggelismos General Hospital of Athens, Greece, identifying the total number of prostate core biopsy cases for one representative semester. Evaggelismos General Hospital is the country’s largest tertiary hospital; its Department of Pathology processes more than 30,000 ascension numbers on a yearly basis, not including those assigned to the separate departments of Haematopathology and Cytopathology.
Ten specimens for whom data retrieval was not possible as well as 16 specimens that featured a final diagnosis other than “prostate adenocarcinoma” or “negative for malignancy” (e.g., ASAP, HPIN, etc.) were excluded from the study. The remaining 132 ascension numbers were reviewed and had a number of workload-related features recorded, including (1) final diagnosis, (2) number of cores per container, (3) number of total produced paraffin blocks, (4) number of original H&E slides, (5) number of sections per slide, (6) number of deeper sections, (7) number of positive cores, (8) number of slides containing at least one positive core, and (9) number of special immunohistochemical stains performed.
Encoding of samples with AABACUS
Encoding samples for workload with AABACUS was performed by employing the guidelines originally established by Cheung et al. [12]. Contrary to other existing workload measurement systems, AABACUS does not rely on a standardized scale of perceived diagnostic difficulty levels in order to quantify actual workload. Instead, it assigns CU to the various diagnostic tasks necessary for the conclusive sign-out of cases, including but not limited to the examination of original, level, and serial H&E sections, the analysis and reporting of immunohistochemistry slides as well as the reporting of synoptic data for specific specimen types (e.g., malignant prostate biopsies). Each activity is considered to have a predefined complexity factor by AABACUS, which in turn is multiplied by each task’s workload activity count to determine the respective CU for that activity. The cumulative CU for a given specimen are aggregated to derive its complexity unit score. An illustrative representation of CU scoring for a prostate core needle biopsy featuring prostate adenocarcinoma is provided in Fig. 1, whereupon the full assignment for almost all possible diagnostic tasks is given, with two notable exceptions: the analysis of intradepartmental consultation slides and the reporting of intradepartmental consults, with a complexity factor of 5 (per 1–5 slides) and 2, respectively. Sign-out of prostate core biopsies follows the current edition of the Cancer Protocol Templates issued by the College of American Pathologists.
Prostate core biopsy with levels (adenocarcinoma with synoptic data) and special stains; an illustrated example of case encoding with Automatable Activity–Based Approach to Complexity Unit Scoring (AABACUS). CU, complexity units.
With regard to working time values, a conversion rate of 1 CU equating to 2 minutes was adopted, in accordance with the methodology of previous investigations [12,16]. As two of the authors have previously employed AABACUS to assess workload in an academic context at Evaggelismos General Hospital [16], no further calibration or modification of CU assignment or CU-to-minutes conversion rate was required.
Application to practice setting
Using internal data as well as data from a previous study of our own [16] a relative benchmark was produced: the modified clinical service full time equivalent, representing an approximation of the work of a full-time pathologist at Evaggelismos General Hospital and equalling ~34.000 CU. In justifying the above number, we utilized a non-weighted average approach based on the total CU generated within a year. This method was chosen for three reasons: firstly, due to the absence of an officially endorsed sub-specialty sign-out model within our department. Secondly, because in instances necessitating intradepartmental consultations for challenging cases, the resulting histopathologic reports are co-signed by a senior pathologist and the original pathologist that the case was assigned to, following which the CU attributed to the former’s contributions are computed independently. And thirdly, because case allocation between consultant pathologists follows a rotational full-day assignment model, thus leading to a (relatively) normalized distribution of cases between consultants.
To estimate the monthly salary rates of pathologists at varying levels of experience, governmental data for fiscal year 2019 were utilized as a basis for analysis. In turn and in association with estimated full-time equivalent (FTE) gains, the above data were used to extrapolate potential cost savings resulting from avoided workload.
Table 1 summarizes the various assumptions, workload parameters and cost basis parameters of the present study.
Assumptions, workload parameters, and cost basis parameters for AI solution monetization
Statistical analysis
The statistical analysis was conducted using Python programming language within a Jupyter Notebook environment ver. 7.1.1, leveraging libraries such as Pandas and Matplotlib for data manipulation and statistical computations.
RESULTS
The total count of H&E slides produced for all samples amounted to 1,318. Out of 1,267 prostate cores examined, 883 (69.7%) were negative for malignancy, whereas 384 (30.3%) exhibited prostate adenocarcinoma invasion, resulting in a total of 502 (38.0%) H&E slides featuring at least one positive core and 816 (62.0%) slides containing solely negative cores. Fig. 2 depicts the ratio of positive to negative slides per ascension number of our sample.
Distribution of positive and negative slides per ascension number.
The total number of slides not containing at least one positive core comprised less than 75% of the total slides, enabling us to subtract the total perceived workload (in CU) attributed to negative slides that a single pathologist would need to manage for sign-out of the respective cases (per the study’s assumptions). Fig. 3A and B illustrates the observed correlations between total slides and actual negative slides per ascension number, as well as between actual positive and actual negative slides per ascension number in the sample. The data reveal that a clinical indication for extensive prostate sampling (as is expected for patients that are clinically suspected to have prostate carcinoma yet whose biopsies fail to yield a positive result) heightens the likelihood of a majority of cores exhibiting no malignancy. Conversely, the more infiltrative a prostate carcinoma, the fewer negative core biopsies are expected in the sampling.
(A) Linear regression lines showing the correlation between total slides and negative slides in prostate core biopsy samples. (B) Linear regression lines showing the correlation between total positive slides and total negative slides in prostate core biopsy samples. Shaded areas indicate the confidence intervals, with wider areas meaning more variability in the data points around the fitted lines.
Fig. 4 demonstrates total CU assigned to our study sample as well as their distribution among the three main diagnostic tasks involved in prostate core biopsy sign-out: specimen reporting and analysis of H&E slides, immunohistochemistry analysis and synoptic data reporting (for positive cores). With an assumed removal of workload attributed exclusively to H&E analysis of negative slides in the entirety of our sample, clinical deployment of the prostate screening tool could result in workload avoidance equalling 4,291 CU on an annual basis (2,145.5 CU per semester, based on the study’s findings), or 16.25 CU per ascension number. Anticipated FTE gains would amount to 0.12 for the histopathology department, based on the study's internal benchmark.
Complexity units (CU) distribution by diagnostic task and CU attributed to negative slides’ hematoxylin and eosin (H&E) analysis. IHC, immunohistochemistry.
To our institutional experience, exceedingly difficult prostate core biopsy cases (that is, requiring intradepartmental consultation) are rare enough to safely allow the assumption of a relatively normal distribution of said cases among consultant pathologists of different salary scales. With that in mind, projected cost savings from workload avoidance would approximate €2,402.34 per year (derived from government data on pathologists' monthly salaries vis-à-vis experience level), or €0.55 per complexity unit, which in turn would equal €8.93 per ascension number. Fig. 5 summarizes the research process leading to the above results in a step-by-step manner. Considering a potential monthly subscription rate of €916–1,373, the ROI would range between –78.4% to –85.42%. In light of the above findings, given the expected yearly caseload of our department, investing in the AI prostate core screening solution in question would not yield adequate returns to offset the associated costs.
Step-by-step depiction of research process and results calculation. AABACUS, Automatable Activity–Based Approach to Complexity Unit Scoring; CU, complexity unit; FTE, full-time equivalent; H&E, hematoxylin and eosin; IHC, immunohistochemistry.
DISCUSSION
In low-resource settings, keeping in mind the need for long-term financial sustainability of any given universal healthcare system, it is imperative that each health intervention demonstrates an overall ROI, particularly so regarding digital health interventions. The integration of AI solutions in pathology workflows is no exception, although the variety of purposes served by these tools in the diagnostic process presents challenges in establishing a universal framework for linking software outcomes to meaningful financial metrics, regardless of the software's designated purpose.
Our study indicates that AABACUS represents a suitable tool for economic evaluation as to the potential implementation of various task-specific AI solutions in pathology, given its inherent design as a task-specific workload measurement system. The financial figures derived by amalgamating CU, internal benchmarks, and the cost basis of workload can guide decision-making processes, investment plans and further benchmarking activities. Fig. 6 illustrates a sample decision-making flowchart template incorporating parameters such as the study's assumptions, the number of prostate cores examined, the proportion of negative cores, and reimbursement factors, all essential in building a business plan ahead of purchasing usage rights for a prostate core biopsy screening tool.
Sample decision-making flowchart template integrating study’s parameters: Should a pathology laboratory implement given artificial intelligence (AI) tool for prostate core screening? CE, Conformité Européenne.
One side-conclusion of our study is that, for a fixed subscription rate or maintenance cost associated with any pathology AI solution, the ROI increases the higher the volume of workload activities it is used for in diagnostic or prognostic context. This in turn delineates the importance of scaling AI solution usage to accommodate as large volumes of diagnostic workload as possible to ultimately achieve economies of scale.
Unavoidably, our model suffers from a number of limitations when applied to real-life routine clinical practice. First and foremost, the current performance of commercially available diagnostic AI algorithms is by no means sufficient to fully substitute the labor of a consultant pathologist. Thus, any AI screening solution would rather serve as a companion diagnostic tool, reducing but not eliminating the respective workload of a given department’s histopathologists. Even in the unlikely event that such a solution would significantly decrease a department’s diagnostic workload, its required maintenance, updating and auditing in regular intervals would necessitate a considerable amount of workload in and by themselves, leading to a lesser than expected reduction in overall workload. Of further note, multiple workflow parameters present with substantial variation in day-to-day clinical practice. Those include, but are not limited to, volume of specimens, case complexity, case distribution and laboratory staffing. As a result, ROI can be appreciated only as a rough approximation, rather than as a solid logistic outcome of investment in a pathology AI solution. Finally, the overall findings of our study may have limited value for laboratories not utilizing CU for workload measurement.
Nevertheless, the growing trend towards workflow digitization and AI integration in pathology suggests that even a rough estimation of ROI from digital health interventions in countries with similar healthcare systems and comparable fiscal capacities is preferable to no estimation at all. Beyond this, our research further outlines a widely acknowledged concept: in a clinical setting, the financial benefits and economic sustainability of AI solutions’ implementation are directly proportional to the volume of diagnostic workload they are involved in.
With the above in mind, various solutions could be proposed in order for institutions with considerable budgetary limitations and/or a low annual number of tests to be able to gain access to diagnostic pathology AI software. One way would be by achieving economies of scale through service consolidation. For example, in the case of Evaggelismos General Hospital of Athens, ROI from acquiring a subscription to software like the one developed by Campanella et al. [14] would be achieved under a model akin to Caltagirone Hospital [17] or the University Health Network Laboratory Medicine Program in Toronto [13], whereupon Evaggelismos General Hospital would buy a single subscription package while serving as a referral center for multiple (between 5 and 7) hospitals with comparable workload volumes. Alternatively, healthcare providers could collaborate with research institutions or industry partners to validate products before commercial release. By offering real-life cases for validation, healthcare institutions could negotiate present and future usage rights of the AI solution in question. Yet another viable option for healthcare institutions would be to invest in developing internal expertise among their staff. By training employees to implement open-source digital pathology and AI solutions independently, institutions could reduce reliance on external technical support and associated costs. This approach would allow healthcare facilities to leverage the benefits of digital pathology freeware while ensuring cost-effectiveness and sustainability in the long term. Finally, building in-house AI solutions would present a highly customizable solution tailored to the unique caseload and case-mix of each healthcare institute, while only carrying the one-time research-and-development costs and any possible software maintenance costs (de facto lower than commercial subscription rates).
In conclusion, further investigation about the monetization of AI solutions in pathology is required, encompassing both theoretical models and empirical analyses drawn from institutions that have already incorporated AI algorithms into routine clinical practice.
Notes
Ethics Statement
Data extraction was performed in full accordance with the General Data Protection Regulation (GDPR—2016/679) of the European Union and the 1964 Helsinki Declaration and its latest amendments. The study protocol received approval by both the Committee of Ethics and Deontology of Evaggelismos General Hospital of Athens (reference number 622, 20-1-2022) and the Committee of Bioethics and Deontology of the School of Medicine, National and Kapodistrian University of Athens (reference number 605, 25-02-2022). Formal written informed consent was not required, as study data were retrieved from archived material.
Availability of Data and Material
The datasets generated or analyzed during the study are available from the corresponding author on reasonable request.
Code Availability
Not applicable.
Author Contributions
Conceptualization: SP. Data curation: SP, MN. Formal analysis: SP, MN. Funding acquisition: SP. Investigation: SP. Methodology: SP. Project administration: SP, GA. Resources: SP. Software: SP, MN. Supervision: SP, GA. Validation: SP. Visualization: SP, MN. Writing—original draft: SP. Writing—review & editing: all authors. Approval of final manuscript: all authors.
Conflicts of Interest
Regarding conflict of interest disclosure, author Stavros Pantelakos was awarded a travel grant for his participation and presentation at the 20th European Congress on Digital Pathology; June 5th–8th, 2024; Vilnius, Lithuania. The grant has been provided by the European Society of Digital and Integrative Pathology (ESDIP) to support the author's attendance and contribution to the scientific program of the event, whereupon preliminary results of the study in question were presented. The other authors have no potential conflicts of interest to disclose.
Funding Statement
No funding to declare.