Maastro is a non-profit social organization with a mission: ensuring the oncological treatment with the greatest chance of cure and/or the fewest possible physical and mental side effects. They do this by deciding on the most suitable treatment together with other oncological specialties and together with the patient.

The hospital's IT support desk handles a substantial workload, addressing up to 1000 inquiries monthly. These tickets encompass a wide array of matters—from reporting a malfunctioning printer on a specific floor to altering letters to patients in the electronic health records. The frontline team is responsible for assigning each ticket to the appropriate department. However, the process of assigning the correct second line team to resolve the ticket is complex and time consuming. Furthermore, it may take some time before the frontline team can check the ticket, which is effectively time lost.

To address this issue, Raccoons developed an AI-driven solution: an automated system designed to classify tickets into various manually selected categories. This solution aimed to streamline the ticket-handling process and reduce the workload on Maastro's frontline team.

Developing the classification system

Phase one: data collection and pilot testing

To build our pilot, we requested data from Maastro covering the past five years. In order to have a high data quality, data that could be irrelevant was cleaned (e.g. data from the COVID period). The filtered data were then categorized and analyzed, with our pilot being trained exclusively on categories with sufficient data.

Next, our model underwent testing to identify inaccuracies. In close collaboration with the hospital, we selected the ultimate categories and conducted the final training of the model. The trained model was then tested on unseen data, delivering impressive results: it not only achieved an accuracy rate of 80%-85% but did so with exceptional speed.

Phase two: production deployment

In the second phase, where we deploy the model in a production environment, it becomes clear why we always opt for a human-in-the-loop approach. This is exemplified by the model's operation: the model assigns a score to each ticket, indicating the probability that the ticket belongs to a specific category. It then produces a top-three list of categories; for example, the model might predict that ticket X has a 90% chance of being in category 1, 20% in category 2, and 18% in category 3.

However, when the model is uncertain of the categories (e.g. a ticket scores more than 50% or less than 10% in all categories), the ticket is escalated to the first line (human in the loop). It then falls to an employee to contact the appropriate department.

Continuous improvement

The final model has been operational at Maastro since May 2024. We plan to retrain it every six months to ensure continuous improvement and optimal performance. This involves evaluating the model's performance across categories and incorporating new categories as needed. We also closely monitor the model to ensure it remains up-to-date and effective.

In the future, Raccoons will maintain a close partnership with Maastro to monitor and enhance the AI-driven ticket classification system. Regular updates and retraining sessions will ensure the model adapts to any new challenges and continues to deliver high accuracy and efficiency, significantly improving Maastro's support desk operations.