Enable's ticketing system, Cello is mainly used by clients for support and UAT, however it is flexible enough to be used by Enable employees for many other tasks. These tasks include logging product recommendations to help improve our applications, tracking research tasks, carrying out estimates, and a whole lot more. As the amount of information held in Cello grows, being able to search existing tickets and organize new ones efficiently becomes more important. Along with some more conventional improvements, we’re adding machine learning capabilities to Cello to make it even easier to use. This has been a good trial of Microsoft’s new ML.NET library, which is a perfect fit for our tech stack.

Automatic category selection

For now, we are trialling machine learning in our internal recommendations backlog. Recommendation tickets are categorized into one of a dozen areas such as features, performance, UX, etc. We wanted to make logging a recommendation easier by automating the choice of category. Automating tasks like this is a great candidate for machine learning. The basic process involves running through the following steps:

1. Take data from a given source and distil it down to comparatively few features. In this case, we used ML.NET to transform the ticket text into recurring words and phrases. As the same individual words can be used in different contexts, the model needs to be advanced enough to take this into account.
2. Pick a model that might find patterns in the featurized data. In this case, we used one of ML.NET’s classification algorithms.
3. Train the model on your data. We used manually labelled tickets to teach the ML model what text corresponds to which category.
4. Use the trained model to make predictions! We withheld some manually categorized tickets from the training data. We then got the ML model to predict the ticket category from their text, and then compared it to the real label to see if it was correct.
5. Now that we can quantify how well the model performs, we can go back to step one with a tweaked feature set and model and repeat until the predictions are accurate enough.
6. To help visualize this, we extracted some of the individual words the ML.NET pipeline associated to the ‘UX’ and ‘Feature’ categories. The larger the word, the more common they were:

Some words we use in recommendations are common to both categories. As the pipeline considers how the words are used in phrases, it doesn’t have to remove shared words to avoid getting confused – this would remove valuable data. All in all, the machine learning model considered well over 10,000 words and phrases and analysed how often they occur in each ticket.

The results

Usually, building and testing a model involves a lengthy trial-and-improvement process. Fortunately, Microsoft’ ML.NET includes tooling reduces the time needed to build a model by automating much of the training process, and it offers lots of different models that developers can configure. This allowed us to test multiple different machine learning algorithms on the same data to determine which produced the most accurate results. We eventually arrived at a model that could correctly predict a ticket’s category 80% of the time. That’s a lot better than the 8% chance it’d have if it chose at random!

Going live

Once we were happy, we deployed the model to suggest categories to recommendation writers on Cello before they save their new ticket. They’re free to choose a different category instead if the model didn’t predict the right one. Since we wanted the model to keep learning from new data, we rolled the model training code into an overnight process. This allows us to re-train the model using new tickets, ensuring its predictions continue to improve.

Coming Soon After this successful proof of concept, we are now researching what other machine learning features can be added to our applications. As the team gets more comfortable with machine learning and as Microsoft adds more features to ML.NET, we’ll be able to roll out new features.