The past few years were quite revolutionary for the chatbot world, every organization with basic customer service has tried automating this process. Because most support service requests are routine and repetitive, these requests can be modelled and therefore automated. Of course, a fallback to human support is always necessary to handle the requests that were not modelled in advance.

The goals of chatbots are very clear and many companies have started to offer cloud based chatbot building platforms: Google, IBM and Microsoft are only few of the big names in this field. Most of these enable a developer to build the perfect English speaking chatbot without a lot of effort. In Api.Ai (now called DialogFlow) for example you can connect your web built chatbot to Amazon Alexa, Slack and Telegram Messenger and create small talk by just entering the replies to some common questions.

Hebrew

The next big challenge is tackling support for more unconventional languages. German, French and Spanish are supported by some platforms, but there are languages which are far more complex for a computer, like Hebrew and Arabic (few companies also support Arabic). I’ll focus on Hebrew because it’s my native language and Arabic isn’t my strong side.

Hebrew is so difficult for text analysis mostly because every change in tense, male/female or singular/plural orientation is reflected in the word itself. For example the word “eat” in english is the same both in plural and singular (the exceptions are minor in comparison to Hebrew). The word “eat” in Hebrew is built from the root “א.כ.ל”, you can construct from this root more than 20 words (all verbs) that all mean “eat” in different tenses, passive or for the different genders.

Systems with language orientation will find it difficult to make changes in order to support Hebrew (some of them use a translation service to English in order to analyse the sentence).I believe that a language independent platform could work better with simplified sentences. The reason is that we usually don’t really care about the specifics of the sentence of people purchasing a product, but we do care about understanding the intent of the user’s sentence and the specified entities in his sentence.

Hebrew analysis

It should now be clear that we need a tool which simplifies Hebrew words to their basic form. After a lot of searching I found: MILA, an open source tool built by the Technion — Israel Institute of Technology. The system is using HMM in order to create a predictor that displays the options for word tagging by decreasing probability, which my Hebrew analysis used to get the simplified form of words. The data is exported in XML format and then we revert every word to it’s base form.

A sentence for example is:

“אתם מוכרים כרטיסים למכבי חיפה?” — Original (“do you sell tickets to Maccabi Haifa”)

“את מכר כרטיס מכבי חיפה” — After analysis (“do you sold a ticket Maccabi Haifa”)

As you can tell there are differences, however all words are now in base form and we can recognise the word “מוכרים” (male, plural, present for sell) the same way we do for female, and singular in the past. The result might not be a grammatically correct sentence, but this enables the framework to classify the sentence a lot easier and faster.

As you can already see, for the demo we have 3 intents (query, order and welcome) and this sentence was labeled correctly(query).

Another problem I had was misunderstanding of words meaning by the predictor. For example the word “בירות” in Hebrew can be interpreted as beers (real meaning) but also as Beirut, the capital city of Lebanon. If the chatbot reaches a higher level of complexity in the future, or aspires to, these issues must be solved.. There are some ways to do that, e.g. by using amounts in sentences, but this approach takes more time than I have available.

Oswald

So far I mostly explained why Hebrew is hard for chatbots and how I simplified the language, but the chatbot framework also has to suit the case. It turns out Oswald (Craftworkz own chatbot platform) matches perfectly. Since Oswald does not require any language knowledge the simplified Hebrew will fit great. We will get a lot of different words (verbs with same root), usually they will have the same meaning. Oswald is built using SVM and deducting the amount of different words (most roots in Hebrew can create more than 20 different words) gave us the ability to create a working chatbot with only 40training sentences!

Another huge advantage of Oswald is the way it handles the chatbot’s responses. Most chatbot frameworks allow detection of entities and adding built in scenarios but Oswald allows you also to build a custom response in python that is triggered by rules (using labels, entities and context). In the response you can access the entities and context from the previous sentences. For example you can query a database for prices, alert the user if a product is not available and make the changes in the backend if a product was purchased.

One of the disadvantages is that the framework can’t learn new synonyms by itself but to my surprise it corrected known entities that were misspelled.

The demo is a chatbot built as a store that sells either a product (beverage or food) or football tickets. The intended clients/users of the bot have 3 main intents: welcome, query and purchase.

Final thoughts

Without having any language knowledge in the chatbot itself we created a demo of a fully functioning store chatbot in Hebrew. Oswald works without adjustments for several languages, for the rest this idea can be implemented easily. The stemming is detached from the “mind” of the chatbot and we can therefore simplify it by adding a stemmer to help Oswald understand the intents and the entities without interrupting the backend processes.

The entire chatbot creation with adjustment of the platform to work in Hebrew took only a couple of weeks. Most of the time was invested in adjusting the platform to work in Hebrew (encoding in python 2.7 was the biggest obstacle) and that shows that under good stemming method we can get a great platform to work in Hebrew. I hope that in the near future we’ll see more chatbots with the ability to work under any language and help globalise these technologies.