An interactive service for ORF.at/wahl
Please note: The original Facebook page & bot have been unpublished after the election. To see how it worked watch this demo video showing all features of the ORF.at-Wahlbot.
The ORF.at-Wahlbot (codename BeeBeeAte) was an experimental chatbot for the Austrian presidential elections 2016. As you might know, the Constitutional Court annulled the standoff round from May, requiring a repeat of the election. In September voting envelopes failed and the election day was postponed from October until the first Sunday of December. So a long road to finally elect a new Austrian president. We planned to launch the bot in mid-September, but this plan broke down with the delay. Compared to the New York Times bot or the SRF election bot we implemented the bot without a third-party tool like Chatfuel. By contrast the whole bot was implemented in-house, including the chat engine itself and the content crawler.
Our first approach was to brainstorm what an election-focused bot could provide to a user. For this we analyzed existing chatbots from other domains, like Swelly, CNN, Quartz, Mica, and others. Why would a user choose a chatbot instead of just staying on the existing standard website? From this we filtered out the following core functionalities:
- news = current storys, videos and audios
- projections during the election day
- results from the municipals
For the news part we processed the content from ORF.at/wahl and gave the user a quick way to read information. On the election day the projection and result functionalities were activated. Users could query the current projection of the winner and also query the local result from their hometown via ZIP code or a location name lookup. Both—projections and results—were available as notifications. If a user subscribed to the result of Vienna, the Wahlbot delivered him the result as soon as they were published by the Federal Ministry of the Interior. Projections become more and more stable and precise over the election night and we notified our users about every change. Two days after the actual election the final results including absentee and postal voting were pushed to all subscribers. For this we mapped each municipal to the corresponding district, since postal votes are only included on district level. At the end every user got the final result of his municipal and the postal votes of the corresponding district.
Creating a useful result query engine was one of the main challenges. We thought it will be easy to enter a four digit short ZIP code. Though, ZIP codes do not correspond with municipal borders, in which the actual votes are counted. In step 1 we implemented a mapping between a ZIP code and the municipal numbers. Sounds easy, but it turned out that some ZIP codes lay over eight different municipalities. Step 2 was to support name-based queries. If a user searched for a municipal name, we implemented fuzzy search to allow slight misspellings. Additional attention had the special case of Vienna, which contains 23 districts and is an individual state at the same time.
Technology stack: Eating your own dog food
The conversation part of our bot is about 1.300 lines of code. Quite long for such a simple bot, but this shows you a basic problem. If you open up the path of a conversation, the amount of code explodes. The Wahlbot is based on simple string pattern matching. This kept it simple and forced users to follow a strict conversation path. Any kind of AI (artificial intelligence) or NLP (Natural Language Processing) raises expectations at your users to your bot. If a bot is not really good in processing free input, users will feel disappointment. Most of our code is based on regular expressions and simple string matching. If a user writes „Was ist der Sinn des Lebens?“, our bot detects the words „Sinn“ and „Leben“ and then responds. No magic involved, just simple indexOf() calls and regex matching. Btw. a lot of users ask for „Sinn des Lebens“ or answer with „Thanks!“. Some even express their love with our bot. We learned this in production and adapted our bot to the most common sentences.
A critical path in the whole system is the Facebook Messenger API. If somethings goes wrong without a clear reason—which happened during the development—you cannot debug deeper into the system. The only debugging tool you have is the developer dashboard and Graph API error codes. Not an ideal situation for such time-critical bots like an election bot, but due the closed platforms a common problem. Two days before the election I was locked out from the test instance of the Wahlbot. Messenger returned a generic HTTP 500 error without any information. All other private chats and bot conversations worked. Only the test instance of the Wahlbot was not reachable for me. I reported the bug to Facebook, but they couldn’t reproduce it. Since I could use every other account, I resigned and uses other Facebook profiles for testing. Maybe I triggered some weird spam protection? Who knows.
This is a general problem in the chatbot development. Popular services (WeChat, Line, Telegram, Messenger, Skype, Viber) are all proprietary and add a black box component in your communication channel. In traditional on-premise environments like we use for all of our websites this is not a problem. Control and understanding of infrastructure is an important good. You begin to value open standards and the Web much more if you once felt the troubles with closed messaging systems.
To summarize the chatbot development experience: It’s not easy if your bot gets popular. But communication directly with a user is exciting and totally new if you built traditional web sites for years. Bugs and pitfalls you might avoid:
- Avoid unnecessary web hook events. Facebook can trigger your web hook for message reads and deliveries. You don’t need that and this takes more resources than you might expect.
- Send out responses to your users async. Never block the conversation analyzer with HTTP requests back to Facebook. Some of you might have noticed some capacity problems around 5 PM at the election night. Sorry for that. The Wahlbot learned it the hard way. A possible solution for the future: use a message queue and send out only batched Graph API requests. Thanks to some emergency fixes this problem was solved as good as possible. 200 ms might not be a problem for one active conversation per second, but 10 or 20 parallel conversations may block a too simple bot.
- It’s hard to test a bot under load. You might not find all bottlenecks in local testing and if you use automated Facebook profiles, they might block your little network of automated users as spam / something else. Be ready to fix stuff in production and under pressure.
- A bot is an experiment. You have to be open for a lot of new insights, but it also generates a lot of work.
- A bot is never finished. Your users will always search for new conversation paths and every user asks the same question in a different way. Your code has to evolve over time.
- Users don’t expect that their messages are also stored in the inbox of the corresponding Facebook page of a bot. Nearly everyone who I asked thought they are directly connected to the bot and not that anyone else related to the page could read messages. We introduced a FAQ section to tackle this confusing situation. Bots are new and don’t expect users to understand them as a whole.
- Provide users some guidance. Most hear the first time in their life what a chatbot is. We posted tutorial videos on the Facebook page. The bot has to react on invalid input in a non-confusing way and should provide buttons to the user what to do next. This helps a confused user to get back on the right conversation path. And our newsroom team wrote an introduction and featured it in the days before the election.
- A bot is not an automated system programmed by the a single person. Our development process involved a project manager, a proof-reading step for all static messages, and a lot of discussions with the news staff and other interested people at ORF.
- Last, but not least: Keep the bot funny. Even if the Wahlbot was very polite and restricted, it had some funny eastereggs. Just two examples: If you sent him a sticker, he replied with a short „backstage“ video. And the TV program answer included „Weißblaue Geschichten“, a classic tv series which is used as a fill program in front of the actual election broadcasts.
For 2017 we see a main challenge for chatbots will be their discoverability. News media can take advantage of their existing websites. Links are a very successful way to jump directly into a bot. Facebook and other services have their own shortlink generator. With the ref-parameter in m.me links bots can be integrated better in existing websites. We also provided a short tutorial video for our users to directly scan a Messenger code. It remains to be seen if scan codes will be used on a regular basis.
We hope that you liked the Wahlbot. If yes, cool! If no, why? It would be great to hear from you feedback and comments on this experimental service.
Special thanks to the whole info team at ORF and ORF.at, which contributed great stories and supported the effort. And not to forget the „Lemmings“ chatbot incubator program. We were happy to invite them to a ORF Chatbot hackathon in October and who provided early expert feedback. If you are interested in this topic, check out their website and follow the upcoming Winter 2017 batch!
- ORF.at-Wahlbot – Welcome Page
- ORF Bot-OX Presentation – Editors Lab Hackathon Munich
- Bot-OX Source Code – Editors Lab Hackathon Munich
- orfon/fbmessenger – open-source Messenger client
- orfon/telegram – open-source Telegram Bot API client
- Lemmings Chatbot Incubator