Isabella: Change to followup

Previously on Dr. Lambda's blog:

In a previous post I presented my newest pet project: Isabella. Isabella is a voice controlled personal assistant, like Siri, Alexa, and others. We have decided to investigate how difficult it is to make such a program. In the last post we talked about the process and problems of moving her to HTTPS.

Now, the continuation...

Followups

In Isabella there is a concept of a followup, meaning that she is able to reply and wait for a response for certain queries. The most important implication of this is that you do not need to start a followup with "Isabella"; you can just say it. This is what we use for playing games, or taking notes.

Something that often annoyed me was saying "Isabella, turn on the lights", which turns on the lights to their last state. This means that in the morning the lights would be dimmed red, because thats their night setting. Therefore I often had to say "Isabella, bright lights", which switches to their bright setting. I did this sequence, or a similar one until I realized that finding the right lighting is an experimental process. I would often switch scenes a few times maybe try dimming, before I found the exact lighting that I wanted. With this in mind it is annoying to have to say "Isabella" again, and again.

Categories

The solution was so simple, because I already had the followup system in place. I simply added a category to every command, then when I execute a command I just push all commands in the same category to the followup database.

Just like the "refresh" command from an earlier post, this was trivial to implement, yet the user experience was tremendous. Last time I argued that you should prioritize changes which benefit the development, so I wont beat that dead horse anymore.

This time I will point out that only by using the product this became apparent to me. This usage over theory was ironically something I learned while taking a theoretical compilers course at university. The course ended with a competition to see who could make the best peephole optimizer, for a byte-code language. To simplify the challenge we were judged only on the number of instructions. Our optimizers were run on several small, but realistic applications, to see who did the best.

Grip by competitive spirit, we thought up hundreds of byte-code patterns which were stupid and could be optimized. Come the judgement, it was revealed that we were the team with the most patterns, one team had only 7 patterns. "They probably slacked off" we thought, and we were confident that with so many patterns we were sure to win. However, that was not what happened. Everybody clapped and cheered as the team with 7 patterns accepted their victory.

It took months of pondering before I fully realized what had happened. The other team had understood when the teachers said "realistic applications". So they had spent their time, not coming up with stupid patterns like my team, but instead coding realistic applications, and looking at the byte-code. Doing this they had spotted 7 weird, but very common patterns.

Then lesson I learned then has stayed with me since. I knew already to "optimize for the common case", yet I did not know the value of investigating what "the common case" is. My advice this time is: remember to walk in your users shoes once in a while. Sometimes it will reveal a tiny change with a huge effect.

"Isabella"

Getting back to the followups. I noticed that with Amazon Echo I would say "Alexa", and then wait, to make sure she was listening. If you have a long, or complicated command it is tedious to repeat it because it didn't hear its name. Again this was a nice touch, which was simple to add to Isabella. I just added an empty command, which would push all commands to the followup database.

Here is a video: