Isabella: Spotify and OAuth

Previously on Dr. Lambda's blog:

In a previous post I presented my newest pet project: Isabella. Isabella is a voice controlled personal assistant (VCPA), like Siri, Alexa, and others. We have decided to investigate how difficult it is to make such a program. In the last post we finally deployed her to the cloud.

Now, the continuation...

Voice Controlled Personal Assistants

As Isabella has grown, I have started to grow more and more dependant on her, and indeed more attached to her. In the beginning this was just a fun experiment to see how difficult it was to make something like Alexa. At the same time I was strongly considering buying a "real" VCPA like Amazon Echo or Google Home. This doesn't seem reasonable anymore. The other VCPAs do offer a few features that Isabella doesn't have... yet. To balance it out I have decided to add a feature to Isabella that aren't available in the other assistants.

Spotify

I listen to music quite a lot. Wether I'm working, or cooking, Spotify is usually playing in the background. Again I don't want to get into a discussion about which music streaming service is best by any measure, I just happen to use Spotify. Unfortunately playing music from Spotify is not supported by Amazon Echo – in my country, at the time of writing. Of course this is due to politics and not technology. However I still want it.

Research

Spotify has great documentation for their web api. My first idea was just to get some audio stream, pipe it into an audio-tag and boom, music from Spotify. Unfortunately this turned out to be impossible. You can only retrieve a 30 second clip of a song.

This was quite the roadblock, and it stumped me for several days. I looked over the API again and again, and it just seemed to have methods for searching, and "clicking" the different buttons in the interface. In a way the commands in the API could make a remote control. Then it hit me. A remote control was exactly what I was trying to build. I didn't want to build an entire music streaming platform, I just wanted to control one.

This does have the limitation that Spotify needs to be constantly running in the background. But it does also mean that Isabella can control Spotify playing on other devices like phones or tablets.

OAuth

The first step when working with the Spotify API is to implement their OAuth protocol. Luckily Spotify's OAuth is super easy to implement due to their documentation. Most people know OAuth only from the "login in with facebook" (or google), but it can do much more. I imagine that we will use this same protocol for many APIs that we add in the future, like calendars, email, etc. Therefore I briefly explain the basics of OAuth. In my experience OAuth is difficult to grasp at first sight, so you should not expect to gain a deep understanding from this presentation.

Because repetition is good for understanding, I'll explain it using two metaphors I like. Then I'll explain it with the technical terms, because repetition is good for understanding.

Imaging that we are managers in a ware house. We have access to many areas, some of them are restricted, meaning only we have access to them. Now, for some reason we want someone else to solve one of our tasks. But in order to solve this task they need access to some of the restricted areas, that we have access to. This is the fundamental problem that OAuth solves.

The protocol states that:

• you ask the person who should perform the task.
• the person asks the secretary for a key to the restricted area.
• the secretary calls you to ask if this person is allowed into this particular restricted area.
• you confirm.
• she writes an official form and gives to the person.
• the person takes the form to the janitor.
• the janitor makes the necessary key and gives it to the person.

At this point the person can perform the task. We could imagine the same procedure if you are applying for a job, and the company wants to know your grades, which are usually secret. The protocol states that:

• you send a job application.
• the company asks your school (or university) for your grades.
• the school calls you to ask if this company is allowed to see your grades.
• you confirm.
• she writes sends a link to the school.
• the company opens this link in a browser.
• the browser shows the company your grades.

Finally let's take the concrete example of Isabella and Spotify. The protocol states that:

• you want Isabella to control Spotify, so you send a request to Isabella.
• Isabella redirect this request to Spotify, adding some authentication information, so Spotify knows who "Isabella" is.
• Spotify then presents you with a "this application wants access to these areas".
• you click confirm/continue – ie. sends a request to Spotify.
• Spotify redirects this request to Isabella adding a special token.
• Using this token Isabella sends a request to Spotify.
• Spotify returns an access_token.

Basically every call in Spotify's API requires this access_token.

The first step

The first step in the protocol is to show that you want Isabella to take control of Spotify. The standard way is to have a button, and that was my first approach too. This is because the first time you click it, it takes you away from Isabella, and you are confronted with a screen. From a Human-Computer Interaction view point, this view change is feedback, so it is fitting to have a button. However, any subsequent times you click it, Spotify remembers your consent and just sends you straight back to Isabella without you noticing it. This means that in the subsequent cases we have a button without noticeable feedback – not good.

Common computer science knowledge teaches us that we should optimize for the common case. Imagining that you want Isabella to take control of Spotify often... very often. We only log in "for the first time" once. The common case is clearly the subsequent times, where it does not make sense to have a button. Therefore I decided in the end to remove the button, and add a command to "log in to Spotify".

Now this does cause a problem with discovery; the process by which a user learns about features. It is easy to see a button and try to click it. It is harder if there is no visual ques. However this is a general problem for VCPAs, how do you know what you can do with it? With human interaction we assume that either the receiver know how to answer our query, or we can teach them. Is this an approach we can take with VCPAs? Start with a broad basis of tasks, and the have the users teach them what they need? How should they teach it? I will certainly look deeper into this in a later post.

For now, here is a video:

Isabella: Games

Previously on Dr. Lambda's blog:

In a previous post I presented my newest pet project: Isabella. Isabella is a voice controlled personal assistant, like Siri, Alexa, and others. We have decided to investigate how difficult it is to make such a program. In the last post we explained our reasoning for adding feelings or moods to Isabella.

Now, the continuation...

Games

Now that Isabella have feelings, it makes sense to think about it a bit. What happens if she gets in a bad mood? How can you make her happy again? How do people make each other happy?

Obviously we cant give her a gift, or a hug. Although now that I'm thinking about it, it is not a bad business idea, making an Isabella gift shop, where you can buy virtual gifts for her, to improve her mood. Especially – let's be honest – she is just a fancy, useful tamagotchi.

Another way we humans improve our moods, is by playing games. This we can do with Isabella, and then give a boost to her mood. But which games can she play? The easy answer is: pretty much every game you can play while driving. The first one I thought of was "20 questions", and the easiest I could think of was: guess a number. So let's look at both in turn.

Guess a number

Guess a number is a very simple game, where one player thinks of a number between 0-100, and then the other player has to guess it, using as few guesses as possible.

The cool thing is that this game is exactly what you would expect a computer to like, based on popular preconceptions.

This game was really easy to implement (both ways), using our follow-up system. When she is thinking of the number, you just compare the input with the number and say higher, or lower. When you thinking of the number, she just uses binary search – like any good computer.

Akinator

20 questions is quite a bit more complex. But luckily, like so many times, somebody has already made a brilliant game called Akinator which is exactly what I want. Even more lucky: it has an API. Unfortunately, the API has no documentation. The closest was a couple of projects on github which tried to use it as well.

Unfortunately their code was not quite what I was looking for, so I, instead, made my own, very thin layer on top of the API. Maybe it will be useful for someone else, so here it is:

type StepInformation = {
  question: string,
  answers: {
    answer: string
  }[],
  step: string,
  progression: string,
  questionid: string,
  infogain: string
}
type AnswerResponse = {
  identification: {
    channel: number,
    session: string,
    signature: string
  },
  step_information: StepInformation
}
type CharacterResponse = {
  elements: {
    element: {
      id: string,
      name: string,
      id_base: string,
      proba: string,
      description: string,
      valide_contrainte: string,
      ranking: string,
      minibase_addable: string,
      relative_id: string,
      pseudo: string,
      picture_path: string,
      absolute_picture_path: string
    }
  }[],
  NbObjetsPertinents: string
}

class RawApinator {
  private session: string;
  private signature: string;
  private step = 0;
  constructor() { }
  hello() {
    return new Promise<AnswerResponse>((resolve, reject) => {
      $.ajax({
        url: 'http://api-us3.akinator.com/ws/new_session?partner=1&player=maxipaxi',
        dataType: "jsonp",
        error: reject,
        success: (data: { completion: string, parameters: AnswerResponse }) => {
          this.session = data.parameters.identification.session;
          this.signature = data.parameters.identification.signature;
          this.step = 0;
          resolve(data.parameters);
        }
      });
    });
  }
  sendAnswer(answerId: number) {
    return new Promise<StepInformation>((resolve, reject) => {
      $.ajax({
        url: 'http://api-us3.akinator.com/ws/answer?session=' + this.session 
           + '&signature=' + this.signature + '&step=' + this.step 
           + '&answer=' + answerId,
        dataType: "jsonp",
        error: reject,
        success: (data: { completion: string, parameters: StepInformation }) => {
          this.step++;
          resolve(data.parameters);
        }
      });
    });
  }
  getCharacters() {
    return new Promise<CharacterResponse>((resolve, reject) => {
      $.ajax({
        url: 'http://api-us3.akinator.com/ws/list?session=' + this.session 
           + '&signature=' + this.signature + '&step=' + this.step 
           + '&size=2&max_pic_width=246&max_pic_height=294&pref_photos=OK-FR&mode_question=0',
        dataType: "jsonp",
        error: reject,
        success: (data: { completion: string, parameters: CharacterResponse }) => {
          this.step++;
          resolve(data.parameters);
        }
      });
    });
  }
}

Here is a short video of us playing a game.

Isabella: Now we're talking

Previously on Dr. Lambda's blog:

In a previous post I presented my newest pet project: Isabella. Isabella is a voice controlled personal assistant, like Siri, Alexa, and others. We have decided to investigate how difficult it is to make such a program. In the last post we continued adding more APIs: functionality for free.

Now, the continuation...

More APIs

As mentioned, this stage of development, is super fun, because so much is happening, so fast. Keeping with this we try to add at least two new things at a time.

The small easy addition this time is a quote API. If we thought the joke API was difficult to find, we did not anticipate how difficult it would be to find a good quote API. In fact, implementing features like this takes about 7 minutes, however finding a good API takes about 2 hours.

In the end we gave up the search for a "general, inspirational quote API". And settled for a "programming quote API". Again, we can take advantage of the target audience of Isabella; me, and maybe a few of my friends. We are fine with programming quotes.

At this point we have also added:

• API for finding out where we are
• API for fetching the weather forecast for that location
• API for BBC News API
• API for Urban Dictionary API
• support for spelling words
• and probably some minor features that we forget.

"Conversations"

A different thing completely, is that we eventually want Isabella to have some contextual understanding. Like saying "Play Ed Sheeran", and then following that up with "How old is he", or something like that.

The most basic examples of contextual understanding comes from saying "what" when you don't hear what she says. In this case we want her to repeat the last thing she said. One characteristic of this follow-up query is that you shouldn't have to say "Isabella" first, like normal, as it comes as part of a "conversation".

We introduced the Isabella name to simplify the command matching algorithm. So we could take advantage of the fact that we knew we should try to match one of the commands, and could do our "backwards trick". Now we want to remove this simplification, and that means that she listens to everything that is said after a command. However, this time not everything that is say should match a command. To facilitate this we needed to add a threshold to our matching algorithm, and say that we only register something if it is a sufficiently good match.

First we just add the option to say "what" after everything and she repeats. Then we add "thank you", to which she will reply "you are welcome". Then we took it to the next level.

We wanted to add notes. Here we did not want to use our wildcards, as the note might be quite long. Instead we wanted to use a custom follow-up, where we could capture everything and save it. So that is what we did.

We are probably going to use this "conversation" feature quite a lot for future commands.

Isabella: Growing

Previously on Dr. Lambda's blog:

In a previous post I presented my newest pet project: Isabella. Isabella is a voice controlled personal assistant, like Siri, Alexa, and others. We have decided to investigate how difficult it is to make such a program. In the last post we integrated Isabella with the first external API: Youtube.

Now, the continuation...

The fun part

This next part of a project like this, is the best! Because all the basic functionality, and structure is already in place, so adding new functionality is super easy. In this case adding more API is just a few lines of code, which means that it really feels like something is happening.

In a lot of the projects we work on there are so much code around that making a change is barely noticeable. To add to this, making even a small change, we usually need to spend a lot of extra time testing that our change didn't break any of the other stuff. Tedious, but necessary, work.

At this stage Isabella is still just a nimble little thing, and so there is barely anything to test. We have also enforced an extremely low coupling between different parts, thus there is no way adding something new could affect any existing commands.

So, for the next while we are just going to be integrating more and more APIs.

Jokes, and political correctness

When we tested Alexa, one of the best features was that she could tell jokes, so of course Isabella should be able to do that too. Therefore we set out of a quest to find a good joke API. This was a surprisingly difficult task, as a lot of them are either outdated, or not-free. I did eventually find this one, and the jokes are exactly my style.

In Alexa, and probably the others as well, the developers have another challenge here. Their product has to appeal to a very wide audience. Therefore they have to be careful not to offend anyone with f.ex. Alexa's jokes. Our goal is not to make a wide spread product, so we can just have Isabella say what we want, no politics.

Convenience is king!

Earlier this year I switched my lights to Phillips Hue, so I could control them from my phone. That was so much easier, as a lot of my light switches are in inconvenient places. Therefore I immediately got super used to controlling all the lights from my phone. I have a few lights which I could not switch over, for different reasons, and now I just avoid using these lights because I can't do it from my phone.

This made me re-realize the truth of the statement "Convenience is king!", and so you can imagine how it felt when I integrated Hue into Isabella, so I could control everything by speech. Someone made a great API for it – for Typescript (yes, he included the typings), so thanks to that guy!

Isabella: Becoming useful

Previously on Dr. Lambda's blog:

In a previous post I presented my newest pet project: Isabella. Isabella is a voice controlled personal assistant, like Siri, Alexa, and others. We have decided to investigate how difficult it is to make such a program. In the last post we discussed the importance of shortening the distance between coding and testing.

Now, the continuation...

Hello, World!

So far all of the functionality we in Isabella (apart from the speech recognition), have been coded directly, Functions for getting the time, date, or day, are coded directly into her source code. And that is fine for some stuff; they are useful functions. However, if we really want her to grow fast, the next obvious step was to connect her to the rest of the world, by making calls to APIs.

There are so many APIs out there (APIs), with tons of functionality which can be integrated for free. I do listen to a lot of music, and it would be nice to integrate that somehow.

We do have an advantage over many of the other personal assistants like Siri, or Alexa; we have a very large screen at our disposal. There is a theater saying "if you bring it to the stage, use it!" the point being that if you have something you should aim to utilize it as much as possible. An example could be, if you bring a cane onto the stage, it can also be an umbrella, a pointing stick, a gun, and many other things. Therefore, as we have a large screen let's do something with it.

Youtube, and wildcards

Personally I spend a lot of time on youtube, listening to music, or re-watching classic videos. Therefore the first API I want to integrate is Youtube's. First we need to play Youtube videos. A quick Google search gives us the a link to Youtube API for embedding videos. This is perfect for what we need. The only slight problem was that we need the ID of the video we want to play.

So, we need some way to search. This took a bit more work, and we had to get the first API key for Isabella: Youtube Search API. However, once we figured out the call we wanted, combining the two APIs was very easy.

However, we still needed to extend our basic command format to support wildcards, so we could say "Play ___ from youtube", and the same command would work wether you said "ed sheeran", "adele", or whatever. This was not too difficult, but it is very powerful!

Finally, Isabella can do something that we are actually going to use!

Field Testing

As a test I challenged myself not to use youtube – manually – for an entire day. While working I constantly listen to music, and it worked great. It was actually easier to ask Isabella to lookup videos instead of: switching window (away from work), going to youtube, typing the name of the video I wanted, click the top result, and switch back to what I was doing. Just like I argued in the last post, not having to fully context switch was a noticeable improvement on my workflow.

This function alone, I think, is enough that I'm going to keep using her, for a long time.

Personal assistants; Siri, Alexa, ...

Recently I tried the Amazon Echo for the first time. I had always thought the voice control was annoying and unnecessary. However Alexa stole my heart. I don't want to get into a discussion about which voice control is better, I don't care.

What I do care about however is: with the current state of the field, how difficult is it to make a voice controlled personal assistant? Here I'm thinking of all the libraries and apis that are freely available.

So I have set out to get a sense for this question. Let's follow one of Googles sayings:

• First do it,
• Then do it right,
• Then do it better.

First do it...

To start something new like this we always start by making an example and a tiny prototype. During this spike we specifically try to get as close as possible, to the thing we think to be hardest in the project.

First things first

First we needed to find a decent speech recognition framework for Typescript. We quickly found Annyang, and started playing with it. We want some context awareness in our PA, so we cannot use Annyangs standard command recognition. Instead we used Annyang to parse everything and then build our own algorithm to match the meaning to a command.

We were also fortunate enough that HTML5 has support for text-to-voice already, so we just use that.

Calculations

Having it recognize (and answer to) "hello" took all but two minutes. So we started thinking about "what do we actually want it to do?". I know, this should have been our first question, but we were blinded by the idea of building Jarvis and becoming real life Tony Starks.

Then it hit us. We want it to build Jarvis. Or put simpler, we want it to be able to help us code. We want to be able to talk to it like we do to colleague, and then it should program what we tell it to. Let me just clearify: we don't want to build an AI, just an assistant, who we can tell "do a linear search, then sort the list, and return the median" or something.

First we wanted it to do simple calculations like 2+2. The easiest way to do this was to just eval what was said. It worked like a charm. Only 15 minutes in and we could already access the values of variables, and add numbers.

Function calls

Function calls were more tricky. Especially because we didn't want to say "open parenthesis", or the likes. We have seen the youtube video, and we don't tell my colleague where to put parens or commas after all.

We made a function to take progressively shorter prefixes of the input, camelcase them and test if they were function names. Here is some pseudo code to show the idea:

tryEval(exp: string)
  try { return eval(exp); }
  catch(e) { return undefined; }

matchFunction(words: string[])
  for i : words.length ... 0
    prefix <- words.take(i)
    identifier <- makeId(prefix)
    evalResult <- tryEval(identifier);
    if(typeof evalResult === "function")
      return identifier;

Notice, that even though we use eval this code does not actually call the function, it just finds the name.

Success! We could tell the computer to define variables, evaluate expressions, and even call functions. This was great news for the viability of the project. As the spike ended, we fulfilled our promise (to the extreme programming gods) and erased everything.

Then do it right...

For the next phase of a project like this, we start in the complete opposite end of the spectrum with all the lowest hanging fruits first. If you are very nerdy you could say that we use "shortest arrival time scheduling". We also make sure to make good decisions as this is potentially long lasting code.

Matching meaning

This time we needed a more solid "meaning" algorithm. We do have a great advantage over general AI: we only need to match the input to a command from a very small list. With this in mind we decided to flip the problem on its head, we have a list of results, what is closest to the input. The code went something like this:

foreach command : database
  match <- 0
  foreach cWord : command
    best <- cWord.length
    foreach iWord : input
      if(best > distance(cWord, iWord))
        best <- distance(cWord, iWord)
    match <- match - best;

Intuitively: for each work in each command look for a word in the input that matches, the command with the most matches wins. So we are matching the command against the input, not the other way around.

Of course we also made some normalization code to remove contractions and such, but that is pretty straight forward.

Isabella say "hello"

There is a hidden assumption in the matching algorithm: everything it hears is a command. This is not always the case, therefore we need some way to know that you are talking to it and not just saying. The way we solve that in the real world is with names, so let's use the same solution here. We needed a name that was distinct enough that we wouldn't say normally, and it shouldn't sound like other words.

For now we have settled on "Isabella", as it is a beautiful name, which no one in our social circle have.

Code written in this phase has to be a lot more maintainable, and so we have a tiny database with inputs and answers. It is trivial to add constant things like "what is your name?" "Isabella", but that isn't very fun. Therefore we built in support for "hooks" ($), where we can specify to call a function instead of just saying the string out loud.

I think that's enough for one day, time to go to bed!