Functionally testing chatbots (part 2)
Introducing BotSpec!
enter site In part one of this series I outlined the problems with testing Bot Framework chatbots and how there is a real gap in the tooling available right now. Today I can happy announce the, what I would consider the first usable version, of my own chatbot testing framework is now available on NuGet!
see url The goal for BotSpec is to provide a simple and discoverable interface for testing chatbots made the with Bot Framework. Hopefully I’m on the right path with the first version.
Simple use of BotSpec
can you buy celebrex over the counter Here’s some example code that shows off the basic features of BotSpec:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
[TestFixture] public class When_talking_to_the_everything_bot { private Expect _expect; [OneTimeSetUp] public void OneTimeSetUp() { _expect = new Expect("YourDirectLineToken"); } [Test] public void Saying_hello_should_return_Hello() { _expect.SendActivity("hello"); _expect.Activity().TextMatching("Hello"); } [Test] public void Saying_thumbnail_card_should_return_a_thumbnail_card() { _expect.SendActivity("thumbnail card"); _expect.Activity().WithAttachment().OfTypeThumbnailCard().TitleMatching("Some title"); _expect.Activity().WithAttachment().OfTypeThumbnailCard().SubtitleMatching("Some subtitle"); _expect.Activity().WithAttachment().OfTypeThumbnailCard().TextMatching("Some text"); } } |
Code walkthrough
Set Up
In the OneTimeSetUp method we create an instance of Expect. This is the root object for interacting with and testing your chatbot. On creation, Expect will take your token and authenticate with the Bot Framework and start a new conversation.
Hello!
The first test, Saying_hello_should_return_Hello, shows the most basic of bot interactions; sending a message to the bot and expecting a response. Activities you send are most likely to be text-based messages (this includes button presses) but more complex activities that include attachments are also supported.
The next test creates the expectation that we will receive some activity that has text matching the phrase “Hello”. Pretty simple stuff so far, I know.
Drilling down into attachments
The last test shown, Saying_thumbnail_card_should_return_a_thumbnail_card, is an example of an expectation for an attachment that meets the 3 given criteria. The three expect statements will check that any activity received satisfies the condition given for each one. In our case one activity will satisfy all three as the one thumbnail card returned has all of the properties that we are looking for.
BotSpec features
As well as the simple things shown in the code example above there are a few nifty things that BotSpec can do to make testing easier.
Regex comparison and regex group matching
All of the methods that test strings are named {Property}Matching where {Property} is the name of the property is being tested. The word Matching is used because it is not an equality check; these methods take a regex and use that to check whether our property is what we are expecting. (Note: I am considering adding in some options for string checking with string.Equals() being the default and regex being one of the options).
In the majority of cases this will be enough but some times you will need to keep a part of a response from your bot to check later. When this is the case, there is an option for using a group matching regex. It’s a bit more complicated than the stuff we’ve seen so far so I will lead with an example:
|
1 2 3 4 5 6 7 8 9 10 11 |
[Test] public void Customer_bot_should_give_my_customer_id_back() { var myCustomerId = "12345"; _expect.SendActivity("What's my customer id?"); var idMatches = new List<string>(); _expect.Activity().TextMatching("Your customer id is [\d]*", "Your customer id is ([\d]*)", out idMatches); Assert.AreEqual(myCustomerId, idMatches.FirstOrDefault()); } |
We’re still using the TextMatching method but this one takes a few more arguments:
- The first one is the regex that we used before; this will similarly be used to check whether the given property matches the regex supplied.
- The second string looks very similar to the first but has brackets around the
[\d]*part. The brackets form a capture group and whenever we see something that matches the regex, we keep the match inside the brackets for later - All of the matches that we get from our group match will collected and made available as a list of matches in an
outparam
Attachment retrieval and extraction
Attachments can either be sent with the activity or be referenced with a URL. Out of the box, BotSpec will work out for you whether to fetch the attachment via the given URL or to deserialise it from the provided JSON.
This is all using the the default attachment extractor which currently extracts attachments in the following way:
- Works out whether the attachment content is a part of the activity or whether it resides remotely and needs to be retrieved
- Selects all attachments that have the ContentType which matches the specified attachment type
- Retrieves content with the provided URLs using the specified attachment retriever (more on that below)
- Deserialises the content JSON to the specified type
The default attachment extractor can be overridden by setting AttachmentExtractorSettings.AttachmentExtractorType to Custom and assigning a custom IAttachmentExtractor implementation to AttachmentExtractorSettings.CustomAttachmentExtractor.
Similarly the default attachment retriever can be overridden by setting AttachmentRetrieverSettings.AttachmentRetrieverType to Custom and assigning a custom IAttachmentRetriever implementation to AttachmentRetrieverSettings.CustomAttachmentRetriever. The default uses a simple WebClient to download the content as as string (it’s expecting JSON so doesn’t handle image content very well at the moment).
Waiting for a number of messages before asserting
Some times you may be expecting a bot to return a set number of responses for a given interaction and can only be sure that your expectation is met once all of the messages have been received. An example of this could be that you ask your bot for your “top 10 selling products”. As bot messages are not guaranteed to be delivered in order, you may want to wait for 11 messages (1 for your bot to inform the user that it is looking and 1 message for each one of the top 10 products) before checking the content of these messages.
This is as simple as telling BotSpec how many activities you’re expecting before carrying on the assertion chain:
|
1 2 3 4 5 6 7 |
[Test] public void Asking_for_top_10_should_return_my_popular_product() { _expect.SendActivity("what are the top 10 products right now?"); _expect.Activity(11).TextMatching("Let me take a look for you"); _expect.Activity().TextMatching(".*Super awesome thing.*"); } |
Currently (subject to change because I know this should be more flexible), BotSpec will wait for one second and then try again for a total of 10 tries before failing.
Part of the reasoning for this feature was also that the Bot Framework is still very new and my personal experience is that clients dealing with it should be as fault tolerant as possible.
Wrap up
If you like the sound of BotSpec, grab the NuGet and start testing all of your bots!
If you’re interested in how BotSpec works under the hood, I’ll delve into the inner workings in my next post but if you can’t wait until then, have a look at the code on GitHub. If you discover any bugs or have any suggestions feel free to raise an issue on the project or even create a PR and become a contributor.