Skip to main content

Giving Facebook Chatbot intelligence with RiveScript, NodeJS server running in AWS Lambda serverless architecture


This is a long Blog describes how I build my Chatbot (named I’m Simon Chatbot) to demonstrate below features:
  • The Chatbot talks with human via Facebook Messenger
  • Chatbot server app running NodeJS (with ExpressJS framework)
  • Giving an A.I. brain to the Chatbot with RiveScript
  • The Chatbot server App hosts in AWS Lambda (comply with serverless architecture)
  • User state management persistent via AWS DynamoDB
  • Use Serverless framework for easy App deployment

1. Create a Facebook Page

I want my Chatbot could be interfaced with my Facebook friends via Facebook Messenger. First of all, I need to create a Facebook page required by Messenger Platform. I make a funny banner picture and avatar to the Facebook Page as below:

2. Create an Facebook App and configure the Messenger Platform

To accomplish chatting in Messenger, Facebook requires a Facebook App to define the configuration of the Chatbot web service. I create the App from the Facebook Developer website. As Facebook developer documentation mentioned, I add “Messenger” product as below screenshot:

Obtain the “Page Access Token” and record it:

Select the messenger events which will be handled:

Now the setup is done. I can start programming the Chatbot…

3. Build the Chatbot in NodeJS

The Chatbot server App is developed in NodeJS along with ExpressJS. Those versions I’m using are:
$ node -v && express --version
v6.10.3
4.15.0
Below is the initial Node-express server for local development (it depends on external modules. The full source-codes in this Github repo:

You can download & install the full source codes as follows:
$ git clone https://github.com/simonho288/rivescript-chatbot-lambda
$ cd rivescript-chatbot-lambda
$ npm install
$ touch aws.config.json
IMPORTANT: You’ll need to edit the aws.config.json file and put your AWS ACCESS KEY ID & ACCESS KEY SECRET in the JSON properties, like:
{
  "accessKeyId": "<YOUR ACCESS_KEY_ID>",
  "secretAccessKey": "<YOUR ACCESS_KEY_SECRET>",
  "region": "<YOUR REGION>"
}
To obtain AWS Key ID & Key Secret, you can create a new IAM user. Please refer to this instruction.

4. Setup the Facebook App Webhook

Webhook is the entry point of my Chatbot server App when Messenger receives messages from users. To setup the Facebook App webhook. Follow the below steps:

1. Run the local express server in the project root directory:
$ node server
Server listening on port 3000
2. Make a secure HTTP tunnel (HTTPS) between the local server and the external Internet to allow Messenger RESTful calls. I’m using amazing Ngrok to setup the tunnel. You can download it for free & run it:
$ ngrok http 3000

You’ll see the below similar screen:

Login to Facebook developer and create a new App. After the App has been created, go to webhook section:

If anything setup properly, it should no error and the subscription dialog disappears. That’s mean verification success.

5. Install RiveScript with brain files

RiveScript is a simple scripting language for giving intelligence to software for handling Human natural languages. After I used it in several projects, I feel it is not only simple but also powerful. Especially it supports asynchronous programming model that perfectly matches with NodeJS’s asynchronize non-blocking model. NodeJS non-blocking programming model leads up PHP at least two times faster, 20% faster than Java and 10% faster than ASP.Net

Another powerful feature of RiveScript has the ability call sub-routines are programming in Javascript Promise as async object macros. I will demo this feature later in this blog for querying real-time stock prices.

To install RiveScript to our server App (locate in project root):
$ npm install -S rivescript
I create a module rivescript.js to handle RiveScript:

Furthermore, to leverage the benefits of RiveScript NLP, I program several RiveScript brain files in /rsBrainFiles directory that are to make the Chatbot more human-like:

6. First talk to my Chatbot

I can’t wait to try chatting with my Chatbot. I open the Facebook Page and click the “send message button”:

Facebook Messenger provides a full-screen view which a better experience, I open it as below screen:

Type below messages to test the Chatbot:

The above answers are programmed in one of brain files /rsBrainFiles/myself.rive. Full scripts are:

To demonstrate the RiveScript asynchronous operation, I add a command for real-time stock price querying. It is sub-routine getStockPrice in /rsBrainFiles/stockprice.rive file:
! version = 2.0
+ [*] stock price [of] *
- stock price of <star1> is <call>getStockPrice <star1></call>
+ * stock price
- <call>getStockPrice <star1></call>
The sub-routine getStockPrice is programmed in Javascript rivescript.js:

There have two processes inside the sub-routine getStockPrice when receiving stock query message: 1. Lookup the global stock symbol by company name, 2. Get the current stock price by stock symbol. All rely on external Yahoo Finance APIs. Which is programmed in /libs/external.js module:

Consequently, the Chatbot performs below stock messages and answers:

You can also be chatting with my Chatbot. Go to this Messenger Page and select “send message test button”, or search “Im Simon Chatbot” from Messenger mobile App. Feel free to chat with it. It does NOT store either your contact info or the messages you entered. (The privacy statement is declared to Facebook here)

7. AWS Lambda preparation and Install Serverless Framework for deployment

After the Chatbot test successfully, I need a server to host the NodeJS server app. I have three options:
  1. Server infrastructure (IaaS) like AWS EC2
  2. NodeJS Application platform (PaaS) like Heroku
  3. Serverless micro-services (FaaS) like Lambda
My final decision is Lambda due to below reasons:
  • Low maintenance (zero server to administrate)
  • Really cheap (no charges if the code not running)
  • Autoscale and high availability
  • Easy to deploy provided via serverless framework
Serverless is a hot topic in the software architecture world. I’ve been using Serverless framework for several months, I so agree with that:

To install the serverless framework, in the project root directory, type:
$ npm install serverless -g
$ aws configure
$ sls create --template aws-nodejs --path hello
Above commands install the serverless framework, setup AWS user access key ID & access key secret, create a serverless function and store the configuration to file serverless.yml

I modify the serverless.yml as below:

8. Handling user state with AWS DynamoDB

User state persistence is another important feature of RiveScript. It provides best user experience and makes chatbot more intelligent. For example:

Due to serverless micro functions are stateless. To achieve user-state persistence, we need a NoSQL server. AWS DynamoDB is the best partner with AWS Lambda. It’s an easy, powerful, and lower price. NoSQL database handling Javascript object persistent perfectly. Below dynamodb.js codes shown how easy is DynamoDB handling user-state database I/O:

In webhook.js, the sequence of message processing when integrates state management becomes important:

As you can see, it loads the user state (calls SDB.loadUserRecord()) before process the message. At the end of processing, it saves the user state (calls SDB.saveUserRecord()) after the reply sent back to the user.

9. Final Deployment

When everything is done, I deploy it to Lambda using serverless. In project root I type:
$ sls deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (8.04 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..............
Serverless: Stack update finished...
Service Information
service: iamsimonchatbot
stage: prod
region: ap-southeast-1
stack: iamsimonchatbot-prod
api keys:
  None
endpoints:
  GET - https://xw8oub6nsd.execute-api.ap-southeast-1.amazonaws.com/prod/webhookfb
  POST - https://xw8oub6nsd.execute-api.ap-southeast-1.amazonaws.com/prod/webhookfb
functions:
  webhookfb: iamsimonchatbot-prod-webhookfb

Just a single command, serverless do the rest until it returns the result URLs.

I copy the URL to Facebook App webhook subscription to change the server location from local to Lambda:

Now I can stop the local server, chat with the chatbot in my mobile Messenger App. The video captured as:


Conclusion

Just a quick recap, highlights of above procedures:
  1. Created a Facebook page.
  2. Created a Facebook App.
  3. Created a Chatbot server App.
  4. Created an AWS IAM user.
  5. Setup Messenger webhook.
  6. Tested in the local server.
  7. Gave a brain to the Chatbot.
  8. Programmed RiveScript subroutine in Javascript.
  9. Non-block calling Yahoo Finance API to get real-time stock price.
  10. User state persistent thru AWS DynamoDB.
  11. Deployed the Chatbot server App to AWS Lamba in serverless architecture.
The full source code is saved in this Github repo.








Comments

Popular posts from this blog

Create An Online Store Theme Used By MyCMS

MyCMS is an open-source Content Management System to generate static online shop website. You can use my hosting to input your products, or you can download the source codes and host it in your own server (running NodeJS). Please refer to my Github repo for the detailed installation instructions. This blog is a step-by-step tutorial that shows you how to create an online-shop theme. In this tutorial, it’s using my hosting to input the shop details and products. If you’re hosting the MyCMS by yourself, just change the domain name to yours will do. Introducing MyCMS Before making the theme, you’ll need to use MyCMS to configure the demo shop and input two demo products. MyCMS generates a static website via a theme. The generated static website is NO server program required. You can put the website files (HTML/CSS/JS) to any CDN, hosting. Shop Configuration You must prepare below settings Before using MyCMS: Setting Description Example Store name Your store name will be displayed in t

How I make a web-components based dynamic Javascript page at the top of Google Search

Introduction Everybody wants their website shown at the first position of Google search. SEO (Search Engine Optimization) is a big topic. I just helped my client's website shows the database records at the top search rankings (at least several Chinese generic keywords). See the three example questions all are listed at top ranking: Website background: My client's website  popa.qa  is a traditional Chinese Q&A site that lets members ask and answer questions. All answers and questions are storing in the database server. Step 1: Create The Project This blog illustrates the problems and the steps to fix that with project source codes. Below is the description of the basic project: NodeJS backend (server.js) Develop an API ( /get-database-records ) to simulate getting database records Web-components frontend (index.html) An example component IndexPage  make use of  LitElement to render database records To start the server type: npm start Then I check the webpage speed using th

Build A Simple Sales System using Google Drive

Google Drive is a free-to-use cloud service with a 15 GB limit storage. 15 GB is large enough to store documents, and "program" too. Yes, it's correct. We can program Google Drive. This blog is a step-by-step tutorial to show you how to build a simple P.O.S. system on Google Drive. Create a Google Sheets to define the products 1. Login to Google Drive using your Gmail account and create a Google Sheets: 2. Input the products in the Google Sheets Create a Google Forms for sale input 1. Create a new Google Form 2. Edit the form and the first field Note that leave the option field unchanged. We'll assign it programmatically in below 3. Add a quantity field 4.  Add a last field for the price input How to copy data from Google Sheets to Google Forms? We need to add a script to do that. Google developed Google App Script (GAS) for us to achieve that. Now go back to the Google Sheet we just created above. Then enter to the script editor You'll see the below screen. Then