/ chatbot

How I develop the WooCommerce Chatbot

What is the Chatbot CMS

I've developed and launched a WooCommerce chatbot (WcBot) business in January 2018. It is a mass commercial product. The chatbot main function is to connect between the WooCommerce and Facebook Messenger. See below diagram:
WcBot-Overview-Diagram-1

The target clients are WooCommerce store owners. WcBot is actually a WordPress plugin. When the clients installed and activated the plugin, the chatbot can be added to the store like below screenshot:
scrcap_001

Chatbot features

The benefits of WcBot for the shop owner:

  1. Instantly answering the messages sent from Facebook Messenger like:
    "Show me the products"
    "List the category"
    "Which products price under 30?"
    "How about the price between 10 and 20?"
    "Find product tshirt"
    "View my shopping cart"
    Example reply screenshot:
    scrcap_002

  2. WcBot has built-in shopping cart:
    Wcbot supports whole buying experience inside the chatbot. The whole buying experience means that allowing end-users to:

  • Add item(s) to cart
  • Review/Modify shopping cart items
  • Input billing & shipping address
  • Select shipping method
  • Online payment: Payment methods support PayPal, Braintree, Stripe, cheque, C.O.D. and self-pickup
  • Automatically tax calculation
  • Automatically shipping fee calculation
  • Example shopping cart screenshot:
    scrcap_003
  1. Seamlessly integrates with WooCommerce:
    The WcBot is not running alone. The info transfer between the Messenger and WooCommerce are frequently. These include:
  • Retrieving products, categories, orders, shipping, payment from WooCommerce.
  • Products lookup, prices lookup always query the WooCommerce instantly.
  • Save the paid orders to the WooCommerce.
  • Tax fee and shipping fee are retrieved from WooCommerce.
  • The store owner can perform order handling via conventional WooCommerce admin.
  • Example order handling in WooCommerce:
    scrcap_001-1

Technical Details

How To Make It Happen

To achieve the Chatbot can connect with Facebook Messenger and WooCommerce, I've developed a web server to communicate with:

  1. Messenger Webhook
  2. WooCommerce via REST API

Messenger Webhook

Messenger Webhook is the core of Messenger chatbot communication channel. Whenever the end-user sending messages and receiving chatbot replies are all via the webhook. I noticed that Webhook requires a Facebook App. Moreover, the Facebook App requires a Facebook Page association. So, the setup steps are quite complicated. I've published a step-by-step guide in the WcBot documentation. Below is the screenshot of Messenger webhook setup page:
scrcap_001-2

WooCommerce REST API

WooCommerce releases a REST API interface for developers to manipulate the data. WcBot is using this API to communicate with the WooCommerce servers. I noticed that WooCommerce REST API implements access control. Every API call requires an authorized API key. The API key is generated by the WooCommerce server when it authenticated by the shop owner. I developed a WordPress plugin for the authentication as below screenshot:
config-04

WcBot web server

WcBot web server is the main service hub between Facebook Messenger and WooCommerce servers. The WcBot web server is hosting in AWS Lambda. Below is the diagram to explain the system architecture:
WcBot-Server-Architecture-Diagram

Flow of Message Handling

WcBot does a lot of message processing. I implemented a Natural Language Processor using RiveScript. When an end-user sends a message and receives the reply. The detail is explained as below flowchart:
WcBot-Message-Handling

Shopping Cart Mini-site

WcBot has a built-in shopping cart which supports online payment. For best user-experience, end-users can buy products directly inside the chatbot without the needs of WooCommerce shopping cart. This is a unique feature of WcBot.

The shopping cart actually is a mini website. It is a Responsive Web Design based on Semantic-UI. The website frequently communicates with WcBot web server via the RESTful interfaces. Such data I/O include:

  • Shopping cart items
  • Product info in WooCommerce
  • Shipping/payment settings in WooCommerce
  • Create and save the orders to the WooCommerce

See below diagram for the web pages rendering flow:
WcBot-Shoping-Cart-Architecture-Diagram-1

The Challenges

Every development project has more or less challenges. WcBot development has the follow challenges.

Challenge 1: Server Programming

The version 1 of WcBot server is written in NodeJS. But the version 2 is re-written in Python. See below table for the characteristics of different versions:

Version Language Framework Database Server Hosting
1 NodeJS ExpressJS MongoDB Ubuntu VPS in DigitalOcean
2 Python Flask DynamoDB Amazon AWS Lambda

I loved NodeJS. Especially loving it's light-weight and async I/O (see this blog post to introduce this). But why I changed from NodeJS to Python? The reason is Python is the most widely used in AI field. It is easy to learn. Take less time to finish the tasks. When comparing with NodeJS, I've written a lot of codes to handle asynchronous and callback. For instance, below is the code fragment to handle view product in NodeJS:

function onPayloadViewProduct(client, rs, userId, recipientId, productId) {
  console.log('%s:%d onPayloadViewProduct()', path.basename(__filename), __line)
  console.assert(client, util.format('%s-%d', path.basename(__filename), __line))
  console.assert(rs, util.format('%s-%d', path.basename(__filename), __line))
  console.assert(userId, util.format('%s-%d', path.basename(__filename), __line))
  console.assert(recipientId, util.format('%s-%d', path.basename(__filename), __line))
  console.assert(productId, util.format('%s-%d', path.basename(__filename), __line))

  return new Promise((resolve, reject) => {
    let fbAccessToken = client.getFacebookAccessToken()
    let crySts = null

    getCurrencySetting(client).then((result) => {
      crySts = result
      return WOOCOMMERCE.wcGetSingleProduct(client, productId)
    }).then((data) => {
      // console.info(data)

      // Assign the product valid period. This improve UX for product querying
      data.liveExpiry = moment().add(1, 'm').toDate().getTime()

      // Save to current product
      rs.setUservar(userId, RSVAR_CUR_PRODUCT, JSON.stringify(data))
      let result = util.format('Thank you interesting on %s. The detail is...', data.name)

      // Using Async to send the messages to user one by one
      // https://caolan.github.io/async/docs.html#series
      Async.series([
        // send a 'typing' message and delay 3 seconds
        ((callback) => {
          setTimeout(() => {
            FBWEBHOOK.sendTypingMessage(fbAccessToken, userId)
            callback(null)
          }, 1000)
        }),
        ((callback) => {
          // Send the product description
          setTimeout(() => {
            // Remove all markup
            let regex = /(<([^>]+)>)/ig
            let msg = 'Description: ' + data.description.replace(regex, '')
            FBWEBHOOK.sendTextMessage(fbAccessToken, userId, recipientId, msg)
            callback(null)
          }, 1000)
        }),
        ((callback) => {
          // Send the product price
          let msg = (data.price != data.regular_price)
            ? util.format('Regular price %s. Now special %s', MISC.wcParseCurrencySetting(crySts, data.regular_price), MISC.wcParseCurrencySetting(crySts, data.price))
            : util.format('Price %s', MISC.wcParseCurrencySetting(crySts, data.price))
          setTimeout(() => {
            FBWEBHOOK.sendTextMessage(fbAccessToken, userId, recipientId, msg)
            callback(null)
          }, 1000)
        }),
        ((callback) => {
          // Send the product category
          let msg = 'Category: '
          data.categories.forEach((catg) => {
            msg += catg.name + ', '
          })
          msg = msg.substring(0, msg.length - 2)
          setTimeout(() => {
            FBWEBHOOK.sendTextMessage(fbAccessToken, userId, recipientId, msg)
            callback(null)
          }, 1000)
        }),
        ((callback) => {
          // Send the product dimension
          if (data.dimensions.length.length > 0 || data.dimensions.width.length > 0 || data.dimensions.height.length > 0) {
            setTimeout(() => {
              let msg = 'Dimension: '
              if (data.dimensions.length.length > 0) {
                msg += ' length=' + data.dimensions.length + ', '
              }
              if (data.dimensions.width.length > 0) {
                msg += ' width=' + data.dimensions.width + ', '
              }
              if (data.dimensions.height.length > 0) {
                msg += ' height=' + data.dimensions.height + ', '
              }
              msg = msg.substring(0, msg.length - 2)
              FBWEBHOOK.sendTextMessage(fbAccessToken, userId, recipientId, msg)
              callback(null)
            }, 50)
          } else {
            callback(null) // No dimension, just skip this function
          }
        }),
        ((callback) => {
          setTimeout(() => {
            let buttons = [{
              text: 'View in web',
              web_url: data.permalink
            }, {
              text: 'Add to Cart',
              payload: util.format('%s_%s', PAYLOAD_ADDTOCART, productId)
            }]
            FBWEBHOOK.sendButtonMessage(fbAccessToken, userId, recipientId, 'Do you want...', buttons)
          }, 50)
        })
      ])

      resolve(result)
    }).catch((err) => {
      logger.error(err)
      FBWEBHOOK.sendTextMessage(fbAccessToken, userId, recipientId, _connectionErrorMsg())
      // return reject(err)
    })
    
  })
} // onPayloadViewProduct()

In contrast, below is the python code fragment to do the same task:

def doViewProductDetail(self, product_id, client_rec, m_nls, user_id):
  logger.debug(str(currentframe().f_lineno) + ":" + inspect.stack()[0][3] + "()")
  assert isinstance(product_id, str)
  if self.m_woocom is None:
    rec_wc = client_rec["woocommerce"]
    self.m_woocom = mod_woocommerce.Wc(rec_wc["url"], rec_wc["consumer_key"],rec_wc["consumer_secret"])
  if self.raw_gensts is None: # needs for formatting currency
    self.raw_gensts = self.m_woocom.getGeneralSetting()
    self.parseGeneralSetting(self.raw_gensts)
  product = self.m_woocom.getProductDetail(product_id)
  acc_tok = client_rec["facebook_page"]["access_token"]
  out_msg = "Thank you interesting on {0}...".format(product["name"])
  time.sleep(1)
  mod_messenger.sendMessengerTextMessage(acc_tok, user_id, out_msg)
  out_msg = mod_misc.strRemoveMarkup(product["description"])
  time.sleep(1)
  mod_messenger.sendMessengerTextMessage(acc_tok, user_id, out_msg)
  if product["price"] != product["regular_price"] and product["regular_price"] != "":
    special = mod_misc.wcMakeCurrencyStr(self.currcy_sts, float(product["price"]))
    price = mod_misc.wcMakeCurrencyStr(self.currcy_sts, float(product["regular_price"]))
    out_msg = "Regular price {0}. Now special {1}".format(price, special)
  else:
    price = mod_misc.wcMakeCurrencyStr(self.currcy_sts, float(product["price"]))
    out_msg = "Price {0}".format(price)
  time.sleep(1)
  mod_messenger.sendMessengerTextMessage(acc_tok, user_id, out_msg)
  out_msg = "Category: " + mod_misc.strMakeComma(product["categories"])
  mod_messenger.sendMessengerTextMessage(acc_tok, user_id, out_msg)
  if product["dimensions"]["length"] or product["dimensions"]["width"] or product["dimensions"]["height"]:
    time.sleep(1)
    out_msg = "Dimension: "
    if product["dimensions"]["length"]:
      out_msg += " length=" + product["dimensions"]["length"] + ", "
    if product["dimensions"]["length"]:
      out_msg += " width=" + product["dimensions"]["width"] + ", "
    if product["dimensions"]["height"]:
      out_msg += " height=" + product["dimensions"]["height"] + ", "
    if len(out_msg) > 0:
      out_msg = out_msg[:len(out_msg) - 2]
    mod_messenger.sendMessengerTextMessage(acc_tok, user_id, out_msg)
  time.sleep(1)
  buttons = [{
    "title": 'View on web',
    "type": "web_url",
    "url": product["permalink"]
  }, {
    "title": 'Add to Cart',
    "type": "postback",
    "payload": "{0}_{1}".format(PAYLOAD_ADDTOCART, product_id)
  }]
  out_msg = "Further act on this product:"
  mod_messenger.sendMessengerButtonMessage(acc_tok, user_id, out_msg, buttons)
  return True

Not only the total number of lines is reduced nearly 50%, but also the codes are more straightforward and easier to code and read. In future, I can put all efforts to improve the chatbot intelligence. Another reason is scalability. Python/Flask perfectly support serverless architecture. Flask can handle all web traffic from Lambda. I just only need to define one handler. See below fragment in my serverless.yml:

...
# Forward all traffic to Flask handle
functions:
  app:
    handler: wsgi.handler
    events:
      - http: ANY /
      - http: 'ANY {proxy+}'
...

The cons of Python is much slower than NodeJS. Slow performance affects scalability. To address this cons, I've selected AWS Lambda and DynamoDB. AWS Lambda provides great support for scalability. DynamoDB support auto-scaling as well.

Challenge 2: WooCommerce Complexity

WooCommerce is a robust e-commerce platform. It supports different shipping classes, shipping zones, and allow shop owners to input complex calculation formulas such as:

  • 10 + [qty] * 0.10
  • 20 + [fee percent="10" min_fee="4"]
  • [qty] * 10 + [fee percent="10" min_fee="4"]

This calculation is handled in the WcBot shopping cart mini-site at the checkout page. I did a little experiment on several Javascript libraries to deal with this. Finally, I chose this lexer javascript package. Actually, it works perfectly.

Another WooCommerce setting I need to handle is checkout settings. The shop owners can input their payment options in there. (such as BACS, cheque, C.O.D., PayPal, Braintree, Stripe). The shopping cart mini-site has the ability to read those settings and render it into payment webpage. It involves the SDKs of PayPal, Braintree and Stripe payment. Braintree and Stripe are tokenized RESTful communication. All these logic are handled in WcBot server Python Flask program module.

Challenge 3: Making Web Pages In AWS Lambda

AWS Lambda is executing in various micro functions instead of a real web server. So that I cannot make this assumption to show a webpage:
http://xxx.amazonaws.com/[A_Web_Page].html

I needed to build a Lambda function to render it into HTML. Now it is:
https://xxx.amazonaws.com/MakeWebPage?page=orderPayment&OrderId=XXX

Thanks to Serverless Framework tightly integrate with Flask. The code is that:

...
return render_template("orderPayment.html")
...

Thus the Lambda function returns the HTML page from template orderPayment.

Conclusion

Now this chatbot (WcBot) business is running well. I launched its Facebook Page (for marketing purpose) and got more than 300 likes less than a month. And 7 members joined its closed group.

From the response of audiences, WcBot has bright future. I have a lot of passion on that. I want to make it supports stock control, various color/size product selling, international languages and more... Maybe convert the shopping cart mini-site into direct sales in chatting.

Resources

Below are the hyperlinks related to this project: