Continuous deployment to Amazon Lambda using Bitbucket Pipeline

I’m not a developer (more of a hack) so I’m always looking for way to figure out efficiencies in my process when playing around with code as I’m a very slow coder. One of those efficiencies found is around deploying my code to Amazon Lambda.

First, let’s talk about your options when deploying code to Lambda. The easiest way is to just do your development using Amazon’s IDE. The benefit here is that you can manually run some tests to validate what you’re writing, however if you’re using any dependencies the IDE has a size restriction and at some point it’s no longer available to you.

image

The next method is doing local development and creating a zip file of all your code and dependencies. Then manually upload your code. You can then run the same manual tests as before on your code, but the process of zipping and uploading the file is tedious specially when working on large code bases.

image

Next process involves the very good Amazon CLI. Using the CLI you’ll be able to save the manual process of uploading the zip file. Below you’ll find the Windows scripts I use one for small code bases (without dependencies) and one for larger ones.


echo on

del index.zip

echo Deleted index.zip

"c:\Program Files\7-Zip\7z.exe" a index.zip index.js

aws lambda update-function-code --function-name mySmallLambdaFunction --zip-file fileb://index.zip

echo done

[/sourcode]</blockquote>
<blockquote>[sourcecode language="bash"]

echo on

del myZip.zip

echo Deleted myZip.zip

"c:\Program Files\7-Zip\7z.exe" a myZip.zip index.js node_modules

echo Zipped myZip.zip

aws lambda update-function-code --function-name myLargeLambdaFunction --zip-file fileb://myZip.zip

echo done

Finally, the process I’ve come to enjoy the most is deploying from git. The main reason being that it forces you have a bit of a process around using git which is pretty much the standard when collaborating with multiple developers. So if you’re dragging your feet around using git take the plunge it’s worth the learning. My favorite, mainly because they have a very generous free offering is Bitbucket. Besides having private repositories they also give you 50 free build minutes which is where our deployment to Lambda from Bitbucket comes in. To get started you first need to setup a few environmental variables. Go to your repository > settings > environment variables. You’ll need these named exactly this way.

image

The next step can be done in two ways. You can commit a bitbucket-pipelines.yml file to your repository or you can go to your repository > pipelines to have Bitbucket commit one for you. What the original yml file looks like doesn’t matter we’re going to change it specifically for Lambda deployment. Here’s what my file looks like with inline comments.


#I like to use the same version of Node as the Lambda function I’m using.

image: node:6.10

pipelines:

default:

- step:

script: # Modify the commands below to build your repository.

- apt-get update

- apt-get install -y zip

- python –version #From here to there is all to enable the AWS CLI installation

- apt-get install -y python-dev

- apt-get install -y python-pip

- pip install awscli #there

- zip index.zip index.js #this is for a Lambda with a small code base. For something large you can use “zip myZip.zip index.js privatekey.json -r node_modules” notice the –r parameter to zip up folders.

- aws lambda update-function-code --function-name botValidationScheduleMeeting --zip-file fileb://index.zip

Assuming you’ve done everything right you should see something like this under Pipelines.

imageThe last 3 commits were successfully built (sent to Lambda). You can click on the commit and see detailed information on the results of every command in your yml file. You’re done, you’ve developed some code locally, committed to git, and pushed it to Lambda all with a few clicks.

~david

Bringing Amazon Lex into your Amazon Connect flows

In this blog we’ll continue our discussion around Amazon Lex. Talk about a few things to keep in mind when integrating your Amazon Lex bot with your Amazon Connect flow. In my particular use case I wanted to use Amazon Lex to look at my Gmail calendar and book a meeting if I’m available. If you want to skip to the very end you can see the end result via video. You’ll see one video of the voice interaction and one of the Facebook Messenger interaction.

First, you might want to reference my previous post around Lex validation. Now let’s talk about our use case:

  • Lex easily allows you to build a bot which understand both voice and text, so our bot needs to handle calls into our call center as well as Facebook Messenger interactions.
  • Bot needs to to ask a few question in order to find out what time the user would like to meet.
  • Bot should only schedule calls between Monday-Friday and 10 AM – 4 PM Easter Time
  • Bot (using Lambda) should schedule a meeting and if slot already taken then suggest an alternate time to meet.

Second, let’s take a quick look at the Lex screen. The bot I created is very simple and it follows closely the Flowers example provided by Amazon. These are the slots I’m requiring my bot to confirm.

image

I used two different Lambda functions. One for validation and one for fulfillment. While most examples seem to focus on using the same function for both, for me it was easier to have different code bases for each with the added benefit of keeping the code manageable. As it is both validation and fulfillment both came in at around 250 lines of code, but fulfillment had around 9 megabytes of dependencies.

image

Finally, here are sample utterances I used for the main intent.

image

What this gets us is the following. The first video is the voice interaction. I went about it the long way to show some of the validation rules being set by the bot, such as no weekend meetings and no meetings too early in the day. At the end of the video you see I refresh the Gmail calendar to show the new appointment has been saved.

In the second video I go through the same Lex bot using Facebook Messenger and then show the calendar to prove that the appointment was saved.

Ultimately, Amazon makes it extremely easy to create a mutli channel bot, however the integration to back end systems is the tricky part. This bot needs a lot of tuning to make it more natural, but for just a few hours of work there’s very little out there that can get your call center to have some bot integration for self service.

~david

Creating a Lambda Function to Validate Lex Input

In this blog we’re going to step a bit away from Amazon Connect and focus on building a conversational interface using Amazon Lex. As you can probably guess down the line, this interface/bot is going to be connected to Amazon Connect for even more contact center goodness. Here we’re going to focus on creating a Lambda function strictly for validation, not for fulfillment.

First, let’s talk about what I’m building. I’m building a bot which can schedule a time to have a call with me. You tell your intention to the bot “schedule a meeting/call” and the bot will then ask you a few questions using directed language to figure out when you want to meet. Once Lex has all the information it needs it goes out to my calendar to figure out if I’m free or busy. Second, the validation code I have is mainly based on one of Amazon’s great blueprint for ordering flowers. I recommend you start with that before trying to write your own from scratch. Finally, read through the code and pay close attention to the comments marked in bold as these were the biggest gotchas as I went through.

A couple of things to keep in mind when building a conversation interface with Amazon Lex and you’re using validation.

– Have a clear scope of the conversation. I’m not a VUI designer by any means, but if you’re planning on going with an open-ended prompt “How may I help you?” you will be working on this for a long time. Instead try to focus on the smallest possible outcome. Ultimately, it is my opinion that no IVR is really NLU and they are all just directed speech apps with a lot more money sunk into them so they can be called NLU IVRs.

– If you’re going to use input validation, every user input will be ran through Lambda. This means that you must account for people saying random things which aren’t related to what your bot does and these random things will be processed through the validation function and might generate errors. Thus, you need to ignore this input and direct the customer to answer your question, so you can move on.

– Separating validation from fulfillment makes the most sense. Other than making your code easier to read and manage, you’re also able to separate responsibilities and permissions between your two Lambda functions.

– Play around with the examples Amazon provides. They are a great tool to get started and give you a ton of building blocks you can use in your own bot.

Here’s the validation code as well as some notes, hopefully this helps someone else along the way.

'use strict';

// --------------- Helpers to build responses which match the structure of the necessary dialog actions -----------------------

//elicitSlot is in charge of building the request back to Lex and tell Lex what slot needs to be re-filled.
function elicitSlot(sessionAttributes, intentName, slots, slotToElicit, message) {
return {
sessionAttributes,
dialogAction: {
type: 'ElicitSlot',
intentName,
slots,
slotToElicit,
message,
},
};
}

function close(sessionAttributes, fulfillmentState, message) {
return {
sessionAttributes,
dialogAction: {
type: 'Close',
fulfillmentState,
message,
},
};
}

function delegate(sessionAttributes, slots) {
return {
sessionAttributes,
dialogAction: {
type: 'Delegate',
slots,
},
};
}

function confirm(sessionAttributes, intentName, slots){
return{
sessionAttributes,
dialogAction:{
type: 'ConfirmIntent',
intentName,
slots,
message: {
contentType: 'PlainText',
content: 'We are set, do you want to schedule this meeting?'
}
},
};
}
// ---------------- Helper Functions --------------------------------------------------

function isDateWeekday(date) {
const myDate = parseLocalDate(date);
if (myDate.getDay() == 0 || myDate.getDay() == 6) {
console.log("Date is a weekend.");
return false;
} else {
console.log("Date is a weekday.");
return true;
}
}

function parseLocalDate(date) {
/**
* Construct a date object in the local timezone by parsing the input date string, assuming a YYYY-MM-DD format.
* Note that the Date(dateString) constructor is explicitly avoided as it may implicitly assume a UTC timezone.
*/

const dateComponents = date.split(/\-/);
return new Date(dateComponents[0], dateComponents[1] - 1, dateComponents[2]);
}

function isValidDate(date) {
try {
return !(isNaN(parseLocalDate(date).getTime()));
} catch (err) {
return false;
}
}
function buildValidationResult(isValid, violatedSlot, messageContent) {
if (messageContent == null) {
return {
isValid,
violatedSlot,
};
}

return {
isValid,
violatedSlot,
message: { contentType: 'PlainText', content: messageContent },
};
}

function validateMeeting(meetingDate, meetingTime, meetingLength) {

if (meetingDate) {
if (!isValidDate(meetingDate)) {
return buildValidationResult(false, 'MeetingDate', 'That date did not make sense. What date would you like to meet?');
}

if (parseLocalDate(meetingDate) < new Date()) {
return buildValidationResult(false, 'MeetingDate', 'You can only schedule meetings starting the next business day. What day would you like to meet?');
}

if (!isDateWeekday(meetingDate)) {
return buildValidationResult(false, 'MeetingDate', 'You can only schedule meetings during the normal weekday. What day would you like to meet?');
}

if (meetingTime) {
if (meetingTime.length !== 5) {
// Not a valid time; use a prompt defined on the build-time model.
return buildValidationResult(false, 'MeetingTime', null);
}
const hour = parseInt(meetingTime.substring(0, 2), 10);
const minute = parseInt(meetingTime.substring(3), 10);
if (isNaN(hour) || isNaN(minute)) {
//Not a valid time; use a prompt defined on the build-time model.
return buildValidationResult(false, 'MeetingTime', null);
}
if (hour < 10 || hour > 16) {
//Outside of business hours

return buildValidationResult(false, 'MeetingTime', 'Meetings can only be scheduled between 10 AM and 4 PM. Can you specify a time during this range?');

}
}

if(!meetingLength){
return buildValidationResult(false, 'MeetingLength', 'Will this be a short or long meeting?');
}
}
return buildValidationResult(true, null, null);
}

//--------------- Functions that control the bot's behavior -----------------------
function orderFlowers(intentRequest, callback) {
const source = intentRequest.invocationSource;
//get appointment slots
const meetingDate = intentRequest.currentIntent.slots.MeetingDate;
const meetingTime = intentRequest.currentIntent.slots.MeetingTime;
const meetingLength = intentRequest.currentIntent.slots.MeetingLength;

//For fullfilment source will NOT be DialogCodeHook
if (source === 'DialogCodeHook') {
//Perform basic validation on the supplied input slots. Use the elicitSlot dialog action to re-prompt for the first violation detected.
const slots = intentRequest.currentIntent.slots;
const validationResult = validateMeeting(meetingDate, meetingTime, meetingLength);

if (!validationResult.isValid) {
slots[`${validationResult.violatedSlot}`] = null;
callback(elicitSlot(intentRequest.sessionAttributes, intentRequest.currentIntent.name, slots, validationResult.violatedSlot, validationResult.message));
return;
}

//Pass the price of the flowers back through session attributes to be used in various prompts defined on the bot model.
const outputSessionAttributes = intentRequest.sessionAttributes || {};
callback(delegate(outputSessionAttributes, intentRequest.currentIntent.slots));
return;
}
}

// --------------- Intents -----------------------
function dispatch(intentRequest, callback) {
const intentName = intentRequest.currentIntent.name;

//Dispatch to your skill's intent handlers
if (intentName === 'MakeAppointment') {
return orderFlowers(intentRequest, callback);
}
throw new Error(`Intent with name ${intentName} not supported`);
}

//--------------- Main handler -----------------------
//Route the incoming request based on intent.
//The JSON body of the request is provided in the event slot.
//Execution starts here and moves up based on function exports.handler => dispatch =>orderFlowers=>validateMeeting=>buildValidationResult is the most typical path a request will take.

exports.handler = (event, context, callback) => {
try {
dispatch(event, (response) => callback(null, response));
} catch (err) {
callback(err);
}
};

Right way to block ANIs using Amazon Connect

In this blog I’ll cover a potential financial issue you might face if you try to ANI block customers and they are calling you through a SIP trunk.

As I continue my journey of getting familiar with Amazon Connect I ran into an interesting and a bit worrisome issue. The use case I was working on was to create a table which blocks or allows specific ANIs to call in. Ultimately, when a blocked caller came in I wanted to just hang up on the call. My original flow looked like this:

image

Pretty straight forward, invoke Lambda, check attribute and if blocked = true, disconnect the call. When calling from my cellphone this worked great. However, when calling from my home phone (using a Flowroute SIP trunk) I got a nice surprise in the logs:

image

What you’re seeing is a partial log of my home phone constantly retrying to connect to Amazon Connect and generating a new call each time. Since there was no prompt play and no ring back heard I assume the network believes there as a connection issue and continues to try and connect. Which means that you could easily incur a huge expense both on your phone provider and on your Amazon AWS bill.

The way to fix this was to play a 1 second of silence prompt before disconnecting the call.

image

~david

Amazon Connect and Sticky Queue

In this blog I’ll discuss how to achieve a sticky queue using Amazon Connect.

When a customer calls back within a short amount of time, it’s fairly safe to assume they are calling back for the same reason as before. This is often referred as sticky agent or sticky queue. Because you’re trying to “stick” a queue or agent to a specific customer. As a best practice avoid using anything sticky as it could force your customer down the wrong path or create long hold times when too many callers are stuck to a single agent or queue, but for my use case it’s safe to use because I want to use it. :)

I assume you already have a Lambda function or two working with your flow, if you don’t then you might want to skip this functionality until you get that working. The first thing you need to do is find the ARN for your queues. I’m going to be honest, this is not intuitive at all and I wish the Connect team would allow you to retrieve this information via the flow without having to do the following steps.

To get your Queue ARN go to your queues via https://<your instance>.awsapps.com/connect/queues, select a single queue, and notice the bold section of the url https://<your instance>.awsapps.com/connect/queues/edit?id=arn:aws:connect:us-east-1:64:instance/4d92ab25-8XXX-4bXX-aXXX-XXXXXXX/queue/XXXXXX-2022-XXXX-a275-xxXXXXxxxXXX That’s your queue’s ARN.

– Set an attribute (e.g. Queue) with your ARN.
– Set your queue name to the attribute you just set.

image

– Save your attribute to your DB.

Next time your customer calls, you can retrieve the last queue they went through and give them the option to go to that queue again, hopefully saving your customer some frustration.

image

~david

Amazon Connect Flow Designer Review

I’m trying to capture my initial notes and reactions to Amazon’s contact center offering. In this blog I’m going to focus solely on their flow designer tool. I’ll provide a brief overview of the tool, some best practices I’ve come up with, as well as some things I wish were different. Remember that I come from the Cisco contact center world, so my view is slightly tainted and what I’ve lived and loved has been the Cisco tools.

Amazon Connect provides a web based call flow tool called flow designer. Those of you familiar with ICM Script Editor and CVP Studio will feel at home. Below is one of the flows I’ve created. Note that the designer allows you to snap steps or what Amazon calls “action blocks” into the grid for cleaner looking flows.

Flow designer with flow

In the left hand side of the designer is your “palette” you can find an explanation of each action block here.

Flow Designer Palete

Building your first flow is truly easy and requires very little technical knowledge. The Play prompt block allows both playing audio files as well as text-to-speech (TTS) in a variety of voices and languages. Setting a queue and building a queue is just as easy.

Now a few items which bother me about contact flow as well as some best practices I’ve found. I touched on a few of these in my earlier post.

  • DO NOT hit the back button or navigate away from the flow designer without saving. There is no auto save!
  • You can’t copy and paste a block. You must build a block from scratch every time. I keep a file with Lambda names and variables I’m using for easy copy/paste.
  • You can’t have the block properties of multiple blocks open at the same time.
  • There is no move of multiple blocks. You must move each one at a time.
  • Build your flow strictly with TTS and only add audio files once you’re happy with the product. If you’re using dynamic speech you’ll have a better sense of what the audio files need to say.
  • Plan your error conditions flow early. This is important when handling error/default/timeout from menus, but applies across multiple different types of blocks. You should come up with a few standard error correction flows and branch out all your error conditions appropriately based on where you are in the flow. This will also avoid a spider web flow.
  • No easy way to get from flow to flow. Once you’re in the designer, you click the back button in your browser or go through the main menu to jump to another flow. Ideally Amazon provides a drop down in the designer to switch between flows to save a few clicks.
  • No infinite scroll. Specifically you can’t scroll and build your flow up or to the left. This means that you should think of starting your flows somewhere in the middle of your screen to give you a bit of real estate for last minutes changes/branches. When you create a new flow Amazon “conveniently” starts you off like in the left image, but you should move your fist block off to the right a bit, like right image. Also make sure you immediately enable “Snap to grid” for cleaner looking flows.

image image

  • You are able to move blocks behind the unmovable left hand margin. The only reason I discovered this is because I wanted to add a log block and didn’t want to pile up blocks on top of each other.

Flow designer with nodes hidden.

  • You need to be aware of where your lines are going and try to avoid overlap and tight spaces, specially when using the Get customer input block. Trying to modify a line in the middle of the block can be difficult and will require for you to delete other lines to get to the line you want to delete or modify.

image

  • DO NOT hit the back button or navigate away from the flow designer without saving. There is no auto save! Yes, it’s a repeat, happened to me multiple times.
  • When saving a flow or publishing a flow you get the same confirmation. It would be nice to be reminded what was the last action you took for those of us who are jumping from screen to screen.

Flow designer save message.

~david

Initial Observations of Amazon Connect

If you don’t know what Amazon Connect is and you’re in the contact center world, you might want to consider a new career path. Amazon Connect is AWS’s answer to the Ciscos, Genesys, Avayas of the world. Not only that, but also a competitor for Twilio, Microsoft, and anyone who carries voice from point A to point B. Needless to say, when the Amazon giant moves everyone pays attention. A lot of these are just a brain dump so pardon the brevity. I’m still trying to dig a bit deeper and come up with specific ideas to blog specially comparing Amazon’s solutions with Cisco’s offering(s).

Things which are awesome:

– Agent logs are in JSON format, holy crap that’s awesome!
– Hours of operations are available out of the box and are granular to the minute. Ability to add exceptions for the same day is a nice touch.
– If you associate an email with your agent your agent can reset their own password.

Things which are strange:

– Can’t change agent state while reserved or talking.
– If you use a desk phone, you can’t reject the call.  
– Changes take about a minute or two to propagate and there’s no notification if your changes are live or not.      
– If you create a new agent and then login as that agent using the same browser as before your admin session will be moved over to the new agent credentials. Painful when trying to test permissions on agents.     
– You can’t re-route a connector by clicking on the start point, you must first delete the existing line and then create your new connector.

Things which absolutely make no sense:

– Every step should have a Lambda invocation option. This would make the scripting a lot cleaner.
– If you reject a call and you’re the only agent you’re automatically set back to ready. Queue must be drained before last agent can change states out of available.
– No default routing? I disabled the only queue and calls just dropped when I tried to route to that queue. You would think that the system would force some sort of default routing option just in case you make a mistake.  
– Contact flow editor, no easy way to get back to all your contact flows.
– Agent auto accept takes about 12 seconds to trigger using softphone, this would impact agent stats and I really don’t see the point of having this feature if it’s going to take this long to connect an agent.
– When you save or publish a contact flow you get the same message "Contact flow saved successfully!" Different message for publish would be nice.
– No easy way to move the whole script. Work area should have infinite scroll to all sides.
– You can’t select multiple nodes and move them, you must move one by one.
– Flows don’t auto save drafts, if for some reason you don’t remember to save you’re SOL.
– How draft flows and published flows are handled is confusing. Not very user friendly.
– Checking contact attributes doesn’t offer a NULL or NOT NULL condition check.
– When a connector goes behind a flow node, you can’t delete the connector.
– No way to duplicate nodes. You must configure a new node from scratch every time.   

~david

Connecing an ESXi host to a QNAP NAS using NFS

Interesting little issue I ran into when trying to create a new datastore in my ESXi server. I had to use NFS v2/v3 even though the ESXi documention states v4 is supported. Here are my specs:

– QNAP TS-219P II 4.3.3.0404

– VMWare ESXi v6.0.0 Build 5050593

To configure the NFS share in the NAS. Control Panel > Win/MAC/NFS > NFS Service

QNAP NFS Service

To ensure your share has NFS partitions click on the link that says “Click here to set the NFS access…” Choose your shared folder and Edit Shared Folder Permissions. And click

QNAP NFS host access screen

In ESXi, click on the host server. Configuration > Storage > Add Storage…

1. Network File System

2. Server IP, Folder, Datastore Name

3. Finish

ESXi Locate Network File System

When done, you should see your NAS as a data store.

ESXi Datastore Configuration

~david

Installing Laravel 5.5 & MySQL in Ubuntu 16.04 with Nginix already installed

Here are my notes on how to install these components.

sudo apt-get install php7.0-mbstring php7.0-xml composer unzip
sudo apt-get install -y php7.0 php7.0-fpm php7.0-mysql php7.0-zip php7.0-gd
sudo apt-get install mcrypt php7.0-mcrypt
sudo apt-get install -y php7.0-mbstring php7.0-xml –force-yes
sudo apt-get install php7.0-curl php7.0-json
sudo vi /etc/php/7.0/fpm/php.ini
    cgi.fix_pathinfo=0
sudo service php7.0-fpm restart
sudo mkdir -p /var/www/laravel
sudo vi /etc/nginx/sites-available/default
server {
        listen 80;
        listen [::]:80 ipv6only=on;

        root /var/www/laravel/public;
        index index.php index.html index.htm;

        # Make site accessible from http://localhost/
        server_name <serverName>;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ /index.php?$query_string;
                # Uncomment to enable naxsi on this location
                # include /etc/nginx/naxsi.rules
        }
        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }
}
cd ~
curl -sS https://getcomposer.org/installer | php
sudo mv composer.phar /usr/local/bin/composer
sudo composer create-project laravel/laravel /var/www/laravel
sudo chown -R :www-data /var/www/laravel
sudo chmod -R 775 /var/www/laravel/storage
sudo chmod -R guo+w storage
sudo apt-get update
sudo apt-get install mysql-server
sudo mysql_secure_installation

Cisco Spark Webhooks using Node.js

I’ve been trying to come up with a reason to play around with node.js. I’ve also been trying to make some time to play around with the Cisco Spark API. So I figured I would merry the two of them. My intention was to setup a webhook on a channel, then in my node application display the messages being posted in my room. Pretty simple example, but it touches a few different things.

First, you need to setup your webhook to point to your server.

Second, you need to have node installed, in this case I’m using Ubuntu 16.04.

Third, write some code. Check out my comments to follow along.

//will use these for the POST request

var express = require("express");
var myParser = require("body-parser");
var app = express();

//will use this for the GET request

var http = require(‘https’);

app.use(myParser.json());
app.post("/", function(request, response){

console.log(request.method);

//console the webhook name as well as the name of the author of the message
console.log(‘Webhook:’+request.body.name);
console.log(‘Email:’+request.body.data.personEmail);

//setup your GET request to find out the actual message posted by the author

var options = {
host: ‘api.ciscospark.com’,
path: ‘/v1/messages/’+request.body.data.id,
headers: {
  ‘User-Agent’: ‘request’,
  ‘Authorization’: ‘Bearer MyAuth’
}
}

//make the request

http.request(options, OnResponse).end();
});

//capture the request response

function OnResponse(response){
var data = ”;
response.on(‘data’, function(chunk){
  data += chunk;
});

response.on(‘end’, function(){
  data = JSON.parse(data);

//console out the actual text
  console.log(data.text);
});
}

app.listen(8080);

~david