Category / Writings

by dmacias

How does Amazon Connect CCP Agent Login and Logout Work

I didn’t think a call center platform could make the topic of an agent login and logout interesting, but Amazon Connect has figured out a way to make me question my sanity with the design decision they have made.

We are going to do a deep dive into some of the uniqueness and outright strange behavior the CCP has when it comes to agent login and logout. This assumes you’re running the out of the box CCP, which means zero customization. Additionally, we’re going to be looking at 5 different screens at the same time to get an overall picture of what is going on and it’s impact to reporting. We will be looking at the Amazon Connect Dashboard, CCP, Real Time Agent reporting, Login/Logout report and Agent activity audit report.

Login to CCP

There are two ways that agents could login to CCP. The first is by logging into the Connect Dashboard (htts://<your-instance>.my.connect.aws/) then having agents launch CCP from there here’s what you need to know. The second way is to go to the CCP directly (https://<your-instance>.my.connect.aws/ccp-v2). Let’s walk through the first method step by step and see some of the reporting implications.

Step 1: A dashboard login:

  • Will show up in the Login/Logout report.
  • Will not show up in the Real Time Agent report.
  • Will not show up in the Agent activity audit report.

Why is this important? From the Dashboard to taking a call center call, the agent still needs to launch CCP and go into the Available state. I would argue that a better metric of login would be when the agent launches the CCP and not just logs into the Dashboard. The documentation for the Login/Logout report state this:

The Login/Logout report displays the login and logout information for the users in your contact center (for example, agents, managers, and administrators).

So the documentation is aligned with the behavior you see, but I would argue that, when it comes to agents, this report is misleading as an agent still has a few steps to follow before they are ready to take a call. Now, if you open the Agent activity audit report you will not see any data either which makes sense based on the Login/Logout report. Thankfully, both reports line up. Now, let’s get our agent from the Dashboard to CCP. By default the CCP should be in the Offline state.

Step 2: Launch CCP and:

  • The Login/Logout report doesn’t change as this is based on the Dashboard login.
  • You still will NOT show up in the Real Time Agent report.
  • You still will NOT show up in the Agent activity audit report.

Our agent has logged in to the Dashboard, launched CCP and is in an Offline state, but we’re still not seeing anything outside of the Login/Logout report. I would have expected the Real Time Agent report to show me the state of the agent, but that’s not happening. What’s even more infuriating is that if the agent changes states and then goes back to Offline, the Real Time Agent metrics have data about my agent. So why is Offline after login any different than going from one state to Offline state? To be fair, the agent will drop off from the statistics in about 3 minutes, however I think AWS should change this behavior and by default show any agent who is logged in and Offline to make things more consistent.

Logout of CCP

You would think that logout would be much easier, but there are still a few gotchas on this. The biggest thing to remember is that just closing the CCP window will not log you out. Another thing to consider is that a CCP logout will put you in Offline first then log you out. This is important to note if you have custom states you want to use for logout. However, if you are in the Dashboard and you launched the CCP and you logout from the Dashboard it will not properly log you out of the CCP. The CCP will eventually give you a login screen, but:

  • Real Time Agent metrics will still show you in the last state you were in.
  • Login/Logout report will show you logged out.
  • Agent Activity audit will still show you in the last state you were in.

Make it Better

Here are what I believe to be some simple suggestions to make this experience better. I hope that someone at AWS takes some of these and puts them in the next sprint.

  1. Offline state shows up in Real Time Agent metrics.
  2. The Routing Profile allows for configuring the default agent state upon login. This will put agents in any state when they first login.
  3. Dashboard or CCP logout behave the same way and reflect correctly across all reports.

Login and Logout at a Glance

Action Real Time Agent Login/Logout Report Agent Activity Audit
CCP Launch No Data Login time recorded either from Dashboard or CCP login No Data
CCP State Change State Data No Change State Data
Dashboard Logout Previous State Continues Logout time recorded No Data
CCP Logout Offline for about 3 minutes then stats disappear. Logout time recorded Offline State

~david

by dmacias

Obtained the AWS Solution Architect Associate Certification

I wanted to capture my experience working towards attaining the AWS SAA certification in hopes that this helps others on the same journey. This information is up to date as of 12/06/2021.

BACKGROUND

I first started working on getting the SAA back in middle 2020. I am the type that I like to  book the exam and then start planning for it. So, I booked the exam for August 19, 2020. At the time I used Adrian Cantril’s course. Which, I’ll be honest is great, very detailed. However, for me I was not ready for that type of detail and it was hard for me to focus and make time for the videos. I started strong, but started falling behind and was never able to catch up. On my first attempt I failed.

AWS SAA Failed Report

AWS SAA Failed Report

GETTING OVER THE HUMP

In the summer of 2021, on my city’s Slack, a few of us started talking about wanting to renew or pass an AWS certification. I put together a quick Google Sheet for people to talk about what they were trying to achieve and why. From there we had an initial meeting in early July and then decided to meet every Monday. One Monday in person and the next virtually via Webex. We ended with 4 total participants. 3 of us going for the SAA and 1 for Security. This was great as it created a constant reminder that I needed to study and stay with the group. I highly recommend to join or start a study group. The extra motivation helped me stay on track.

RESOURCES

As a group we talked about what resources worked best and compared notes about what we’ve checked out and what we liked/disliked about them. At the end of the day we all focused on a single primary resource. I personally supplemented my studies with a few other just to get multiple perspectives. Here’s what I used and in the order I used them:

A Cloud Guru (primary resource)

Tutorials Dojo Study Guide eBook (I printed this out and had it bound and would keep next to my bed to review the parts I felt that I needed further review)

Tutorials Dojo SAA Practice Exams

r/AWSCertifications

I enjoyed ACG’s video course. The videos were short and full of information and while the labs were ok, the gold was found in the videos alone. The TD eBook was good too, but I would skip it if you want to save money. Finally, the week before the exam I would go through practice exam question and read in detail the description for the questions I failed. Additionally, I would then go to the AWS documentation and get a bit more in depth to hopefully gain some new knowledge. I kept an eye on Reddit to see what others who had taken the test had to say about the topics covered. This allowed me to focus my studies. Personally, I feel that the practice exams was what got me to pass the exam. Not only because I was very used to the question style, but I was able to get a good feel for spotting the gotchas in the answers.

My exam had a lot of EFS, auto scaling, application and network load balancing. Good luck!

~david

PS: From our group all 3 passed the SAA and in a few days the last member of our study group will take the security exam and I’m sure will pass.

by dmacias

Adding Text to Speech to Your IVR Using SaaS (Google Cloud Functions)

A bit of code to allow your on premise IVR to have TTS using a cloud function.

I’ve been on a text-to-speech and speech-to-text kick lately. My last post talked about using AWS S3 and Amazon Transcribe to convert your audio files to text and in previous articles I’ve covered how to create temporary prompts using Poly so you can build out your contact center call flows. Well, now we’re going to expand our use case to allow a traditional on premise call center to leverage the cloud and provide dynamic prompts. My use case is simple. I want my UCCX call center to dynamically play some string back to my caller without having to use a traditional TTS service.

First, this is not new in any way and other people have solved this in different ways. This Cisco DevNet Github repo provides a method to use voicerss.org to generate TTS for UCCX. However, this process requires loading a jar file in order to do Base64 decoding. Then there’s this Cisco Live presentation from 2019, by the awesome Paul Tindall, who used a Connector server to do something similar. To be fair the Connector server allowed for a ton more functionality than what I’m looking for.

Screen Shot 2021-09-15 at 3.38.30 PM

Cisco Live Presentation

Second, I wanted this functionality to be as easy to use as possible. While functionality keeps getting better for on premise call center software there are still limitations around knowledge to leverage new features and legacy version that can’t be upgraded that makes it harder to consume cloud based services. I wanted the solution to require the least amount of moving parts possible. That means no custom Java nor additional servers to stand up.

The solution I came up with leverages Google’s cloud (GCP) specifically Cloud Functions. However, the same functionality can be achieves used AWS Lambda or Azure’s equivalent. At a high level we have an HTTP end point where you pass your text string to and in return you will get a wav file in the right format which you can then play back.

Blank diagram

Flow Diagram

The URL would look something like this:

https://us-central1-myFunction.cloudfunctions.net/synthesize_text_to_wav?text=American%20cookies%20are%20too%20big

The Good Things About This

  • Pay as you go pricing for TTS. Looking at the pricing calculator a few hours of TTS a month would run under $2.00/month.
  • Infinitely scalable. If you’re handling 1 call or 100 calls your function will always return data.
  • Easy to use.

The Bad Things About This

  • There is a delay between making the request and getting the wav file. I’ve seen as long as 7 seconds at times. I would only use this in a very targeted manner and ensure it didn’t affect the caller experience too drastically.
  • Requires your on premise IVR to have internet access. Often time this is a big no no for most businesses.

Some initial testing with UCCX is showing some positive results. I’m going to investigate if there’s a way to accelerate the processing in order to keep the request and response in under 3 seconds as well as adding the ability to set language, voice, and even SSML via arguments. If you want to build this yourself here’s the code for the function.

def synthesize_text_to_wav(request):
"""Synthesizes speech from the input string of text."""
text = request.args.get('text')

client = texttospeech.TextToSpeechClient()
input_text = texttospeech.SynthesisInput(text=text)
voice = texttospeech.VoiceSelectionParams(
language_code="en-US",
name="en-US-Standard-C",
ssml_gender=texttospeech.SsmlVoiceGender.FEMALE,
)
audio_config = texttospeech.AudioConfig(
audio_encoding=texttospeech.AudioEncoding.MP3
)
response = client.synthesize_speech(
request={"input": input_text, "voice": voice, "audio_config": audio_config}
)

src_file_path = '/tmp/output.mp3'
dst_file_path = '/tmp/output.wav'

# make sure dir exist
os.makedirs(os.path.dirname(src_file_path), exist_ok=True)

# The response's audio_content is binary.
with open(src_file_path, "wb") as out:
out.write(response.audio_content)
print('Audio content written to file "output.mp3"')
AudioSegment.from_mp3(src_file_path).export(dst_file_path, format="wav", codec="pcm_mulaw", parameters=["-ar","8000"])
return send_file(dst_file_path

Be awesome!

~david

Contact Flow Block
by dmacias

Amazon Connect Contact Flow Editor Frustrations and Annoyances

The last few weeks have given me an opportunity to dig deeper into the AWS’s Amazon Connect solution. The purpose of this post will be to document some of my frustrations with the contact flow editor which I believe should be easy to fix. There are a few more frustrations which I don’t believe are easy, so I’ll save those for another post.

In the Contact flows screen there are a few things I would love to see. Firsts, make the Name, Type, and Description columns sortable. When there’s a very long list of flows this allows to quickly see specific groupings of flows together. Second, add an action menu at the end of the row which allows for a quick way to publish, duplicate, or delete a flow. This simple menu would save at least 2 extra clicks and covers some of the common actions one takes once inside the editor.

Amazon Connect Contact Flows Screen

Amazon Connect Contact Flows Screen

Let’s click on a specific contact flow and take it from the top down. Perhaps the most important thing here would be an auto save or confirmation that you might lose your changes if you try to navigate away from the editor screen. It’s very common that you have to go find a queue or prompt ARN, as they are not available in the editor, so if you browse away from the editor without saving you’ve lost your progress.

In an ideal world you would be able to easily switch from flow to flow without having to go back to the main contact flows screen. A drop down next to the flow name to quickly switch between flows would be great. This is specially useful when you want to copy and paste blocks between flows.

Contact Flow Editor Top Bar

Contact Flow Editor Top Bar

I’ve covered this one before, but it fits nicely with this post. You can save a flow or publish a flow and you can do both by using the publish button. However, the confirmation notification that you performed one or the other doesn’t confirm which one you did. This is minor annoyance as you can then move your eyes from the far left of the screen where the notification is found to the top right of your screen to see the Latest button label which will tell you the status of your current flow. It would be nice if the notification confirmed which action you took, I find myself  savings when I meant to publish and it always takes me a second to confirm which one I did.

Screen Shot 2021-06-23 at 6.07.16 AM

The workspace or the big white area with a grid where you place your blocks is the next focus. It’s nice to have infinite room to the right and bottom, but it would be nice if the same would be applied to the top and left. As your flow has more and more branching you find yourself scrolling further and further down, it would be nice if by default your start of the flow would be centered by default.

Remembering the zoom level or setting a default zoom level per flow will save a few clicks. As the flows gets bigger you start having to zoom in and out a lot more. When the flow is rather large, there’s very little reason why the default zoom level is the same as a flow which only has a few blocks. Below is the default view of two different flows. While this default view is useful for smaller flows, for bigger flows it’s not as useful and you immediately have to scroll around or zoom out to get to where you want to go faster.

Default View of Two Different Flows

Default View of Two Different Flows

One of the great things about contact flows is how many things you can do dynamically. However, you first have to bring in a lot of that data over to the flow. You do this with a Set contact attributes block. However, as you add more attributes it starts to become harder to find the right attribute you want to reference. Adding a way to sort or collapse every attribute details would make it much easier to find the attribute you’re looking for. This is specially important as the editor doesn’t tell you what attributes have already been defined. If that was the case this point might not be as important. Imagine having to scroll through a dozen of the aqua blocks below, having to  read each name to get to the one you want. Painful!

Amazon Connect Set Contact Attribute

Amazon Connect Set Contact Attribute

And finally, my biggest annoyance: block details. Block details are the configuration options each block has. You can get to them by clicking the top dark gray bar of any block.

Contact Flow Block

Contact Flow Block

There are two annoyances here. First, you can’t click on the workspace to close the block details. Even if you didn’t make a change you have to move your mouse to the far bottom left or right to click the cancel/x buttons. Heck, let me use the escape key to get out of there. So often I just need to see the configuration without making changes and having to cancel out every time takes unnecessary steps.

Contact Flow Configuration Details

Contact Flow Configuration Details

And finally, do not show me the details of a block that has nothing for me to configure. If by mistake I clicked on the gray bar of a terminate block. There’s nothing of value that this screen adds. Others might find the link to the documentation helpful, but I don’t. This annoyance would be lessen if I could quickly  escape from this screen, but since I’m being held captive by it and there’s no useful information displayed it just makes my blood boil.

Terminate Block Details

Terminate Block Details

I hope this post is not taken as me saying that Amazon Connect is not a good solution, far from it. This post is out there in hopes that someone at AWS who has the power to make things better sees what I believe are common frustrations.

~david

by dmacias

Microsoft and Nuance a Brilliant Play for Redmond

Bloomberg reports that Microsoft “is in advanced talks” to purchase Nuance. I’ve not been able to stop thinking of this move and I’ll be the first to admit that it surprised me, but the more I think about it and the more I talk to others in the industry this is an absolutely brilliant move. Here are my crystal ball predictions:

The patent play: Nuance comes with over 2000 patents. This is a huge cache which will no doubt be useful for the upcoming AI wars. This will be a huge boosts to Microsoft’s already huge R&D commitment in this space.

This hospital bill is brought to you by Microsoft: Nuance makes the majority of their money from the healthcare sector. We’re not talking just dictations or document management, we’re talking EHR, billing, and diagnosis software. Windows and Office are already prevalent in the healthcare space, this puts MS in the heart (get it?) of hospital operations and processes.

Cortana, it was the best of times it was the worst of times: Did you know that Windows 10 was Cortana’s big debut in the desktop space? Yeah neither did anyone else. Cortana began in 2014 as a direct competition to Alexa which was released the year before. At the time Microsoft was beginning to make some heavy bets in to the mobile space with Windows mobile. Well it’s a decade later and Windows mobile is dead, Cortana’s OS integration has been neutered and I’ve never seen another human being speak to their Windows PC. I believe this is going to change that with a huge marketing push of some college student dictating their final paper to their Microsoft Surface device while getting a manicure.

Where we’re going we don’t need passwords: Nuance comes with perhaps one of the oldest if not best speech biometrics software. Imagine joining a Microsoft Teams meetings where you call in and start speaking and you’re authenticated immediately. Or allow for “signatures” based solely on your voice. Verification and authentication continue to be huge and the rise of better and better “deep fake” technology will allow for some sort of trust verification service with Nuance biometrics in the middle of it.

(Part 1) We’re taking our ball and going home: This one is near and dear to me heart. If a Cisco call center customer wanted to have speech recognition or text to speech there was only one name in town. Nuance. This has changed a bit in the last few years with the introduction of LumenVox as an additional option. And this has changed even further in the last 18 months with Cisco supporting Google’s DialogFlow, but Nuance still reigns supreme. I can see MS increasing the pricing of an already very expensive product making it prohibitive for some call centers to run their software.

(Part 2) I can see clearly now the rain is gone: Did you know that Azure stands for the color blue of a cloudless sky? Microsoft will be able to create a very defensible moat around their Azure offerings by being the only provider to have the latest and greatest Nuance services. In addition, some telephony cloud provider, who are already battling Amazon and who white label Nuance products as part of their offering, might be forced to either consume more Azure resources to get better pricing or completely get priced out from this technology and watch the competition pass them by. This sets the stage for Microsoft to be in the driver’s seat of what UC or CC provider you might choose next if you have an already deep Nuance integration or if your call center must use Nuance.

 

 

 

 

 

 

by dmacias

What “Zoom Fatigue” can teach us about using video in the contact center

My significant other researches human behavior at work and she brought this study to my attention. She thought it would be very relevant to what I do. I want to summarize some key points and how they relate to the contact center, but first, I really wanted to title this post as “Why video will never kill the phone contact center star”, but that seemed too childish.

So what can Zoom Fatigue teach us about using video in the call center? Let me highlight a few key points from the article.

…in one-on-one meetings conducted over Zoom, coworkers and friends are maintaining an interpersonal distance reserved for loved ones.

Is your customer relationship what you would consider intimate? I can’t think of a single service or product where I would use that term, so the answer is more than likely no. Now, imagine having to handle customer video calls all day and feeling your personal space invaded. It would be exhausting for agents and off-putting for customers. Video calls should be reserved for customers with a long-standing relationship and limited to a few key agents who know the customer well. Additionally, considerations should be made around how many video calls an agent should handle in a short amount of time.

One of the remarkable aspects of early work on nonverbal synchrony (i.e., Kendon, 1970) is how nonverbal behavior is simultaneously effortless and incredibly complex. On Zoom, nonverbal behavior remains complex, but users need to work harder to send and receive signals.

Processing these extra nonverbal signals contributes to what my significant other and other researchers call “cognitive load” or the amount of information our brains can process at any given time. Video calls divert precious mental resources away from the task at hand, making it more likely your agents will make mistakes on complex tasks like financial services or billing. Traditional audio-only phone calls enable them to focus better while doing their work.

There is no data on the effects of viewing oneself for many hours per day. Given past work, it is likely that a constant “mirror” on Zoom causes self-evaluation and negative affect.

Self-view is very distracting for me, however that’s the only way for me to know if I’m in view or not as I use a standing desk. In addition to being distracting, this article argues that it is also stressful. If your agents are handling video calls, consider the ability to turn on and off self-view. Vendors should come up with a technology solution which notify the users when they are out of view without relying on the equivalent of looking in the mirror all day long.

…cultural norms are to stay centered within the camera’s view frustrum and to keep one’s face large enough for others to see. In essence users are stuck in a very small physical cone, and most of the time this equates to sitting down and staring straight ahead.

We first had handsets and they were terrible to hold and work at the same time. We then got wired headsets and life was better, but we needed to make sure not to get tangled or have someone kick them. We then moved on to wireless headsets and we got freedom! … only to have it taken away by video that creates a “lock in” effect. You can no longer just stand up and stretch. You can’t just run to the fax machine or to refill your water bottle. If your agents collaborate in a team to handle customer requests or handle calls which can be very lengthy avoid having them on camera.

~david

UCCE Call Flow Outbound Option with SIP Proxy
by dmacias

Cisco UCCE Outbound Dialer IVR Campaign Log Walkthrough

I wanted to document this specifically as I feel it’s the most complex flow you will see within UCCE. Before you go through this you will absolutely want to get familiar with the documentation as there are a lot of moving pieces. The diagram below comes straight from said documentation and should help you visualize what you see the logs doing.

 

UCCE Call Flow Outbound Option with SIP Proxy

UCCE Call Flow Outbound Option with SIP Proxy

Ideally I would have had all the logs for all the devices at the same time, but unfortunately that wasn’t the case. You will notice that the timestamps jump around as some logs are from other time frames. However, I’ve tried to match up all unique identifiers across all the logs so you can follow it along. The unique identifiers you want to take note of are:

2145551234: Customer phone number campaign is going to dial.
016: The dialer port.
10241: Correlation ID of initial call to customer.
10242: Correlation ID of call to IVR.

1. An unattended IVR campaign starts. Customer records are delivered to the Dialer. (The ANI and the port are key to be able to trace what the dialer is doing.)

badialer:
10:55:33:059 dialer-baDialer Trace: (Customer) SetCallResults(): ID: [-2147483601 in DL_5008_5031], skill: [6924], result: [1] [DIAL_RESERVED], now: [Thu Mar 4 10:55:33 2021], callback: [Thu Mar 4 10:55:32 2021]. 
10:55:33:059 dialer-baDialer Trace: (CUSTMGR) SendRecord(): Send customer: [2145551234] record ID: [-2147483601], in: DL_5008_5031 to port: [016]. 
10:55:33:059 dialer-baDialer Trace: (IVR) Record available event, port: [016], phone: [2145551234]. 
10:55:33:059 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [PORT_DEVICE_ATTRIB].

2. The Dialer asks the SIP Proxy to forward an invite to an available gateway to start a call. (CUSP logs are a bit of a pain, I recommend you download them via FTP instead of doing it from the GUI. This is a great guide on log settings. Ultimately, here we want to make sure that the CUSP knows where to send the request.)

badialer:
10:55:33:059 dialer-baDialer Trace: (SIPDisp) Dial, port: [016], phone: [22222912145551234], lDialTimeoutSec: [32]
CPA parameters: AP: [2500], MinSP: [608], MinVSpeech: [112], MaxTA: [5000], MaxTone: [30000]. 
10:55:33:059 dialer-baDialer Trace: (IVR) Dialing, phone: [22222912145551234], port: [016], ring timeout: [32], state: [PORT_DEVICE_ATTRIB]. 
10:55:33:059 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [DIAL_CUSTOMER].
CUSP:
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:468 nrs.FieldSelector - getUriPart: URI - sip:22222912145551234@{CUSP} part 6
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:468 nrs.FieldSelector - Requested field 45
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:468 nrs.FieldSelector - Returning key 22222912145551234
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:468 nrs.XCLPrefix - Leaving getKeyValue()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:468 modules.XCLLookup - table=Prod-CCE-Table, key=22222912145551234
[REQUESTI.26] INFO 2021.03.04 14:27:34:469 modules.XCLLookup - table is Prod-CCE-Table
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:469 routingtables.RoutingTable - Entering lookup()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:469 routingtables.RoutingTable - Looking up 22222912145551234 in table Prod-CCE-Table with rule prefix and modifiers=none
...
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 loadbalancer.LBBase - Entering getServer()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 loadbalancer.LBBase - Entering initializeDomains()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 servergroups.ServerGlobalStateWrapper - Prod-CCE:{Gateway}:5060:2 numTries=1--->isServerAvailable(): true
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 servergroups.AbstractNextHop - Entering compareDomainNames()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 servergroups.AbstractNextHop - Leaving compareDomainNames()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 servergroups.ServerGlobalStateWrapper - Prod-CCE:{Gateway0}:5060:2 numTries=1--->isServerAvailable(): true
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 loadbalancer.LBBase - Leaving initializeDomains()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:471 servergroups.AbstractNextHop - Entering compareDomainNames()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:471 servergroups.AbstractNextHop - Leaving compareDomainNames()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:471 loadbalancer.LBBase - Server group dc-dialer.fqdn.tld selected {reSgElementQValue=1.0, reSgElementHost={Gateway}, reSgElementTransport=TCP, reSgElementPort=5060, reSgElementWeight=100, reSgElementSgName=dc-dialer.fqdn.tld}
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:471 loadbalancer.LBBase - Leaving getServer()

3. The Gateway places the call (ccapi inout and ccsip mess are the debugs you need to enable to get relevant information. Biggest gotcha are dial peers not matching.)

Gateway:
2131549: Mar 4 10:55:33.260: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received: 
INVITE sip:22222912145551234@dc-dialer.fqdn.tld SIP/2.0
Via: SIP/2.0/TCP {CUSP}:5060;branch=z9hG4bKPpG+UtfThNEwGF7BsjXL3Q~~23121513
Via: SIP/2.0/UDP {MRPG}:58800;branch=z9hG4bK-d8754z-400d871379428a5a-1---d8754z-;rport=58800
Max-Forwards: 69
To: <sip:22222912145551234@{CUSP}>
From: <sip:5551412012@{MRPG}>;tag=0e52a576
Contact: <sip:5551412012@{MRPG}:58800>
Require: 100rel
Remote-Party-ID: <sip:18885461234@{CUSP}>;party=calling;screen=no;privacy=off
Call-ID: 3d5bfc40-093d7432-4a122a63-e2664942
CSeq: 1 INVITE
Content-Length: 630
Session-Expires: 1800
Min-SE: 90
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, UPDATE, NOTIFY, PRACK, REFER, NOTIFY, OPTIONS
Content-Type: Multipart/mixed;boundary=uniqueBoundary
Supported: timer, resource-priority, replaces
User-Agent: Cisco-SIPDialer/UCCE10.0

2132806: Mar 4 10:55:36.362: //16876944/44B8C2C0A6A6/SIP/Msg/ccsipDisplayMsg:
Received: 
SIP/2.0 180 Ringing
Via: SIP/2.0/UDP {PSTN}:5060;branch=z9hG4bK72D75B260F
From: <sip:18885461234@{PSTN}>;tag=4E815A86-1116
To: <sip:12145551234@{PSTN}>;tag=gK02b0b650
Call-ID: 44B9FBA8-7C4111EB-A6ACFA5D-15A1A7BA@{PSTN}
CSeq: 101 INVITE
Contact: <sip:12145551234@{PSTN}:5060>
Allow: INVITE,ACK,CANCEL,BYE,UPDATE
Content-Length: 236
Content-Disposition: session; handling=required
Content-Type: application/sdp

4. Voice Gateway does Call Progress Analysis and detects an answering machine. The Dialer is notified (I don’t think the above gateway debug levels will show you CPA information so I was not able to capture CPA from the gateway.)

baDialer:
10:58:41:845 dialer-baDialer Trace: (DDist) Softphone connection event: phone: [22222912145551234], result: [VOICE], port: [016], state: [DIAL_CUSTOMER]. 
10:58:41:845 dialer-baDialer Trace: (Customer) SetCallResults(): ID: [-2147483599 in DL_5008_5031], skill: [6924], result: [10] [VOICE], now: [Thu Mar 4 10:58:41 2021], callback: [Thu Mar 4 10:58:33 2021]. 
10:58:41:845 dialer-baDialer Trace: (IVR) Received telephony event port: [016], connection state: [20].

5. The Dialer asks the MR PG where the IVR is

baDialer:
10:58:41:845 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [TRANSFER]. 
10:58:41:845 dialer-baDialer Trace: (IVR) Transferring Customer port: [016], to IVR route point: [6515555678]. 
10:58:41:845 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [GET_TARGET].

6. MR PG forwards the request to the Router (It is important to note that up to this point everything that was happening was outside of the central controller.)

PIM:
14:23:09:556 PG2A-pim1 Trace: VRU->PG:REQUEST_INSTRUCTION(172 bytes):DID=570876 SendSeq#=1 TrkGrpID=200 TrkNum=1 SrvID=2 ANI=12145551234 DNIS=666666666610241 CorrID=10241 CallGUID=410885C97C5E11EBAABBFA5D15A1A7BA PstnTrkGrpID={Gateway} PstnTrkGrpChann#=2147483647 SIPHeader=f:<sip:12145551234@{Gateway}>;tag=4F3F6BE6-173D 
14:23:09:556 PG2A-pim1 Trace: FromVRU_RequestInstruction:REQUEST_INSTRUCTION RCID=5001 PID=5001 DID=570876 DIDRelSeq#=1 CorrID=10241 CalledParty#= CallingParty#=12145551234 CallGUID=410885C97C5E11EBAABBFA5D15A1A7BA PstnTrkGrp(ID={Gateway} ,Chann#=2147483647) SIPHeader=f:<sip:12145551234@{Gateway}>;tag=4F3F6BE6-173D 
14:23:09:556 PG2A-pim1 Trace: ProcessConnect:CONNECT RCID=5001 PID=5001 DID=570876 DIDRelSeq#=0 CRS(RtrDate=153464,RtrCID=18384) RCKSeq#=0 ErrorCode=0 TRTargetID=-1 CorrID=10242 EventSel=119 SvcType=4 NICCallID={PCID=5001,RCID=5001,Remote=0,0,DID=0x8b5fc,RemDID=0x0,Grp=0,Data=0,RtrData=0,CCID=x00000001/x00000000} PGCallID={N/A} OperationCode=CLASSIC OperationFlags=COOP_NONE NetworkTransferEnabled=F ECCPayloadID=1 Label(Type=8)=8888888881 NICCalledParty#=6515555678 SGSTID=-1 PQID=-1 SvcSTID=-1 AGSTID=-1 AGInfo=, MRDID=0 Interruptible=0 CallGUID=410885C97C5E11EBAABBFA5D15A1A7BA SIPHeader=f:<sip:12145551234@{Gateway}>;tag=4F3F6BE6-173D
rtr:
14:23:09:446 ra-rtr Trace: (1741 x 0 : 0 0) NewCall: CID=(153464,18384), DN=6515555678, ANI=2145551234, CED=, RCID=5003, MRDID=1, CallAtVRU=1, OpCode=0

7. Routing Script identifies the IVR and notifies the MR PG. (The script used here is the Hello World CVP script. Note that at this point we’re working with one corrID, but when the call goes to the IVR we will have a second corrID.)

rtr:
14:23:09:446 ra-rtr Trace: Script-Execute CID=(153464,18384) Default\\ZZZ_HelloIVR Start 1 
14:23:09:446 ra-rtr Trace: Script-Execute CID=(153464,18384) Default\\ZZZ_HelloIVR Set Variable 2 
14:23:09:446 ra-rtr Trace: Script-Execute CID=(153464,18384) Default\\ZZZ_HelloWorld Send To VRU 6 
14:23:09:446 ra-rtr Trace: (1741 x 0 : 0 0) Customer (1) has no valid network vru defined - using default. 
14:23:09:446 ra-rtr Trace: (1741 x 0 : 0 0) Customer (1) has no valid network vru defined - using default. 
14:23:09:446 ra-rtr Trace: (1741 x 0 : 0 0) Correlation id for dialog is (10241). 
14:23:09:446 ra-rtr Trace: (1741 x 10241 : 0 0) TransferToVRU: Label=6666666666, CorID=10241, VRUID=5000, RCID=5003 ECCPayloadID=1 
14:23:09:446 ra-rtr Trace: (1741 x 10241 : 0 0) TransferConnect sent. Dialog pending.
PIM:
14:23:09:556 PG2A-pim1 Trace: PG->VRU:TEMPORARY_CONNECT(214 bytes):DID=570876 SendSeq#=1 Label=8888888881 CorrID=10242 RCK=18384 RCKDay=153464 RCKSeq#=0 CallGUID=410885C97C5E11EBAABBFA5D15A1A7BA SIPHeader=f:<sip:12145551234@{Gateway}>;tag=4F3F6BE6-173D

8. The MR PG forwards the route response to the Dialer

baDialer:
10:58:41:877 dialer-baDialer Trace: (IVR) MR target acqusition succeeded for port: [016], state: [GET_TARGET], target: [666666666610241]. 
10:58:41:877 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [TRANSFER]. 
10:58:41:877 dialer-baDialer Trace: (SIPDisp) Transfer, port: [016], phone: [666666666610241].

9. The Dialer notifies the voice gateway to transfer the call to the IVR

baDialer:
10:55:42:249 dialer-baDialer Trace: (RESIP) Adding message to tx buffer to: [ V4 {Gateway}:5060 UDP target domain={Gateway} mFlowKey=0 ] 
10:55:42:265 dialer-baDialer Trace: (IVR) MR target acqusition succeeded for port: [016], state: [GET_TARGET], target: [666666666610241]. 
10:55:42:265 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [TRANSFER]. 
10:55:42:265 dialer-baDialer Trace: (SIPDisp) Transfer, port: [016], phone: [666666666610241]. 
10:55:42:265 dialer-baDialer Trace: (CLMGR) Agent event, agent: [111100208], ext: [5551510241], state: [TALKING]. 
10:55:42:265 dialer-baDialer Trace: (CLMGR_SIP) tOnBeginCallEvent(): Port: [003], ID: [38914319], device ID: [5551510241], IsReservationPort: [No]. 
10:55:42:265 dialer-baDialer Trace: (CLMGR) Agent event, agent: [111100208], ext: [5551510241], state: [TALKING]. 
10:55:42:281 dialer-baDialer Trace: (RESIP) Dialog::makeRequest: 
...
10:55:42:281 dialer-baDialer Trace: (RESIP) SEND: 
REFER sip:22222912145551234@{Gateway}:5060 SIP/2.0
Via: SIP/2.0/ ;branch=z9hG4bK-d8754z-5178e07be820b14e-1---d8754z-;rport
Max-Forwards: 70
Contact: <sip:5551412012>
To: <sip:22222912145551234@{CUSP}>;tag=4E816590-135
From: <sip:5551412012@{MRPG}>;tag=0e52a576
Call-ID: 3d5bfc40-093d7432-4a122a63-e2664942
CSeq: 3 REFER
User-Agent: Cisco-SIPDialer/UCCE10.0
Refer-To: <sip:666666666610241@{CUSP}>
Referred-By: <sip:5551412012@{MRPG}>
Content-Length: 0

10 The Gateway initiates the transfer to the SIP Proxy, and the SIP Proxy forwards the invitation onto Unified CVP.

Gateway:
2134751: Mar 4 10:55:42.320: //16876944/44B8C2C0A6A6/CCAPI/ccCheckClipClir:
In: Calling Number=12145551234(TON=Unknown, NPI=Unknown, Screening=Not Screened, Presentation=Allowed)
2134752: Mar 4 10:55:42.320: //16876944/44B8C2C0A6A6/CCAPI/ccCheckClipClir:
Out: Calling Number=12145551234(TON=Unknown, NPI=Unknown, Screening=Not Screened, Presentation=Allowed)
2134753: Mar 4 10:55:42.321: //16876944/44B8C2C0A6A6/CCAPI/ccCallSetupRequest:
Destination Pattern=6666666666....., Called Number=11102666666666610241, Digit Strip=FALSE
2134754: Mar 4 10:55:42.321: //16876944/44B8C2C0A6A6/CCAPI/ccCallSetupRequest:
Calling Number=12145551234(TON=Unknown, NPI=Unknown, Screening=Not Screened, Presentation=Allowed),
Called Number=11102666666666619114(TON=Unknown, NPI=Unknown),
Redirect Number=, Display Info=
Account Number=18885461234, Final Destination Flag=TRUE,
Guid=44B8C2C0-7C41-11EB-A6A6-FA5D15A1A7BA, Outgoing Dial-peer=2208
CUSP:
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 nrs.FieldSelector - getUriPart: URI - sip:11102666666666610241@{CUSP} part 6
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 nrs.FieldSelector - Requested field 45
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 nrs.FieldSelector - Returning key 11102666666666610241
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 nrs.XCLPrefix - Leaving getKeyValue()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 modules.XCLLookup - table=Prod-CCE-Table, key=11102666666666610241
[REQUESTI.7] INFO 2021.03.04 14:27:39:643 modules.XCLLookup - table is Prod-CCE-Table
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 routingtables.RoutingTable - Entering lookup()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 routingtables.RoutingTable - Looking up 11102666666666610241 in table Prod-CCE-Table with rule prefix and modifiers=none
...
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Entering getServer()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Entering initializeDomains()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 nrs.NRSRoutes - routes before applying time policies: [Ruri: dc1-cvp.fqdn.tld, Route: null, Network: Prod-CCE, q-value=1.0radvance=[502, 503]]
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 nrs.NRSRoutes - routes after applying time policies: [Ruri: dc1-cvp.fqdn.tld, Route: null, Network: Prod-CCE, q-value=1.0radvance=[502, 503]]
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Leaving initializeDomains()
[REQUESTI.7] INFO 2021.03.04 14:27:39:644 loadbalancer.LBHashBased - list of elements in order on which load balancing is done : Ruri: dc1-cvp.fqdn.tld, Route: null, Network: Prod-CCE, q-value=1.0radvance=[502, 503], 
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Server group route-sg selected Ruri: dc1-cvp.fqdn.tld, Route: null, Network: Prod-CCE, q-value=1.0radvance=[502, 503]
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Leaving getServer()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 nrs.XCLNRSShiftRoutes - Leaving ShiftRoutes.execute()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBFactory - Entering createLoadBalancer()
[REQUESTI.7] INFO 2021.03.04 14:27:39:644 loadbalancer.LBFactory - lbtype is 5(weight)
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBFactory - Leaving createLoadBalancer()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Entering getServer
rtr:
07:55:48:615 ra-rtr Trace: (1743 x 10241 : 0 0) TransferToVRU: Label=6666666666, CorID=10241, VRUID=5000, RCID=5003 ECCPayloadID=1 
07:55:48:615 ra-rtr Trace: (1743 x 10241 : 0 0) TransferConnect sent. Dialog pending. 
07:55:48:709 ra-rtr Trace: (1743 585271 10241 : 0 0) RequestInstr: CID=(153465,18384), CallState=1 
07:55:48:709 ra-rtr Trace: (585271 585271 10241 : 0 0) Dialog initiating 2nd phase of transfer. 
07:55:48:709 ra-rtr Trace: (585271 585271 10241 : 0 0) Correlation id for dialog is (10242). 
07:55:48:709 ra-rtr Trace: (585271 585271 10242 : 0 0) TransferToVRU: Label=8888888882, CorID=10242, VRUID=5000, RCID=5006 ECCPayloadID=1 
07:55:48:709 ra-rtr Trace: (585271 585271 10242 : 0 0) TransferConnect sent. Dialog pending. 
07:55:48:802 ra-rtr Trace: (585271 585272 10242 : 0 0) RequestInstr: CID=(153465,18384), CallState=1 
07:55:48:802 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog resuming (Request Instruction received.) status (0) 
07:55:48:802 ra-rtr Trace: Script-Continue CID=(153465,18384) Default\\ZZZ_HelloIVR Send To VRU 6 
07:55:48:802 ra-rtr Trace: Script-Execute CID=(153465,18384) Default\\ZZZ_HelloIVR Set Variable 7 
07:55:48:802 ra-rtr Trace: Script-Execute CID=(153465,18384) Default\\ZZZ_HelloIVR Run External Script 8 
07:55:48:802 ra-rtr Trace: (585271 585272 10242 : 0 0) Skipping the VRU verification because of Peripheral's ClientType is DBCT_MEDIA_ROUTING 
07:55:48:802 ra-rtr Trace: (585271 585272 10242 : 0 0) Runscript sent. ECCPayloadID = 1 Dialog pending. 
07:55:54:366 ra-rtr Trace: (585271 585272 10242 : 0 0) CallEventReport: CID=(153465,18384),Event=DISCONNECT, DlgEnds=1, FromVRU=0, CallState=2, Cause=NORMAL 
07:55:54:366 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog received event report 6 from NIC during RunScript. 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) CallEventReport: CID=(153465,18384),Event=DISCONNECT, DlgEnds=1, FromVRU=0, CallState=22, Cause=NORMAL 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog (callstate:22) received event(6)(Call disconnected. (Event has dialog end set.)) 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog resuming (Call disconnected. (Event has dialog end set.)) status (3) 
07:55:54:570 ra-rtr Trace: Script-Continue CID=(153465,18384) Default\\ZZZ_HelloIVR Run External Script 8 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog aborted and was deleted. 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog sending release call to VRU 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) Deleting Dialog.

From this point forward it’s just an inbound CVP call.

~david

by dmacias

VMware Fusion 12 and Big Sur

As many people are slowly upgrading to the latest MacOS version, Big Sur, there are a few of us who are running into some compatibility issues. For me the only one that has given me some grief is VMWare Fusion. Many in my industry run multiple VMs to be able to connect to various customer or just to be able to use some of the Cisco tools which do not support any other operation system other than Cisco. There are two issues which I needed to fix in order to get one of my VMs back. First, the error below “VMware Fusion does not support virtualized performance counters on this host.” This error prevented the VM from booting up.

VMware Fusion Performance Counters Error

VMware Fusion Performance Counters Error

This can be resolved by going to Settings > Processor & Memory > Advanced options and unchecking “Enable code profiling applications…”

VMware Fusion Processor & Memory Settings

VMware Fusion Processor & Memory Settings

The second error was “You are running this virtual machine with side channel mitigations enabled.” This error could be ignored and didn’t seem to affect the VM at all, but it was still annoying. This kb had all I needed to get that message out of the way.

~david

 

2020 Cisco Forums Profile
by dmacias

It’s nice to be recognized

I got a nice surprise in my inbox today. An email from Cisco letting me know that I was the first ever winner of the English Community Developer of the Month. Per Cisco the Community Spotlight Awards:

… recognizes members whose significant contributions designate leadership and commitment to their peers within their respective communities, including the Cisco Learning Network (CLN) and Cisco Community. Spotlight Awards Program is designed to recognize and thank individuals who help make our communities the premier online destination for Cisco enthusiasts.

I get a cool badge to show off too.

2020 Cisco Forums Profile

2020 Cisco Forums Profile

You can find current and past winners here or try to spot me in the picture below.

Current Spotlight Winners

Current Spotlight Winners

Looking back through my blog posts in 2008 I talked about getting my first star due to my contributions in the Cisco NetPro forums and how happy I was about it. In that blog I have a picture of my profile showing a total of 103 posts made with 8 questions resolved. That number, 12 years later, has ballooned to 3030 posts and 208 solutions.

I encourage anyone starting out or a seasoned veteran to contribute in the various Cisco Communities it’s a great way to network with your Cisco peers and try to tackle some very interesting technical problems while you procrastinate from your not as interesting technical problems.

~david