Category / Writings

by dmacias

Microsoft and Nuance a Brilliant Play for Redmond

Bloomberg reports that Microsoft “is in advanced talks” to purchase Nuance. I’ve not been able to stop thinking of this move and I’ll be the first to admit that it surprised me, but the more I think about it and the more I talk to others in the industry this is an absolutely brilliant move. Here are my crystal ball predictions:

The patent play: Nuance comes with over 2000 patents. This is a huge cache which will no doubt be useful for the upcoming AI wars. This will be a huge boosts to Microsoft’s already huge R&D commitment in this space.

This hospital bill is brought to you by Microsoft: Nuance makes the majority of their money from the healthcare sector. We’re not talking just dictations or document management, we’re talking EHR, billing, and diagnosis software. Windows and Office are already prevalent in the healthcare space, this puts MS in the heart (get it?) of hospital operations and processes.

Cortana, it was the best of times it was the worst of times: Did you know that Windows 10 was Cortana’s big debut in the desktop space? Yeah neither did anyone else. Cortana began in 2014 as a direct competition to Alexa which was released the year before. At the time Microsoft was beginning to make some heavy bets in to the mobile space with Windows mobile. Well it’s a decade later and Windows mobile is dead, Cortana’s OS integration has been neutered and I’ve never seen another human being speak to their Windows PC. I believe this is going to change that with a huge marketing push of some college student dictating their final paper to their Microsoft Surface device while getting a manicure.

Where we’re going we don’t need passwords: Nuance comes with perhaps one of the oldest if not best speech biometrics software. Imagine joining a Microsoft Teams meetings where you call in and start speaking and you’re authenticated immediately. Or allow for “signatures” based solely on your voice. Verification and authentication continue to be huge and the rise of better and better “deep fake” technology will allow for some sort of trust verification service with Nuance biometrics in the middle of it.

(Part 1) We’re taking our ball and going home: This one is near and dear to me heart. If a Cisco call center customer wanted to have speech recognition or text to speech there was only one name in town. Nuance. This has changed a bit in the last few years with the introduction of LumenVox as an additional option. And this has changed even further in the last 18 months with Cisco supporting Google’s DialogFlow, but Nuance still reigns supreme. I can see MS increasing the pricing of an already very expensive product making it prohibitive for some call centers to run their software.

(Part 2) I can see clearly now the rain is gone: Did you know that Azure stands for the color blue of a cloudless sky? Microsoft will be able to create a very defensible moat around their Azure offerings by being the only provider to have the latest and greatest Nuance services. In addition, some telephony cloud provider, who are already battling Amazon and who white label Nuance products as part of their offering, might be forced to either consume more Azure resources to get better pricing or completely get priced out from this technology and watch the competition pass them by. This sets the stage for Microsoft to be in the driver’s seat of what UC or CC provider you might choose next if you have an already deep Nuance integration or if your call center must use Nuance.

 

 

 

 

 

 

by dmacias

What “Zoom Fatigue” can teach us about using video in the contact center

My significant other researches human behavior at work and she brought this study to my attention. She thought it would be very relevant to what I do. I want to summarize some key points and how they relate to the contact center, but first, I really wanted to title this post as “Why video will never kill the phone contact center star”, but that seemed too childish.

So what can Zoom Fatigue teach us about using video in the call center? Let me highlight a few key points from the article.

…in one-on-one meetings conducted over Zoom, coworkers and friends are maintaining an interpersonal distance reserved for loved ones.

Is your customer relationship what you would consider intimate? I can’t think of a single service or product where I would use that term, so the answer is more than likely no. Now, imagine having to handle customer video calls all day and feeling your personal space invaded. It would be exhausting for agents and off-putting for customers. Video calls should be reserved for customers with a long-standing relationship and limited to a few key agents who know the customer well. Additionally, considerations should be made around how many video calls an agent should handle in a short amount of time.

One of the remarkable aspects of early work on nonverbal synchrony (i.e., Kendon, 1970) is how nonverbal behavior is simultaneously effortless and incredibly complex. On Zoom, nonverbal behavior remains complex, but users need to work harder to send and receive signals.

Processing these extra nonverbal signals contributes to what my significant other and other researchers call “cognitive load” or the amount of information our brains can process at any given time. Video calls divert precious mental resources away from the task at hand, making it more likely your agents will make mistakes on complex tasks like financial services or billing. Traditional audio-only phone calls enable them to focus better while doing their work.

There is no data on the effects of viewing oneself for many hours per day. Given past work, it is likely that a constant “mirror” on Zoom causes self-evaluation and negative affect.

Self-view is very distracting for me, however that’s the only way for me to know if I’m in view or not as I use a standing desk. In addition to being distracting, this article argues that it is also stressful. If your agents are handling video calls, consider the ability to turn on and off self-view. Vendors should come up with a technology solution which notify the users when they are out of view without relying on the equivalent of looking in the mirror all day long.

…cultural norms are to stay centered within the camera’s view frustrum and to keep one’s face large enough for others to see. In essence users are stuck in a very small physical cone, and most of the time this equates to sitting down and staring straight ahead.

We first had handsets and they were terrible to hold and work at the same time. We then got wired headsets and life was better, but we needed to make sure not to get tangled or have someone kick them. We then moved on to wireless headsets and we got freedom! … only to have it taken away by video that creates a “lock in” effect. You can no longer just stand up and stretch. You can’t just run to the fax machine or to refill your water bottle. If your agents collaborate in a team to handle customer requests or handle calls which can be very lengthy avoid having them on camera.

~david

UCCE Call Flow Outbound Option with SIP Proxy
by dmacias

Cisco UCCE Outbound Dialer IVR Campaign Log Walkthrough

I wanted to document this specifically as I feel it’s the most complex flow you will see within UCCE. Before you go through this you will absolutely want to get familiar with the documentation as there are a lot of moving pieces. The diagram below comes straight from said documentation and should help you visualize what you see the logs doing.

 

UCCE Call Flow Outbound Option with SIP Proxy

UCCE Call Flow Outbound Option with SIP Proxy

Ideally I would have had all the logs for all the devices at the same time, but unfortunately that wasn’t the case. You will notice that the timestamps jump around as some logs are from other time frames. However, I’ve tried to match up all unique identifiers across all the logs so you can follow it along. The unique identifiers you want to take note of are:

2145551234: Customer phone number campaign is going to dial.
016: The dialer port.
10241: Correlation ID of initial call to customer.
10242: Correlation ID of call to IVR.

1. An unattended IVR campaign starts. Customer records are delivered to the Dialer. (The ANI and the port are key to be able to trace what the dialer is doing.)

badialer:
10:55:33:059 dialer-baDialer Trace: (Customer) SetCallResults(): ID: [-2147483601 in DL_5008_5031], skill: [6924], result: [1] [DIAL_RESERVED], now: [Thu Mar 4 10:55:33 2021], callback: [Thu Mar 4 10:55:32 2021]. 
10:55:33:059 dialer-baDialer Trace: (CUSTMGR) SendRecord(): Send customer: [2145551234] record ID: [-2147483601], in: DL_5008_5031 to port: [016]. 
10:55:33:059 dialer-baDialer Trace: (IVR) Record available event, port: [016], phone: [2145551234]. 
10:55:33:059 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [PORT_DEVICE_ATTRIB].

2. The Dialer asks the SIP Proxy to forward an invite to an available gateway to start a call. (CUSP logs are a bit of a pain, I recommend you download them via FTP instead of doing it from the GUI. This is a great guide on log settings. Ultimately, here we want to make sure that the CUSP knows where to send the request.)

badialer:
10:55:33:059 dialer-baDialer Trace: (SIPDisp) Dial, port: [016], phone: [22222912145551234], lDialTimeoutSec: [32]
CPA parameters: AP: [2500], MinSP: [608], MinVSpeech: [112], MaxTA: [5000], MaxTone: [30000]. 
10:55:33:059 dialer-baDialer Trace: (IVR) Dialing, phone: [22222912145551234], port: [016], ring timeout: [32], state: [PORT_DEVICE_ATTRIB]. 
10:55:33:059 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [DIAL_CUSTOMER].
CUSP:
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:468 nrs.FieldSelector - getUriPart: URI - sip:22222912145551234@{CUSP} part 6
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:468 nrs.FieldSelector - Requested field 45
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:468 nrs.FieldSelector - Returning key 22222912145551234
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:468 nrs.XCLPrefix - Leaving getKeyValue()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:468 modules.XCLLookup - table=Prod-CCE-Table, key=22222912145551234
[REQUESTI.26] INFO 2021.03.04 14:27:34:469 modules.XCLLookup - table is Prod-CCE-Table
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:469 routingtables.RoutingTable - Entering lookup()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:469 routingtables.RoutingTable - Looking up 22222912145551234 in table Prod-CCE-Table with rule prefix and modifiers=none
...
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 loadbalancer.LBBase - Entering getServer()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 loadbalancer.LBBase - Entering initializeDomains()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 servergroups.ServerGlobalStateWrapper - Prod-CCE:{Gateway}:5060:2 numTries=1--->isServerAvailable(): true
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 servergroups.AbstractNextHop - Entering compareDomainNames()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 servergroups.AbstractNextHop - Leaving compareDomainNames()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 servergroups.ServerGlobalStateWrapper - Prod-CCE:{Gateway0}:5060:2 numTries=1--->isServerAvailable(): true
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:470 loadbalancer.LBBase - Leaving initializeDomains()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:471 servergroups.AbstractNextHop - Entering compareDomainNames()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:471 servergroups.AbstractNextHop - Leaving compareDomainNames()
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:471 loadbalancer.LBBase - Server group dc-dialer.fqdn.tld selected {reSgElementQValue=1.0, reSgElementHost={Gateway}, reSgElementTransport=TCP, reSgElementPort=5060, reSgElementWeight=100, reSgElementSgName=dc-dialer.fqdn.tld}
[REQUESTI.26] DEBUG 2021.03.04 14:27:34:471 loadbalancer.LBBase - Leaving getServer()

3. The Gateway places the call (ccapi inout and ccsip mess are the debugs you need to enable to get relevant information. Biggest gotcha are dial peers not matching.)

Gateway:
2131549: Mar 4 10:55:33.260: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received: 
INVITE sip:22222912145551234@dc-dialer.fqdn.tld SIP/2.0
Via: SIP/2.0/TCP {CUSP}:5060;branch=z9hG4bKPpG+UtfThNEwGF7BsjXL3Q~~23121513
Via: SIP/2.0/UDP {MRPG}:58800;branch=z9hG4bK-d8754z-400d871379428a5a-1---d8754z-;rport=58800
Max-Forwards: 69
To: <sip:22222912145551234@{CUSP}>
From: <sip:5551412012@{MRPG}>;tag=0e52a576
Contact: <sip:5551412012@{MRPG}:58800>
Require: 100rel
Remote-Party-ID: <sip:18885461234@{CUSP}>;party=calling;screen=no;privacy=off
Call-ID: 3d5bfc40-093d7432-4a122a63-e2664942
CSeq: 1 INVITE
Content-Length: 630
Session-Expires: 1800
Min-SE: 90
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, UPDATE, NOTIFY, PRACK, REFER, NOTIFY, OPTIONS
Content-Type: Multipart/mixed;boundary=uniqueBoundary
Supported: timer, resource-priority, replaces
User-Agent: Cisco-SIPDialer/UCCE10.0

2132806: Mar 4 10:55:36.362: //16876944/44B8C2C0A6A6/SIP/Msg/ccsipDisplayMsg:
Received: 
SIP/2.0 180 Ringing
Via: SIP/2.0/UDP {PSTN}:5060;branch=z9hG4bK72D75B260F
From: <sip:18885461234@{PSTN}>;tag=4E815A86-1116
To: <sip:12145551234@{PSTN}>;tag=gK02b0b650
Call-ID: 44B9FBA8-7C4111EB-A6ACFA5D-15A1A7BA@{PSTN}
CSeq: 101 INVITE
Contact: <sip:12145551234@{PSTN}:5060>
Allow: INVITE,ACK,CANCEL,BYE,UPDATE
Content-Length: 236
Content-Disposition: session; handling=required
Content-Type: application/sdp

4. Voice Gateway does Call Progress Analysis and detects an answering machine. The Dialer is notified (I don’t think the above gateway debug levels will show you CPA information so I was not able to capture CPA from the gateway.)

baDialer:
10:58:41:845 dialer-baDialer Trace: (DDist) Softphone connection event: phone: [22222912145551234], result: [VOICE], port: [016], state: [DIAL_CUSTOMER]. 
10:58:41:845 dialer-baDialer Trace: (Customer) SetCallResults(): ID: [-2147483599 in DL_5008_5031], skill: [6924], result: [10] [VOICE], now: [Thu Mar 4 10:58:41 2021], callback: [Thu Mar 4 10:58:33 2021]. 
10:58:41:845 dialer-baDialer Trace: (IVR) Received telephony event port: [016], connection state: [20].

5. The Dialer asks the MR PG where the IVR is

baDialer:
10:58:41:845 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [TRANSFER]. 
10:58:41:845 dialer-baDialer Trace: (IVR) Transferring Customer port: [016], to IVR route point: [6515555678]. 
10:58:41:845 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [GET_TARGET].

6. MR PG forwards the request to the Router (It is important to note that up to this point everything that was happening was outside of the central controller.)

PIM:
14:23:09:556 PG2A-pim1 Trace: VRU->PG:REQUEST_INSTRUCTION(172 bytes):DID=570876 SendSeq#=1 TrkGrpID=200 TrkNum=1 SrvID=2 ANI=12145551234 DNIS=666666666610241 CorrID=10241 CallGUID=410885C97C5E11EBAABBFA5D15A1A7BA PstnTrkGrpID={Gateway} PstnTrkGrpChann#=2147483647 SIPHeader=f:<sip:12145551234@{Gateway}>;tag=4F3F6BE6-173D 
14:23:09:556 PG2A-pim1 Trace: FromVRU_RequestInstruction:REQUEST_INSTRUCTION RCID=5001 PID=5001 DID=570876 DIDRelSeq#=1 CorrID=10241 CalledParty#= CallingParty#=12145551234 CallGUID=410885C97C5E11EBAABBFA5D15A1A7BA PstnTrkGrp(ID={Gateway} ,Chann#=2147483647) SIPHeader=f:<sip:12145551234@{Gateway}>;tag=4F3F6BE6-173D 
14:23:09:556 PG2A-pim1 Trace: ProcessConnect:CONNECT RCID=5001 PID=5001 DID=570876 DIDRelSeq#=0 CRS(RtrDate=153464,RtrCID=18384) RCKSeq#=0 ErrorCode=0 TRTargetID=-1 CorrID=10242 EventSel=119 SvcType=4 NICCallID={PCID=5001,RCID=5001,Remote=0,0,DID=0x8b5fc,RemDID=0x0,Grp=0,Data=0,RtrData=0,CCID=x00000001/x00000000} PGCallID={N/A} OperationCode=CLASSIC OperationFlags=COOP_NONE NetworkTransferEnabled=F ECCPayloadID=1 Label(Type=8)=8888888881 NICCalledParty#=6515555678 SGSTID=-1 PQID=-1 SvcSTID=-1 AGSTID=-1 AGInfo=, MRDID=0 Interruptible=0 CallGUID=410885C97C5E11EBAABBFA5D15A1A7BA SIPHeader=f:<sip:12145551234@{Gateway}>;tag=4F3F6BE6-173D
rtr:
14:23:09:446 ra-rtr Trace: (1741 x 0 : 0 0) NewCall: CID=(153464,18384), DN=6515555678, ANI=2145551234, CED=, RCID=5003, MRDID=1, CallAtVRU=1, OpCode=0

7. Routing Script identifies the IVR and notifies the MR PG. (The script used here is the Hello World CVP script. Note that at this point we’re working with one corrID, but when the call goes to the IVR we will have a second corrID.)

rtr:
14:23:09:446 ra-rtr Trace: Script-Execute CID=(153464,18384) Default\\ZZZ_HelloIVR Start 1 
14:23:09:446 ra-rtr Trace: Script-Execute CID=(153464,18384) Default\\ZZZ_HelloIVR Set Variable 2 
14:23:09:446 ra-rtr Trace: Script-Execute CID=(153464,18384) Default\\ZZZ_HelloWorld Send To VRU 6 
14:23:09:446 ra-rtr Trace: (1741 x 0 : 0 0) Customer (1) has no valid network vru defined - using default. 
14:23:09:446 ra-rtr Trace: (1741 x 0 : 0 0) Customer (1) has no valid network vru defined - using default. 
14:23:09:446 ra-rtr Trace: (1741 x 0 : 0 0) Correlation id for dialog is (10241). 
14:23:09:446 ra-rtr Trace: (1741 x 10241 : 0 0) TransferToVRU: Label=6666666666, CorID=10241, VRUID=5000, RCID=5003 ECCPayloadID=1 
14:23:09:446 ra-rtr Trace: (1741 x 10241 : 0 0) TransferConnect sent. Dialog pending.
PIM:
14:23:09:556 PG2A-pim1 Trace: PG->VRU:TEMPORARY_CONNECT(214 bytes):DID=570876 SendSeq#=1 Label=8888888881 CorrID=10242 RCK=18384 RCKDay=153464 RCKSeq#=0 CallGUID=410885C97C5E11EBAABBFA5D15A1A7BA SIPHeader=f:<sip:12145551234@{Gateway}>;tag=4F3F6BE6-173D

8. The MR PG forwards the route response to the Dialer

baDialer:
10:58:41:877 dialer-baDialer Trace: (IVR) MR target acqusition succeeded for port: [016], state: [GET_TARGET], target: [666666666610241]. 
10:58:41:877 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [TRANSFER]. 
10:58:41:877 dialer-baDialer Trace: (SIPDisp) Transfer, port: [016], phone: [666666666610241].

9. The Dialer notifies the voice gateway to transfer the call to the IVR

baDialer:
10:55:42:249 dialer-baDialer Trace: (RESIP) Adding message to tx buffer to: [ V4 {Gateway}:5060 UDP target domain={Gateway} mFlowKey=0 ] 
10:55:42:265 dialer-baDialer Trace: (IVR) MR target acqusition succeeded for port: [016], state: [GET_TARGET], target: [666666666610241]. 
10:55:42:265 dialer-baDialer Trace: (CPORT) SetState(): Port: [016], state: [TRANSFER]. 
10:55:42:265 dialer-baDialer Trace: (SIPDisp) Transfer, port: [016], phone: [666666666610241]. 
10:55:42:265 dialer-baDialer Trace: (CLMGR) Agent event, agent: [111100208], ext: [5551510241], state: [TALKING]. 
10:55:42:265 dialer-baDialer Trace: (CLMGR_SIP) tOnBeginCallEvent(): Port: [003], ID: [38914319], device ID: [5551510241], IsReservationPort: [No]. 
10:55:42:265 dialer-baDialer Trace: (CLMGR) Agent event, agent: [111100208], ext: [5551510241], state: [TALKING]. 
10:55:42:281 dialer-baDialer Trace: (RESIP) Dialog::makeRequest: 
...
10:55:42:281 dialer-baDialer Trace: (RESIP) SEND: 
REFER sip:22222912145551234@{Gateway}:5060 SIP/2.0
Via: SIP/2.0/ ;branch=z9hG4bK-d8754z-5178e07be820b14e-1---d8754z-;rport
Max-Forwards: 70
Contact: <sip:5551412012>
To: <sip:22222912145551234@{CUSP}>;tag=4E816590-135
From: <sip:5551412012@{MRPG}>;tag=0e52a576
Call-ID: 3d5bfc40-093d7432-4a122a63-e2664942
CSeq: 3 REFER
User-Agent: Cisco-SIPDialer/UCCE10.0
Refer-To: <sip:666666666610241@{CUSP}>
Referred-By: <sip:5551412012@{MRPG}>
Content-Length: 0

10 The Gateway initiates the transfer to the SIP Proxy, and the SIP Proxy forwards the invitation onto Unified CVP.

Gateway:
2134751: Mar 4 10:55:42.320: //16876944/44B8C2C0A6A6/CCAPI/ccCheckClipClir:
In: Calling Number=12145551234(TON=Unknown, NPI=Unknown, Screening=Not Screened, Presentation=Allowed)
2134752: Mar 4 10:55:42.320: //16876944/44B8C2C0A6A6/CCAPI/ccCheckClipClir:
Out: Calling Number=12145551234(TON=Unknown, NPI=Unknown, Screening=Not Screened, Presentation=Allowed)
2134753: Mar 4 10:55:42.321: //16876944/44B8C2C0A6A6/CCAPI/ccCallSetupRequest:
Destination Pattern=6666666666....., Called Number=11102666666666610241, Digit Strip=FALSE
2134754: Mar 4 10:55:42.321: //16876944/44B8C2C0A6A6/CCAPI/ccCallSetupRequest:
Calling Number=12145551234(TON=Unknown, NPI=Unknown, Screening=Not Screened, Presentation=Allowed),
Called Number=11102666666666619114(TON=Unknown, NPI=Unknown),
Redirect Number=, Display Info=
Account Number=18885461234, Final Destination Flag=TRUE,
Guid=44B8C2C0-7C41-11EB-A6A6-FA5D15A1A7BA, Outgoing Dial-peer=2208
CUSP:
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 nrs.FieldSelector - getUriPart: URI - sip:11102666666666610241@{CUSP} part 6
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 nrs.FieldSelector - Requested field 45
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 nrs.FieldSelector - Returning key 11102666666666610241
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 nrs.XCLPrefix - Leaving getKeyValue()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 modules.XCLLookup - table=Prod-CCE-Table, key=11102666666666610241
[REQUESTI.7] INFO 2021.03.04 14:27:39:643 modules.XCLLookup - table is Prod-CCE-Table
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 routingtables.RoutingTable - Entering lookup()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:643 routingtables.RoutingTable - Looking up 11102666666666610241 in table Prod-CCE-Table with rule prefix and modifiers=none
...
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Entering getServer()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Entering initializeDomains()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 nrs.NRSRoutes - routes before applying time policies: [Ruri: dc1-cvp.fqdn.tld, Route: null, Network: Prod-CCE, q-value=1.0radvance=[502, 503]]
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 nrs.NRSRoutes - routes after applying time policies: [Ruri: dc1-cvp.fqdn.tld, Route: null, Network: Prod-CCE, q-value=1.0radvance=[502, 503]]
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Leaving initializeDomains()
[REQUESTI.7] INFO 2021.03.04 14:27:39:644 loadbalancer.LBHashBased - list of elements in order on which load balancing is done : Ruri: dc1-cvp.fqdn.tld, Route: null, Network: Prod-CCE, q-value=1.0radvance=[502, 503], 
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Server group route-sg selected Ruri: dc1-cvp.fqdn.tld, Route: null, Network: Prod-CCE, q-value=1.0radvance=[502, 503]
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Leaving getServer()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 nrs.XCLNRSShiftRoutes - Leaving ShiftRoutes.execute()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBFactory - Entering createLoadBalancer()
[REQUESTI.7] INFO 2021.03.04 14:27:39:644 loadbalancer.LBFactory - lbtype is 5(weight)
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBFactory - Leaving createLoadBalancer()
[REQUESTI.7] DEBUG 2021.03.04 14:27:39:644 loadbalancer.LBBase - Entering getServer
rtr:
07:55:48:615 ra-rtr Trace: (1743 x 10241 : 0 0) TransferToVRU: Label=6666666666, CorID=10241, VRUID=5000, RCID=5003 ECCPayloadID=1 
07:55:48:615 ra-rtr Trace: (1743 x 10241 : 0 0) TransferConnect sent. Dialog pending. 
07:55:48:709 ra-rtr Trace: (1743 585271 10241 : 0 0) RequestInstr: CID=(153465,18384), CallState=1 
07:55:48:709 ra-rtr Trace: (585271 585271 10241 : 0 0) Dialog initiating 2nd phase of transfer. 
07:55:48:709 ra-rtr Trace: (585271 585271 10241 : 0 0) Correlation id for dialog is (10242). 
07:55:48:709 ra-rtr Trace: (585271 585271 10242 : 0 0) TransferToVRU: Label=8888888882, CorID=10242, VRUID=5000, RCID=5006 ECCPayloadID=1 
07:55:48:709 ra-rtr Trace: (585271 585271 10242 : 0 0) TransferConnect sent. Dialog pending. 
07:55:48:802 ra-rtr Trace: (585271 585272 10242 : 0 0) RequestInstr: CID=(153465,18384), CallState=1 
07:55:48:802 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog resuming (Request Instruction received.) status (0) 
07:55:48:802 ra-rtr Trace: Script-Continue CID=(153465,18384) Default\\ZZZ_HelloIVR Send To VRU 6 
07:55:48:802 ra-rtr Trace: Script-Execute CID=(153465,18384) Default\\ZZZ_HelloIVR Set Variable 7 
07:55:48:802 ra-rtr Trace: Script-Execute CID=(153465,18384) Default\\ZZZ_HelloIVR Run External Script 8 
07:55:48:802 ra-rtr Trace: (585271 585272 10242 : 0 0) Skipping the VRU verification because of Peripheral's ClientType is DBCT_MEDIA_ROUTING 
07:55:48:802 ra-rtr Trace: (585271 585272 10242 : 0 0) Runscript sent. ECCPayloadID = 1 Dialog pending. 
07:55:54:366 ra-rtr Trace: (585271 585272 10242 : 0 0) CallEventReport: CID=(153465,18384),Event=DISCONNECT, DlgEnds=1, FromVRU=0, CallState=2, Cause=NORMAL 
07:55:54:366 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog received event report 6 from NIC during RunScript. 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) CallEventReport: CID=(153465,18384),Event=DISCONNECT, DlgEnds=1, FromVRU=0, CallState=22, Cause=NORMAL 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog (callstate:22) received event(6)(Call disconnected. (Event has dialog end set.)) 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog resuming (Call disconnected. (Event has dialog end set.)) status (3) 
07:55:54:570 ra-rtr Trace: Script-Continue CID=(153465,18384) Default\\ZZZ_HelloIVR Run External Script 8 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog aborted and was deleted. 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) Dialog sending release call to VRU 
07:55:54:570 ra-rtr Trace: (585271 585272 10242 : 0 0) Deleting Dialog.

From this point forward it’s just an inbound CVP call.

~david

by dmacias

VMware Fusion 12 and Big Sur

As many people are slowly upgrading to the latest MacOS version, Big Sur, there are a few of us who are running into some compatibility issues. For me the only one that has given me some grief is VMWare Fusion. Many in my industry run multiple VMs to be able to connect to various customer or just to be able to use some of the Cisco tools which do not support any other operation system other than Cisco. There are two issues which I needed to fix in order to get one of my VMs back. First, the error below “VMware Fusion does not support virtualized performance counters on this host.” This error prevented the VM from booting up.

VMware Fusion Performance Counters Error

VMware Fusion Performance Counters Error

This can be resolved by going to Settings > Processor & Memory > Advanced options and unchecking “Enable code profiling applications…”

VMware Fusion Processor & Memory Settings

VMware Fusion Processor & Memory Settings

The second error was “You are running this virtual machine with side channel mitigations enabled.” This error could be ignored and didn’t seem to affect the VM at all, but it was still annoying. This kb had all I needed to get that message out of the way.

~david

 

2020 Cisco Forums Profile
by dmacias

It’s nice to be recognized

I got a nice surprise in my inbox today. An email from Cisco letting me know that I was the first ever winner of the English Community Developer of the Month. Per Cisco the Community Spotlight Awards:

… recognizes members whose significant contributions designate leadership and commitment to their peers within their respective communities, including the Cisco Learning Network (CLN) and Cisco Community. Spotlight Awards Program is designed to recognize and thank individuals who help make our communities the premier online destination for Cisco enthusiasts.

I get a cool badge to show off too.

2020 Cisco Forums Profile

2020 Cisco Forums Profile

You can find current and past winners here or try to spot me in the picture below.

Current Spotlight Winners

Current Spotlight Winners

Looking back through my blog posts in 2008 I talked about getting my first star due to my contributions in the Cisco NetPro forums and how happy I was about it. In that blog I have a picture of my profile showing a total of 103 posts made with 8 questions resolved. That number, 12 years later, has ballooned to 3030 posts and 208 solutions.

I encourage anyone starting out or a seasoned veteran to contribute in the various Cisco Communities it’s a great way to network with your Cisco peers and try to tackle some very interesting technical problems while you procrastinate from your not as interesting technical problems.

~david

Firebase default Hosting webpage
by dmacias

Serverless Development with Firebase Emulator

I’m getting more and more into serverless development. Trying to avoid handling any sort of hardware sounds like a dream come true. No more handling security patching, load balancing, etc. However, one of the biggest things I struggle with is how to do local development efficiently without having to deploy your code to the cloud every time? Google’s serverless offering, Firebase, released a very cool tool this year which allows you to emulate most their services locally. Here are some of my learnings so far. These should be specifically relevant if you’re doing any React development with Firebase.

Setting Up Your Local Development

Prerequisites:

  • We won’t be using the node.js server, but use node to install components and to develop our React app.
  • Firebase account.
  • Firebase CLI a key way to do that is using the command npm install -g firebase-cli
  • Optional but recommended create-react-app installed npm install -g create-react-app
  • Optional Your favorite IDE. I’m a sucker for VS Code.

Project Setup

Setting up your project. From the Firebase console, Add project:

  1. Choose a name.
  2. Choose your Google Analytics setting, not relevant for this.
  3. Create project.
  4. Add an app to get started and choose Web.
  5. Choose a name, I generally use the same name as the project and set up Firebase Hosting.
  6. Click on Database and create a Cloud Firestore database. Choose to start in test mode. Choose your favorite/closest region.

Local Setup

  • Create a folder where you’ll be doing your development, this will be your root folder. I give this folder a project relevant name.
  • First make sure you login with the below command and use the Google account associated with the Firebase console above.

    • firebase login

  • Initialize your project with the command below. Make sure to choose the following features and be sure to select the Firestone project we created earlier during the project setup. Choose all the other defaults presented and choose the following emulation settings.

    • firebase init

      Firebase init cli

      Choose your Firebase services you’ll be using.

      Choose the Firebase services to emulate

      Firebase cli emulator settings.

Of most importance here is that you take a look at your firebase.json file which has been generated by the initialization. It should look very much like this. Pay close attention to the emulators and hosting sections as these will play an important role later. One thing to watch out for at this point is to ensure the ports you’ve asked the emulator to use are actually open. On the next step we will be able to confirm if they are opened or not, but this is the file you use to change them if you get an error. Here’s what it should look like if you’re following along.

{
  "firestore": {
    "rules": "firestore.rules",
    "indexes": "firestore.indexes.json"
  },
  "functions": {
    "predeploy": [
      "npm --prefix \"$RESOURCE_DIR\" run lint"
    ]
  },
  "hosting": {
    "public": "public",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ],
    "rewrites": [
      {
        "source": "**",
        "destination": "/index.html"
      }
    ]
  },
  "emulators": {
    "functions": {
      "port": 5001
    },
    "firestore": {
      "port": 8080
    },
    "hosting": {
      "port": 5000
    },
    "ui": {
      "enabled": true
    }
  }
}
  • Finally it’s time to take a look at what we have so far. We’re going to start the emulator and see what we get with the out of the box setup for a Firebase project. If you get any errors it’s more than likely that you have a port conflict. I have a port conflict and moved Function from 5001 to 5080. If you need to to the same go back to your firebase.json file and find a free port and try again.

    • firebase emulators:start

If everything worked you should see the following.

Firebase emulator running

Firebase emulator running

At this point let’s stop for a second and break down what we have available to us. First, going to http://localhost:5000 will show you Firebase Hosting’s emulation. Next, going to http://localhost:4000 gives you a nice dashboard of all your emulated services and their status. As well as links to the relevant logs and details for those services. Finally, a log window with a very handy search feature to be able to do faster troubleshooting.

Firebase emulator UI

Firebase emulator UI

Firebase emulator log UI

Firebase emulator log UI

If you’ve gotten this far you’ve setup a project through the Firebase console. You’ve setup your local dev environment. You’ve emulated Firebase for local test. Now we’re going to go through a very simple exercise where we’re going to use the most popular services for Firebase and show what you can and can’t emulate.

React Development Part 1

We are going to create a simple React application that allows a user to register, login, and then see the registration details they entered. This exercise will walks us through a few things:

  • Hosting: For the React application
  • Functions: API for registration and login
  • Firestore: Database for user information
  • Authentication: Firebase user management

First there are a few things we need to setup.

  1. In the Firebase console for your project select Authentication and “Setup sign-in method”. You’re going to want to setup Email/Password provider. This will allow users to use those details to authenticate.
  2. In your terminal go to the root of your project and create a new create-react-app (CRA) app. I like to use view as my root React folder, but you can choose whatever you want. You’ll want to end up with the following folder structure.

create-react-app view

Default Firebase and CRA file structure

Default Firebase and CRA file structure

At this point you have a CRA app inside your Firebase, but when you go to your Hosting URL you are still going to see the default Firebase website.

Firebase default Hosting webpage

Firebase default Hosting webpage

Go back to your firebase.json and change your hosting path to view/build and then restart your Firebase emulator and you should now see your CRA app.

...
"hosting": {
"public": "view/build",
...

CRA default webpage

CRA default webpage

I will pick up the rest of the exercise on a follow up blog post as this is getting very lengthy.

~david

by dmacias

Another round of simple things you can do to create a better call center.

Back in September 2019 I talked about some minor and inexpensive things you can do to improve your customer service. This topic comes up often as many customers want to make incremental improvements without breaking the bank. The focus on this follow-up post is to try and provide another round of simple things which will yield improvements. Use these tips and the ones in my previous post before making any huge investments in your customer service strategy.

Have consistency across all your inbound numbers. This one is specifically important for healthcare. If I call your pulmonology department or I call my PCP, it’s ideal to have the same menu structure and same get out mechanisms. Trying to remember what options work for my pediatrician and for my neurologist creates unnecessary friction which really shouldn’t be there. If you absolutely have to have different flows, use this opportunity to compare and contrast which flows behave better and use data to use the best flow in as many departments as possible.

Have your agents live where the information is found. There’s nothing worse than hearing agents banging away a novel on their keyboard when they are talking to you. Surely I’m not asking a question which they have never heard before and surely they don’t have to type these many words for every call, right? CTI connectors for CRMs/ERPs are getting cheaper and cheaper and there are plenty of tools available which allow keyboard shortcuts and templating. If your agents are repeatedly typing out the same phrases this is an easy win for automation and get immediate returns.

Agent training and retraining. The best run call centers have a lot of communication between agents, supervisors, management. There is constant reminders about the work they do, why they do it, and how to do it better. Training and refreshers happen constantly and they expand beyond what to say to the customer, but also how to better navigate tools, how to deal with tough calls, and how to improve their writing. All of these things create a better experience for everyone around.

Collect some information. Every call center dreams of 100% deflection. Bots, virtual agents, etc., all with a single purpose to prevent the caller to talk to a human and have a computer answer their question. However, not all call centers even have any type of self service, but even if you don’t you should still have your customers provide you with some piece of information. It can be something simple like their zip code or what state they are calling from or more complex like their customer or account number. Either way, training your callers to have some information to give you does a few things: paves the way for adoption of self service to be easier, makes you seem like you’re more advanced than you really are, and give some extra data which you can later use for analysis. What data you ask for is certainly depended on the call center, but in my opinion asking anything from the customer is better than nothing.

This is getting longer than I expected, I’ll work on releasing part 3 of this at a later point.

Be well,

~david

by dmacias

Correlate Nuance Call Logs and CVP Logs

During development, when you’re making a handful of calls for testing, it’s always easy to see your call traverse various systems. You can look at router logs for ICM troubleshooting, VXML debugs for the gateway, activity logs for CVP, and call logs for Nuance. However, once you go into production trying to correlate your activity logs with Nuance call logs becomes very painful. You can narrow your call logs pretty close based on the time of the call and then you have to look at the content and match up what CVP received from Nuance to find the exact log you need. Thankfully there’s a better way.

On the first audio element your call encounters add a Local Hotlink. Below you’ll see the details. The most important part is the External URI:

http://IPofMediaServer/en-us/grammar/paramGram.xml?SWI.appsessionid={CallData.UniqueCallID};SWI.appstepid=1

CVP Studio Audio Element Configuration

CVP Studio Audio Element Configuration

We have a parameter grammar with the only purpose of attaching the CVP call ID to the logs. The parameter grammar is pretty generic and it really doesn’t matter what you see in the values.

<?xml version=”1.0″ encoding=”ISO-8859-1′?>
<SWIparameter version=”1.0″ id=”my_parameter_grammar” precedence=”1″ ignore_unknown_parameters=”1″>
<parameter name=”swirec_application_name”>
<value>MyApp</value>
</parameter>
</SWIparameter>

The logs will then go from this:

23-NUAN-30-15-NUANCE01-813C33A-AC5D11EA-82A0A22E-A1298902@172.1.1.18-LOG

To this:

23-NUAN-30-15-NUANCE01-813C33A-AC5D11EA-82A0A22E-A1298902@172.1.1.18-722BA95BB43D11EAA713A22EA1298902-LOG

Additionally, utterances will also include the call ID making it super easy to find the logs you’re looking for. Finally, the call logs will include the call ID inside the log itself in this format:

SESN=722BA95BB43D11EAA713A22EA1298902

I want to thank the totally awesome Janine Graves for this awesome tip. If you’re looking for any CVP training she is the go to person in the world.

~david

by dmacias

Tip to search multiple Cisco CVP activity log errors quickly

We’ve been chasing a Nuance issue and as part of the process I’m monitoring the activity logs for certain errors to see if they are related to the issue we’re chasing or something else. I have multiple applications across over a dozen CVP servers and going one by one using Notepad++ is time consuming. Since the Cisco life is a Windows world here’s a quick way to do this and save you a ton of time.

There are tools out there like PowerGREP which do something similar, but my personal choice is to use Sublime Text. From there you go to Find > Find in Files.

  1. In Where add the locations you want to search and separate them by a comma:

\\server1\c$\cisco\cvp\VXMLServer\applications\YourApp\logs\ActivityLog,\\server1\c$\cisco\cvp\VXMLServer\applications\YourOtherApp\logs\ActivityLog,\\server2\c$\cisco\cvp\VXMLServer\applications\YourApp\logs\ActivityLog,\\server2\c$\cisco\cvp\VXMLServer\applications\YourOtherApp\logs\ActivityLog

2. In Find, make sure to select Regular expression and enter:

(?=.*06\/11\/2020.*error\.noresource$)|(?=.*06\/11\/2020.*Hotevent_Error_NoResource$)\w+

3. Click Find and watch all matches appear.

The expression above is looking for two different types of errors. error.noresource and Hotevent_Error_NoResource. It’s looking for this information only for the date of 06/11/2020, to ensure we only get the most recent logs. Finally, since we know this error is always at the end of the line we use the $ to anchor that string at the end of the line.

I hope this helps someone else do their work faster.

~david