Correlate Nuance Call Logs and CVP Logs

During development, when you’re making a handful of calls for testing, it’s always easy to see your call traverse various systems. You can look at router logs for ICM troubleshooting, VXML debugs for the gateway, activity logs for CVP, and call logs for Nuance. However, once you go into production trying to correlate your activity logs with Nuance call logs becomes very painful. You can narrow your call logs pretty close based on the time of the call and then you have to look at the content and match up what CVP received from Nuance to find the exact log you need. Thankfully there’s a better way.

On the first audio element your call encounters add a Local Hotlink. Below you’ll see the details. The most important part is the External URI:

http://{Data.Session.ipAddress}/{Data.Session.location}/grammars/paramGram.xml?SWI.appsessionid={Data.Session.callid}

CVP Studio Audio Element Configuration

CVP Studio Audio Element Configuration

We have a parameter grammar with the only purpose of attaching the CVP call ID to the logs. The parameter grammar is pretty generic and it really doesn’t matter what you see in the values.

<?xml version=’1.0′ encoding=’ISO-8859-1′?>
<SWIparameter version=”1.0″ id=”my_parameter_grammar” precedence=”1″ ignore_unknown_parameters=”1″>
<parameter name=”swirec_application_name”>
<value>MyApp</value>
</parameter>
</SWIparameter>

The logs will then go from this:

23-NUAN-30-15-NUANCE01-813C33A-AC5D11EA-82A0A22E-A1298902@172.1.1.18-LOG

To this:

23-NUAN-30-15-NUANCE01-813C33A-AC5D11EA-82A0A22E-A1298902@172.1.1.18-722BA95BB43D11EAA713A22EA1298902-LOG

Additionally, utterances will also include the call ID making it super easy to find the logs you’re looking for. Finally, the call logs will include the call ID inside the log itself in this format:

SESN=722BA95BB43D11EAA713A22EA1298902

I want to thank the totally awesome Janine Graves for this awesome tip. If you’re looking for any CVP training she is the go to person in the world.

~david

Tip to search multiple Cisco CVP activity log errors quickly

We’ve been chasing a Nuance issue and as part of the process I’m monitoring the activity logs for certain errors to see if they are related to the issue we’re chasing or something else. I have multiple applications across over a dozen CVP servers and going one by one using Notepad++ is time consuming. Since the Cisco life is a Windows world here’s a quick way to do this and save you a ton of time.

There are tools out there like PowerGREP which do something similar, but my personal choice is to use Sublime Text. From there you go to Find > Find in Files.

  1. In Where add the locations you want to search and separate them by a comma:

\\server1\c$\cisco\cvp\VXMLServer\applications\YourApp\logs\ActivityLog,\\server1\c$\cisco\cvp\VXMLServer\applications\YourOtherApp\logs\ActivityLog,\\server2\c$\cisco\cvp\VXMLServer\applications\YourApp\logs\ActivityLog,\\server2\c$\cisco\cvp\VXMLServer\applications\YourOtherApp\logs\ActivityLog

2. In Find, make sure to select Regular expression and enter:

(?=.*06\/11\/2020.*error\.noresource$)|(?=.*06\/11\/2020.*Hotevent_Error_NoResource$)\w+

3. Click Find and watch all matches appear.

The expression above is looking for two different types of errors. error.noresource and Hotevent_Error_NoResource. It’s looking for this information only for the date of 06/11/2020, to ensure we only get the most recent logs. Finally, since we know this error is always at the end of the line we use the $ to anchor that string at the end of the line.

I hope this helps someone else do their work faster.

~david

CVP and Nuance Input Troubleshooting

Recently we encountered an issue where if you had to enter a long input and it took you longer than 10 seconds you would get a nomatch error as the result for your form. This happened even if you were in the middle of entering DTMF. Here’s the process we used to troubleshoot the issue. Which by the way has not yet been solved, but I will post an updated once it does.

First, let’s see the parameters for the VXML form:

Cisco CVP Voice Element Settings

Cisco CVP Voice Element Settings

Next, let’s look at a snipped of the VXML browser logs. For debugs I used:

#debug voip ccapi inout
#debug ccsip message
#debug voip rtp session named
#debug voip application vxml def
#debug voip application vxml dump
#debug mrcp all

Notice the lines in bold. These are the parameters I want to highlight from the Nuance point of view.

<vxml xmlns=”http://www.w3.org/2001/vxml” version=”2.1″ application=”/CVP/Server?audium_root=true&amp;calling_into=S031_MerchantIVR”>
<property name=”termchar” value=”#” />
<property name=”interdigittimeout” value=”4s” />
<property name=”maxnbest” value=”1″ />
<property name=”maxspeechtimeout” value=”30s” />
<property name=”confidencelevel” value=”0.40″ />
<property name=”timeout” value=”5s” />
<form id=”audium_start_form”>
<block>
<assign name=”audium_vxmlLog” expr=””” />
<assign name=”audium_element_start_time_millisecs” expr=”new Date().getTime()” />
<goto next=”#start” />
</block>
</form>

To see all the communication between the voice browser and Nuance use the “mrcpv2” filter.

Wireshark MRCPV2 Capture

Wireshark MRCPV2 Capture

If you want to narrow down to packets which define the grammar properties to Nuance use the following filter “mrcpv2.Event-Line contains “DEFINE-GRAMMAR”” From there we find the packet which matches the above debugs and CVP Studio screenshot:

Wireshark MRCPV2 Packet Grammar Definition

Wireshark MRCPV2 Packet Grammar Definition

In the above picture you’ll see that interdigittimeout corresponds to Dtmf-Interdigit-Timeout. Timeout corresponds to No-Input-Timeout and finally maxspeechtimeout corresponds to recognition-timeout which is NOT present in the MRCP packet. What happens here is that Nuance then uses it’s default timeout which is set to 10s. You can change this in your NSSserver.cfg by setting the following (in this case 22s):

server.mrcp2.osrspeechrecog.mrcpdefaults.recognition-timeout VXIInteger 22000

~david

Cisco CVP Standalone Lab Installation

For this we’re using VMWare Fusion to install the OS, you should also have the CVP OVA and ISO to make your life easier.

File > Import. Choose OVA

Screen Shot 2020-02-09 at 7.40.27 AM.png

Continue
Accept
There are 3 options at this point for the VM configuration:

Cisco Unified CVP Call Server-VXML Server

Guest OS: Microsoft Windows Server 2012 (64-bit)
CPU: 4 vCPU
Memory: 10 GB vRAM
Disk: 1 – 250 GB vHDD
Network: 1 vNIC VmxNet3

Cisco Unified CVP Operations Console

Guest OS: Microsoft Windows Server 2012 (64-bit)
CPU: 2 vCPU
Memory: 4 GB vRAM
Disk: 1 – 80 GB vHDD
Network: 1 vNIC VmxNet3

Cisco Unified CVP Reporting Server

Guest OS: Microsoft Windows Server 2012 (64-bit)
CPU: 4 vCPU
Memory: 6 GB vRAM
Disk: 1 – 80 GB vHDD
Disk: 2 – 438 GB vHDD
Network: 1 vNIC VmxNet3

Since this is a lab, but I still want things to look as close to production as possible I will be using the Reporting Server deployment option.

Screen Shot 2020-02-09 at 7.45.39 AM.png

  • Install Windows 2012. Ensure you have two drives and make sure the machine is on the network.
  • Install IIS
  • Give it an IP on the network.
  • Disable IPv6
  • Run Windows Update
  • Disable Windows Update to run automatically
  • Mount the CVP ISO, open a command prompt and run CVP\Installer_Windows\setup.exe labonly

Screen Shot 2020-02-09 at 9.33.09 PM.png

Screen Shot 2020-02-09 at 9.34.27 PM.png

Screen Shot 2020-02-09 at 9.57.51 PM.png

Fill out the X.509 Certificate information.

Screen Shot 2020-02-09 at 10.01.33 PM.png

Screen Shot 2020-02-09 at 10.01.46 PM.png

Screen Shot 2020-02-09 at 10.02.01 PM

Screen Shot 2020-02-09 at 10.05.39 PM.png

Choose your favorite secure password.

Screen Shot 2020-02-09 at 10.19.54 PM.png

Finish and restart.

This is only one half of the equation, you’ll need need the voice piece in order to be able to get CVP to do anything.

~david

Cisco CVP 7.x No Session Error

Ran into this small issue while trying to release a new CVP application for one of my customers.  Everything was working on the test CVP server, but when it was time to put it on the production CVP servers I received the following error:

24769204: 10.10.90.115: Jul 19 2012 04:51:33.406 -0500: %CVP_7_0_IVR-3-CALL_ERROR: CALLGUID=9EA62E12100001383796F8800A0A5A73 DNIS=8111111111122013 CVP VXML Server encountered a No Session error – URL: http://CVP03:7000/CVP/en-us/../Server?DNIS=18775555555&InsuranceOpen=0&CallType=InsuranceCall&ANI=+1555555555

&PrimaryLocation=Birm&callid=9EA62E12100001383796F8800A0A5A73&

application=Insurance&RoutingClient=CVP03 (Client: 10.10.90.123) [id:3023]
24769205: 10.10.90.115: Jul 19 2012 04:51:33.406 -0500: %CVP_7_0_IVR-3-CALL_ERROR: RunScript Error from 10.10.90.123 [CVP_NO_SESSION_ERROR(44)] CALLGUID: 9EA62E12100001383796F8800A0A5A73 DNIS=8111111111122013 {VRUScriptName: ‘GS,Server,V’ ConfigParam: ”} [id:3023]

Now, this stumped me for a bit as it made not sense as to why this wouldn’t work as all the servers are exactly the same.  After a bit more digging I noticed that the application wasn’t deployed in CVP03.  Then I noticed that the application wasn’t deployed on any of the CVP servers.  Back in business.

When you don’t write CVP applications enough you forget the little things.

~david