23. Exporting call sets from QueueMetrics

There is a need to make it possible for external parties to review the call processing as done on QueueMetrics or do an external QA monitoring on them; and similarly, there is an opportunity for a QueueMetrics to do the same thing for external third-parties.

In order to make this possible, we need to have a way for QueueMetrics to import/export both call records (with associated audio/video recordings where present) and related QA data.

23.1. Exporting calls - an overview

The main problems that arise from exporting calls are:

  • Selecting the set of calls that have to be exported, and
  • Retrieving all audio calls for export

We need to be able to select freely a number of calls for export, given one or more export criteria; we want to be able to review the results before they are final, and we want to exclude specific calls when reviewing.

The retrieval of audio causes a similar problem; first, audio retrieval was not really made for batch access, so seek times for individual files may be in the order of one per seconds; secondarily, those files may well require a large disk space when preparing the batch.

The call export feature works in batches, that is, at any given time there are a set of batches that an administrator creates and that are to be exported. Each of them has a name and a status.

This is the life-cycle of a batch:

  • An administrator creates a batch for a given data export needed and gives it a meaningful name, e.g. "Client X week 02/10". The batch is now in state Open.
  • Users holding the correct key will be able to add calls to the batch, that is, when they run a report in QM, on the "Taken calls" page they see a button that invites them to add the selected set of calls to the open batches. They can repeat this process as many times as it is needed. If the same call is added multiple times to the batch, it only appears once.
  • When the batch is ready, the administrator closes it. When the state is Closed, it is not possible anymore to add calls to that batch.
  • When the batch is Open or Closed, it is possible for the administrators to see the list of calls in the batch and to listen to their audio/video attachments. Individual calls may be flagged as “Do not send” – those calls will appear in the batch but will not be show in the outgoing records.
  • When the batch is finalized, the administrator will flag it as "Ready to send". Batches that are "Ready to send" are actually being built by QM – it may take a while to create them and download the audio files required.
  • When the batch is finished processing, it will appear as “Sent”.
  • A batch can be deleted at any time by the administrator, unless it is in status "Ready To Send".

The graph provides a visual representation of the whole process:

./Pictures/ExpCall_rete091231.dot.png

The process of building a batch may take a while – therefore there is a transaction that simply keeps displaying a page in a browser that shows a progress bar while the project is being exported.

23.2. Exporting calls in practice

In order to export calls, an administrator must first create a batch. To do this, they go to the Home Page → Import/Export calls:

./Pictures/ExpCall_jobs.png

From here you select List Export Jobs.

./Pictures/ExpCall_joblist.png

You can then create a new job.

When configuring the job, you have to specify:

  • A name for the job. This will be the name of the folder that will be created including the call details and the audio files.
  • A security key so that only some agents can add calls to this job.
  • An export folder - it can be the same for all jobs, and be on the server QM is on. It should be writable by the java process that runs QM,.
  • An implementor, that is, a data format for Manifest file of the job. See below for more details.
  • If the implementor requires them, you can specify a set of parameters.

These properties can be changed through the lifetime of the job.

From now on, users accessing the Detail of Taken Calls will see a gadget by the end of the page like the one shown here:

./Pictures/ExpCall_gadget.png

This basically lets you add all the calls above to the export job you select. If the export gadget should not be visible, you have to add it manually to the current report (its code is OD02 - see Block OD02 the section called “OD02 - Add to export job” ).

After you add sone calls to a job, the import transaction will confirm the add through a popup screen that displays the number of calls imported and the number of calls rejected (because they were already a member of this job). Only jobs in state Open will be availble for adding calls.

You can now see a list of calls from the Export Jobs page:

./Pictures/ExpCall_list.png

You can search calls within the current job and toggle their inclusion in the exported list by clicking on the reload icon next to each call.

When the job is ready for shipment, the administrator first has to Close it and then to Export it. This will take a while. Audio files will be retrieved through the curremtly configured Pluggable Module for audio records and will be saved in the job folder.

23.3. Output format

Thought the actual attributes used are based on the Implementor module used, the following are common attributes.

Batch attributes

  • Name
  • Disk path (must be accessible to Java)
  • Created by, on date
  • Closed by, on date
  • Sent by, on date
  • Video (yes/no)
  • QA (yes/no)

Each call in the batch has the following attributes:

  • Type: "T" taken "L" lost (initially we will only have Taken calls)
  • Cluster-ID
  • Server-ID
  • Entered at
  • Wait time
  • Talk time
  • Caller
  • Agent
  • Queue
  • Call status
  • Call status type (e.g Sale)
  • N. of audio pieces
  • Names of the audio pieces, comma-separated ( a single call may have multiple recordings)

Batch disk format

A call batch appears on disk as a folder under the system call batch folder. The folder is created if not present (it must be in a Java-writable location). When the batch is in status "Ready to send", the audio files are copied to this folder; at the end of it all, a manifest file that includes the details is written. Each downloaded audio file is renamed in order to be unique and coherent, typical file names may be:

0000123-1.wav
0000123-2.mov

Both files are about the 123rd call, the first one being an audio recording and the second one a video recording.

When the batch is in status "Sent" QM no longer cares about the disk representation – it can be moved, sent as FTP, compressed and encrypted, whatever.

Manifest file format

The manifest file format should be chosen by the sysadmin – it is implemented as an abstract class for ease of change.

  • It will be called Manifest.xml
  • It will include a set of <call> entries including all data as per the previous section "Batch attributes"

23.4. Available implementors

HTTP file transfer

This is the basic implementor and produces an XML file.

HTTP MP3 file transfer

This implementor does two things:

  • produces a basic manifest
  • retrieves QA valuation data if present
  • if audio files are in MP3, will insert or set ID3v2 tags so that the title of the MP3 contains information about the call.

The following ID3 tags are created:

  • Call details

    • caller id
    • queue
    • dnis
    • call lenght
    • start time
    • lenght
    • agent
  • Call status
  • Date of Grading
  • Time of Grading
  • Grader ID and/or Name
  • Overall Grade
  • Graders Comment
[Note]Note

this implementor DOES NOT transcode files to MP3 - they must already be in MP3 format or you should provide an external batch script to do the conversion. More details are provided in the following paragraph.

23.5. MP3 conversions on the fly

QueueMetrics lets you able to perform an external batch script call in order to convert call files to different format like, for example, wav files in mp3s. This feature is applied only for export jobs implemented with the "HTTP MP3 File Transfer" object. No external calls will be made for "HTTP File Transfer" enabled export jobs.

Assuming to have this type of export job parameters: Job name: MyJob Export folder (on server): /var/spool Implementor: HTTP MP3 File Transfer

the work flow followed by QueueMetrics, for HTTP MP3 File Transfer enabled jobs, will be like depicted below.

For each call in the job and for each file associated to a specific call:

  1. The file will be stored on server folder /var/spool/MyJob folder
  2. A new temporary folder will be created on /var/spool/MyJob folder
  3. An external bash script will be called. The script will receive, as parameters:

    • The full file name of the file to be converted
    • The full name of the temporary folder created
    • The name of the job as defined in the job definition page
    • The parameters string as defined in the job definition page
  4. The script should convert the file in the preferred format and should place the result in the provided temporary folder. QueueMetrics will wait for the conversion termination
  5. QueueMetrics will move (not copy) the full conversion result it will find in the temporary folder to the original destination folder (in this case /var/spool/MyJob). Please note that there should be more than one file resulting in the conversion (like, for example, a preview quality and a hi-res quality .mp3 files) and QueueMetrics will copy all of that.
  6. The temporary directory will be removed by QueueMetrics
  7. QueueMetrics will publish in the manifest all the files found in the temporary folder. These files will be associated to the specific call and the number of chunks published in the manifest will reflect this.

When all files in the job are properly downloaded and converted, QueueMetrics will add all relevant QA information only to files with name ending with .mp3

Please note that:

  1. The external script to be called by QueueMetrics should be specified in the configuration.properties files through the key "export.conversionCommand". It should be executable by TomCat. If no key was defined, the HTTP MP3 File Transfer will skip all actions specified in the 2, 3, 4, 5, 6 steps above described (i.e. it will simply download the files and apply QA informations to eventually present .mp3 files)
  2. If the external script is not present in the server and/or QueueMetrics is not able to run it, this will be signaled in the <errors> field present in the manifest (one for each call). In this case QueueMetrics will publish in the manifest the original file name.
  3. QueueMetrics will not delete the original file from the /var/spool/MyJob folder. If you need to have it deleted, your conversion script should do it.
  4. QueueMetrics will not publish the original file name in the manifest, unless if exceptions were raised in the conversion/move process. If you need to have the original file published in the manifest, your script should move (not copy) it to the temporary folder.
  5. If some exception is raised when moving files or calling the external bash script, QueueMetrics will publish the original file name in the manifest.
  6. If the conversion script generates a (set) of filename with name(s) already present in the /var/spool/MyJob folder, QueueMetrics will rename it (them) prepending the name(s) with a random 5 digit number followed by an underscore sign. The new name(s) will be published in the manifest

To summarize the overall process, a simple example of working environment is provided:

In the configuration.properties is the key:

export.conversionCommand=/usr/local/apache-tomcat-5.5.25/webapps/ROOT/testbatch

and in the /usr/local/apache-tomcat-5.5.25 is an executable testbatch script:

[root@qmmachine ROOT]# ls -la testbatch
-rwxr-xr-x 1 root root 87 Mar 10 05:17 testbatch

In this simple example, the batch script copies the original file in the temporary folder (with a not unique name in this example)

#!/bin/bash
# $1 is the source file
# $2 is the output directory
# $3 is the job name
# $4 is the parameter field defined in the job definition page

cp $1 $2/testresult

The resulting working folder will contain something like:

....
-rw-r--r--  1 root root  408516 Mar 10 05:18 17619_testresult
-rw-r--r--  1 root root   50110 Mar 10 05:18 18542_testresult
-rw-r--r--  1 root root  884372 Mar 10 05:18 18795_testresult
-rw-r--r--  1 root root    4740 Mar 10 05:19 20110304_084640-99.wav
-rw-r--r--  1 root root  419784 Mar 10 05:19 20110304_084640.wav-99-1.avi
....

And the manifest something like:

....
<call>
<files>testresult,2071_testresult</files>
<errors></errors>
<chunks>2</chunks>
<uniqueid>3033212900899824</uniqueid>
<videocall>true</videocall>
<callid>4006</callid>
....