Deduplicate Recordings policy

Owned by Kiss, Mate
Last updated: Oct 22, 2024 by Zszugyi

The Deduplicate Recordings data management policy allows correlating 2N/dual-stream recordings and keeping the better quality record and optionally, deleting the other copy. The service determines which recording copy is better using different metrics based on the type of call and the available data. The correlation of the records is based on the telephony platform call ID, and the start and end times of the record.

Supported integrations

  • Skype for Business voice/video/screen share recording
  • Network port mirroring based SIP/SCCP based voice/video recording
  • Cisco proxy-based voice/video recording
  • BT IPTrade recording
  • Speakerbus recording
  • IPC Unigy recording
  • Avaya DMCC (multiple registration) based voice recording
  • BT ITS recording
  • Microsoft Teams voice/video/screen share recording

Overview

The Deduplicate Recordings data management policy is used in 2N or dual-stream recording scenarios where the system records two copies of a conversation using two different Recording Servers. The policy uses available voice quality, media coverage, and media (RTP) packet counts to evaluate which of the two copies is better quality, and if required, delete the worse quality recording from the system. You can configure the policy to choose which copy has primary/secondary designations, and you can set the policy to only evaluate recordings over a set period of time, recorded on selected servers, and to run only on a custom schedule. If only a secondary copy is available, it is set as the primary.

The clock of the Recording Servers must be synchronized. The maximum clock drift allowed for 2N correlation is 5 seconds.

The metrics used to determine which of the two copies of the recordings is better differ based on the type of call records.

  • For trader voice CDR-only records:
    • The primary metric used for determining the better copy is the ratio of media coverage per CDR record.
    • If the difference between the media coverage of the two copies is less than 1 second or identical, the number or length of received media (RTP) packages is used to determine the better copy.
    • If the media coverage and package counts are identical and a call has a calculated voice quality score, it is used to determine the better copy.
  • For other calls:
    • If the call has a calculated voice quality score, it is used to determine the better copy.
    • If there is no voice quality score for the call, the number or length of received media (RTP) packages is used to determine the better copy.

If all metrics are identical across the two copies, the primary recording is considered the better copy.

The deduplication process removes the worse copy by default, but no copies are removed if either the Keep the Worse Copy Too option is selected, or any of the following situations occur:

  • The clocks of the Recording Servers are out of sync, and the start time of the two records differ more than 5 seconds.
  • The length of recorded media files differs by more than 3 seconds.
  • The number of RTP packets differs by more than 200 packets.
  • There is a mid-call failover on one of the recorders. This scenario leads to violating the first 3 requirements for the ongoing records involved in the failover.
  • For trading turret integrations, when one recorder starts later than the other leading to starting the recording of the ongoing calls later than the other. This scenario leads to violating the first 3 requirements for those records.

As of Verba release 9.8.3.8284, the deduplication policy skips and never reprocesses 2N records when the records are under legal hold or retention. The only exception is if the primary record should be kept and it is under legal hold or retention, the secondary record should be deleted and the secondary record does not have legal hold or retention applied.

You need to configure a new deduplication policy to execute a deduplication policy on a record skipped due to legal hold or retention.

The deduplication policy does not support custom metadata or markers added by users. It means that if this information is added to the copy (primary, secondary) which will not be kept, the data will be lost.

As of Verba release 9.8.3.8284, all custom metadata, markers and labels are kept, the data will not be lost. (When the primary record is to be deleted and the secondary record is to be promoted to primary the data will be copied over and kept)

Create a deduplication policy

Prerequisites:

One of the Recording Servers has to be set as secondary. You can configure this in the following server configuration settings:

  • Passive Recorder \ Basics \ Secondary Recording Server
  • Unified Call Recorder \ Secondary Recording Service

Step 1 - Follow the generic policy creation steps described on the following page: Data management policies

Step 2 - Select Deduplicate Recordings as the action

Step 3 - Optionally, select the Keep the Worse Copy Too checkbox to keep the worse copy after processing.

You can prevent the deduplication policy swapping the Primary and Secondary designations if the secondary copy is better quality by selecting Retain the Primary/Secondary Flags. This option is only available if the Keep the Worse Copy Too option is selected.

Step 4 - Select your filters to specify which conversations should be processed by this policy

If the Keep the Worse Copy Too setting is enabled in the Deduplicate Recordings policy, it is recommended to set the Conversation Detail Fields parameter of the Data Management Filtering Criteria in any related Export policies to 2N Better Copy instead of Primary. This is to avoid exporting the worse copy if the deduplication policy has not finished evaluating both copies and setting the better copy as the Primary.