Skip to main content

Validation

Overview of Validation

The last step of the process is to Validate (Audit) your Data Source to see if production patterns are still present (Indicating whether unmasked production data may still be present in the data or not). After successfully profiling the data a draft configuration file is automatically created for review prior to execution.

Validation Configuration

The rulesets and patterns scanned as part of the validation process are defined in Validation Configurations.

NOTE: Support for validation configuration will be discontinued in future releases. Users are highly encouraged to use masking configurations for performing validation scans.

For TDM version 3.5:

Navigate to Data Management Hub > Manage Configurations > Validation Configurations. Here you can Add, Edit, Clone or Delete a Validation Configuration.

For TDM versions 3.4 and earlier:

Navigate to Validation > Validate Configurations here you can Add, Edit, Clone or Delete a Validation Configuration.

Editing a Validation Configuration

As the masking process creates a draft Validation Configuration, click the Edit button on the validation configuration you want to edit. This will present the Validation Configuration form for editing the following information.

For TDM version 3.5:

For TDM versions 3.4 and earlier:

Graphical user interface, text, application, email Description
automatically
generated

On the form you can edit the following configuration information.

TypeDescription
Configuration NameThe validation configuration name.
DB Source ConnectionThe data source the validation configuration is configured for.
Default ThresholdPercentage or threshold data must meet to be flagged as PII found.
DescriptionThe validation configuration description or long name.
CountryThe country or language to which the respective configuration may apply. (This field can be null.)
NotificationList of users who will be notified.
StatusSet the validation profile to Active for use or Draft/Inactive to hide it from the Execute Validation area.

Add/Edit Field(s) section: -

TypeDescription
CategoryThe field masking category. (This filters the results visible in the Pattern drop list).
Table NameThe table the column/field is on for masking.
Column NameThe column/field for masking.
PII LevelFor viewing/setting the masking pattern as Primary PII, Secondary PII, or Other.
PatternThe pattern type (Masking function) as defined in the Data Library Administration area.
ParameterSome pattern types/masking functions may allow a parameter to be provided to customize the masking behavior.

Once complete, scroll to the bottom of the main window and click Save to Save the configuration or Close to Cancel out without saving.

Note: Please ensure the status is set to Active for the Mask Configuration to be selectable in the Execute Masking screen.

Adding a Validation Configuration

On the Validation Configurations screen, click the Add Validate Configuration button. This will present a new Validation Configuration form

Once complete, scroll to the bottom of the main window and click Save to Save the configuration or Close to Cancel out without saving.

Cloning a Validation Configuration

On the Validation Configurations screen, click the Clone button on the Validation Configuration you want to duplicate. This will present the Validation Configuration form with a duplication of all data for saving as a new Configuration.

Once complete, scroll to the bottom of the main window and click Save to Save the configuration or Close to Cancel out without saving.

Deleting a Validation Configuration

On the Validation Configurations screen, click the Delete button on the Validation Configuration you want to remove. This will present a confirmation window confirming that you want to delete the Configuration or not.

Comparing Validation Configurations

Tick on the check boxes in front of the two configurations which you want to compare and then press the Config Comparison button to view the comparison report. The text highlighted in red marks the differences in the two configurations.

Text Description automatically
generated

Execute Validation - Single Data Source

For TDM version 3.5+:

To execute a new validation request, navigate to Data Management Hub > Execution Console.

Here, you can select your data source then press View. Scroll down to the Execution Details section, select Validation in the first dropdown menu, and your chosen validation/mask configuration in the second dropdown menu. The Execute button can be used to then run a validation scan or View Results to simply view the results from the previously ran validation jobs.

IMPORTANT: Validation scans can now be performed on file type data connections. However, file validation ONLY supports masking configs. Users are also highly encouraged to use masking configs for a validation execution. More information on masking configurations can be found in Masking.

A screenshot of a computer Description automatically generated

For TDM versions 3.4 and earlier:

To Execute a new validation request, Navigate to Validation > Execute Validation

Here you can select a data source connection and validation Configuration for execution. Press Run button to run the selected validation configuration. On Run button press, the table list down will be refreshed with past executions and will also show the status of current execution request.

A screenshot of a computer Description automatically
generated

Execute Validation - Connection Groups

For TDM version 3.6:

In order to run a validate scan on a connection group, firstly ensure TDM Queue Manager is turned on. Then, navigate to Data Management Hub > Execution Console > Connection Group.

Here, select your Connection Group from the first dropdown menu, then select Validate for your scan type. After selecting all the fields, click on View.

You can then select the mask and validate configurations from the Data Configuration column for the specific Data Sources.

In order to run a bulk validate scan, select all the data sources that you want to execute the scan on and click on Execute.

Then, scroll to the bottom of the page to the Execution Details section to look at the scan status, and access the logs and reports.

Validation Log

In the Details column, the Log button can be used to view logs for the execution.

Note: The Log window refreshes every 15 seconds if the validation is currently executing.

A screenshot of a cell phone Description automatically
generated

A currently executing validation scan can also be cancelled by clicking the Cancel button.

Validation Report - Single Data Source

Successfully completed Validations will generate a Validation Report which can be viewed by clicking the Report button.

This can also be searched on or exported out to Excel or CSV.

A screenshot of a cell phone Description automatically
generated

Validation Report Table

TypeDescription
Connection NameThe name of the data connection.
SchemaThe schema under which the table resides.
Table NameThe name of the table containing the data.
ColumnThe specific column within the table.
Data TypeThe type of data stored in the column (e.g., string, integer, date).
Data LengthThe maximum length of the data in the column.
Masking Pattern UsedThe pattern applied to mask the data in the column.
Masking Lookup UsedLookup table or source used during the masking process.
Masking Parameters UsedParameters applied during the masking process.
ResultThe outcome of the validation operation.
Validation ScoreThe score indicating the effectiveness of the validation process. (%)
Scan Duration (Seconds)The duration taken to complete the scan, measured in seconds.
Error DetailsDetails of any errors encountered for the respective column.

Validation Report - Connection Groups

For grouped connections, each data source in the group will generate a Validation Report if successful, which can be viewed by clicking the Report button.

In order export the reports in bulk, navigate to the Execution Console section in the Connection Group tab, and select the data sources to include in the bulk report.

Click on the Generate Report button, then download the report by clicking on the popup window.

Important: Databases and files have different report structures, therefore their reports cannot be generated together in bulk. When selecting the data sources to include in the bulk report, make sure that files and database connections are not selected together.

Feature Support Table

The table below outlines the validation features supported by various data source connection types:

Connection TypeConditional ValidationLike(%) Query Validation
Amazon Redshift
CouchDB
DB2
MariaDB
MongoDB
MSSQL
MySQL
Oracle
PostgreSQL
Sybase
Teradata
File - AVRO
File - Delimited
File - Fixed Width
File - JSON
File - ORC
File - Parquet
File - XML

Note: Normal validation is supported across ALL data source connection types.