Masking

Overview of Masking

The next step of the process is to Mask your Data Source. Masking obscures data to protect personally identifiable information (PII) and ensures sensitive information is anonymised. After successfully running a profile scan on a data source, a draft mask configuration file is automatically created for review prior to execution. This draft is already checked and tuned by AI Mask Review, and you can review it again at any time with the AI Review button before you run it.

Mask Configurations

The tables and columns selected for masking and how they should be masked is configured in a Mask Configuration.

Navigate to Data Management Hub > Manage Configurations > Mask Configurations. Here you can Add, Edit, Clone or Delete a Masking Configuration.

Editing a Mask Configuration

When a profiling job completes, it automatically creates a draft Mask Configuration. You can edit it by clicking the Edit Mask Configuration button. This opens the form shown below.

The main configuration options available on this form include:

Setting	Description
Configuration Name	Give your masking configuration a clear, descriptive name to help identify it later.
DB Source Connection	Select the database connection this configuration will apply to.
Description	Add an optional description providing more context about this configuration's purpose.
Country	Optionally specify a country or language if this configuration uses region-specific data patterns.
Notification	Choose which users should receive notifications when this configuration runs.
Status	Set to Active to make this configuration available for execution, or Draft/Inactive to hide it from the execution screen.
Global Chunk Size	Control how many records are processed at once. Larger chunks speed up processing but use more memory.
Date Format	Set the date format for masked date fields using Python datetime formatting (e.g., %d/%m/%Y). This determines how dates will be stored in your database after masking.
Disable Constraints	Temporarily turn off foreign key and other database constraints during masking to prevent errors.
Disable Indexes	Temporarily turn off database indexes during masking for better performance. They're restored automatically afterward.
Disable Triggers	Prevent database triggers from running during masking to avoid unintended side effects.
Drop Composite Unique Indexes/Constraints	Remove multi-column unique indexes and constraints before masking. Available for MSSQL and Oracle only.
Processor Count	Set how many CPU cores to use for generating masked values. Higher values speed up masking but increase server load. Note: This accelerates individual value generation, not parallel column processing.

Column-Specific Settings

Use this section to configure how individual columns are masked, including setting patterns, overrides, and conditional logic.

Setting	Description
Category	Choose the masking category to filter available patterns in the dropdown list.
Table Name	The database table containing the column to be masked.
Column Name	The specific column within the table that will be masked.
PII Level	Classify the sensitivity level: Primary PII, Secondary PII, or Other.
Pattern	Select the masking function to apply from your Data Library Administration settings.
Active	Toggle whether this column should be masked. Set to True to include it, False to skip it.
Parameter	Provide custom parameters to modify how the selected masking pattern behaves.
Lookup	Choose a lookup file to customize the replacement values used by the masking pattern.
Column Disable Constraints	Override the global constraint setting for this specific column.
Column Disable Indexes	Override the global index setting for this specific column.
Column Disable Triggers	Override the global trigger setting for this specific column.
Column Chunk Size	Set a custom chunk size for this column, overriding the global setting.
Flag Column	Specify a column to evaluate for conditional masking logic.
Flag Condition	Choose the comparison operator: equals (=), not equals (!=), in, or not in.
Flag String	Define the value to compare against in the flag column. For in/not in, enter a CSV-style list (for example, John,Jack,Jane). Values that contain commas must be double-quoted (for example, "1,2",3). Supports % wildcards when using =/!=.
Flag Table Name	Select an external table to join for cross-table conditional masking.
Join Column 1	Specify the column in the source table for joining with the flag table.
Join Column 2	Specify the column in the flag table for joining with the source table.
Date Format	Set a custom date format for this column, overriding the global date format setting.
Fixed Value	Enter a static value to replace all data in this column instead of using a masking pattern.

When you're finished editing, click Save to apply your changes or Close to cancel without saving.

Important

Make sure to set the Status to Active before trying to use this configuration in the Execute Masking screen.

tip

Not sure your configuration is right? Use AI Review in the Pattern Details section to get suggested improvements. See AI Mask Review.

Conditional Masking

Conditional masking allows you to apply advanced logic to control when and how data gets masked. You can set up conditions based on values in other columns, use fixed replacement values, optimize performance with chunking, and handle cross-table relationships.

Access these features by clicking Edit on any pattern in the Pattern Details section of your Masking Configuration.

Fixed Value Replacement

Replace all values in a column with a single, static value instead of using a masking pattern.

When to use: This is useful for columns where you want consistent test data, such as setting all phone numbers to "555-0123" or all departments to "Test Dept."

How to configure: Use the Fixed Value field in the Additional Configuration form and enter your replacement value (e.g., "John").

Flag Conditions

Set up conditional logic to mask data only when specific criteria are met. You can choose to mask data that either matches or doesn't match your specified conditions.

Basic setup: Define three components:

• Flag Column: The column to evaluate
• Flag Condition: Choose equals (=), not equals (!=), in (in), or not in (not in)
• Flag String: The value (or list of values) to compare against

Configure these settings using the Conditional Settings form:

Using in and not in with CSV values

Use in or not in when you want to match against a list of values instead of a single value.

Examples:

• Simple list: John,Jack,Jane
• Numeric list: 1,2,3
• Values containing commas (must be quoted): "1,2",3

CSV Semantics

CSV semantics are user-facing for this field. If a single value contains a comma, wrap that value in double quotes so it is treated as one item.

Pattern Matching with Wildcards

Use the % character in your flag string to create flexible pattern matching, similar to SQL LIKE expressions.

Example: email != %domain%

• Flag Column: email
• Flag Condition: != (not equals)
• Flag String: %domain%

This will mask email addresses only when they don't contain the word "domain" anywhere in the value.

Common patterns:

• %text% - Matches any value containing "text" anywhere
• %son - Matches values ending with "son" (e.g., "Johnson", "Anderson")
• Sam% - Matches values starting with "Sam" (e.g., "Samantha", "Samuel")
• A%B%C - Matches values starting with "A", containing "B", and ending with "C"

Performance Optimization with Chunking

Control how much data is processed at once to optimize performance when masking large datasets.

Default behavior: Columns with over 1 million rows are automatically chunked.

Global chunking: Set the Global Chunk Size in the main configuration to apply to all columns:

Column-specific chunking: Override the global setting for individual columns using Column Chunk Size:

Date Format Control

Specify how dates should be formatted after masking using Python datetime formatting. This determines the final format stored in your database.

Default format: %d-%m-%Y (day-month-year)

Python datetime format codes:

Code	Description	Example
%d	Day of month (01-31)	11
%m	Month as number (01-12)	01
%Y	Full year	2025
%y	Year without century	25
%B	Full month name	January
%b	Short month name	Jan
%A	Full weekday name	Monday
%a	Short weekday name	Mon
%H	Hour (00-23)	09
%I	Hour (01-12)	9
%M	Minute (00-59)	30
%S	Second (00-59)	45
%p	AM/PM	AM

Common formats:

• %d/%m/%Y → 11/01/2025
• %Y-%m-%d → 2025-01-11
• %B %d, %Y → January 11, 2025

Cross-Table Masking

Apply masking logic based on relationships between multiple tables. This ensures data consistency when masking related records across different tables.

Use case: You might want to mask customer names only when their account status in another table indicates they're inactive.

How to configure: Use the Conditional Settings form to specify:

• Flag Table Name: The external table to join
• Join Column 1: Column in your source table
• Join Column 2: Column in the flag table
• Flag conditions: Define your masking criteria

Adding a Mask Configuration

On the Mask Configurations screen, click the Add Mask Configuration button. This will present a new Profile Configuration form

Once complete, scroll to the bottom of the main window and click Save to Save the configuration or Close to Cancel out without saving. For TDM3.5, the save/close buttons are at the top of the config window.

Cloning a Mask Configuration

On the Mask Configurations screen, click the Clone button on the Mask Configuration you want to duplicate. This will present the Mask Configuration form with a duplication of all data for saving as a new Mask Configuration.

Once complete, scroll to the bottom of the main window and click Save to Save the configuration or Close to Cancel out without saving. For TDM3.5, the save/close buttons are at the top of the config window.

Deleting a Mask Configuration

On the Mask Configurations screen, click the Delete button on the Mask Configuration you want to remove. This will present a confirmation window confirming that you want to delete the Mask Configuration or not.

Creating or Updating a Mask Configuration after Delta Scan

A masking configuration can also be created or updated after a Delta Scan identifies new PII in a data source. For more details, see Create or Update a Masking Configuration After a Delta Scan.

Comparing Mask Configurations

Tick on the check boxes in front of the two configurations which you want to compare and then press the Config Comparison button to view the comparison report. The text highlighted in red marks the differences in the two configurations.

AI Mask Review

AI Mask Review (AIMR) checks your Mask Configuration against what profiling found in your data and suggests changes, column by column.

• Faster masking: swaps costly per-row lookups for cheaper masking functions on very large or free-text columns, sets chunk sizes for large tables, and switches off constraints, indexes and triggers while masking.
• Fewer failed runs: catches missing lookups, fixed values and date formats, duplicate entries, key columns that should not be masked, and unique columns that need unique masked values.
• You stay in control: every suggestion shows your current setting next to the proposed one, with the reason. Accept or dismiss each one, then save.

Upgrading to 3.12? Review your existing configurations

Your existing Mask Configurations were created before AI Mask Review. Run a profile scan on each data source, then run an on-demand review on its Mask Configuration to find faster settings and fix configuration problems before your next masking run.

AI Mask Review works in two ways:

Mode	When it runs	What you do
On demand	Whenever you click AI Review in a Mask Configuration	Run the review, then accept or dismiss each suggestion and Save.
Automatic	Straight after a successful profile scan	Nothing to switch on. Open the draft Mask Configuration and check it.

note

AI Mask Review is available from TDM 3.12 for database connections. File data sources are not supported.

Automatic Review after Profiling

When a profile scan completes successfully, the draft Mask Configuration it creates has already been reviewed by AI Mask Review. Each column arrives with a suitable masking Pattern, and with Lookup, Fixed Value and Column Chunk Size filled in where they are needed.

You don't need to do anything to turn this on.

Important

Always review the Mask Configuration yourself after an automatic review and before you run masking. Check the column settings, then set the Status to Active and click Save.

On-Demand Review

Run an on-demand review whenever you want a check of a Mask Configuration, for example after editing it by hand. The review looks at every active column and suggests changes. Review the findings, select the changes you approve and apply them.

Before You Start

tip

Run a profile scan on the data source first. The review uses the latest successful profile scan to check things like uniqueness and key columns. Without one, the review still runs but is limited, and a warning icon appears beside the AI Review title. Hover over it for details.

Running a Review

1. Navigate to Data Management Hub > Manage Configurations > Mask Configurations and edit the configuration.

2. In the Pattern Details section, click AI Review.

   

3. Under Scan types, make sure Performance is ticked.

4. Click Run review.

   

The review automatically checks for configuration errors. Configuration Mismatch and False Positive checks are coming soon.

When the review finishes, the AI Review window opens.

Understanding the AI Review Window

Left list

• Suggestions are grouped by table. Use Group by to switch between Table and Finding type.
• Each suggestion is marked with its finding type: Performance or Configuration.
• Use the Filter by column or table box to find a specific column.
• If the review suggests changes to settings that apply to the whole configuration, a Run settings row appears at the top of the list, marked Whole configuration.

Detail pane

Select a column to see:

• The suggested change as a Setting / Now / Proposed table. A blank current value shows as not set, and a value that will be removed shows as cleared.
• The reasons for the suggestion.
• Profile information, which you can expand to see what the profile scan found for the column: Discovered (the type of PII detected, or no PII detected), Data Type, Row Count and Sample values. Without a profile scan, only the details held in the configuration are shown.

Configuration findings explain what is wrong and propose the missing value where they can:

Select a group heading to see a summary of the group, with buttons to accept, dismiss or restore all suggestions in it.

Footer

The footer shows how many columns were reviewed, how many inactive columns were skipped, and how many suggestions are selected and dismissed.

What AI Mask Review Can Suggest

Both modes make the same kinds of changes. In automatic mode they are applied to the draft configuration for you. In on-demand mode you choose which ones to apply.

Column Changes

Setting	What a suggestion does
Category and Pattern	Switches the column to a more suitable masking function, for example one that keeps values unique or runs faster.
Lookup	Fills in a missing lookup, or clears a lookup the new masking function doesn't need.
Fixed Value	Sets a value where the masking function needs one, or reformats a date so it matches the Date Format.
Chunk Size	Sets a column chunk size suited to a very large table.
Disable Constraints, Disable Indexes, Disable Triggers	Turns these on for the column to speed up masking.
Active	Sets the column to False (exclude column) when it should not be masked, such as a key column.

Run Settings

These apply to the whole configuration rather than a single column:

Setting	What a suggestion does
Disable Constraints, Disable Indexes, Disable Triggers	Turns these on to speed up masking.
Date Format	Sets a Date Format when one is needed and missing.
Global Chunk Size	Sets a Global Chunk Size when one is missing.

Finding Labels

Each suggestion is tagged with its finding type (Performance or Configuration) and a label describing the issue:

Label	What it means	What to do
Performance	A change that would make masking this column or table faster.	Accept the suggestion.
Config error	A required setting is missing or doesn't fit, such as a missing lookup, fixed value, Date Format or Global Chunk Size.	Accept the suggestion, or correct the setting by hand.
Duplicate entry	The same column appears more than once in the configuration.	Check the conditions that separate the entries.
Do not mask	The column is a key column or should not be masked.	Accept the suggestion to exclude the column.
Uniqueness	The column holds unique values, so the masked values must stay unique too.	Accept the suggestion to switch to a unique-value masking function.
Not assessable	There wasn't enough profile information to check the column.	Re-run the profile scan, then review again.
Free-text blob	The column holds free text, which is faster to replace with a fixed value.	Accept the suggestion, or keep your current setting.
Not profiled	The column isn't in the latest profile results, so it wasn't reviewed.	Re-run the profile scan, then review again.

note

Some suggestions only describe a problem and can't be applied automatically. These have no Accept button. Fix them by editing the column in Pattern Details.

Accepting, Dismissing and Applying Suggestions

1. Select a column and click Accept to include its suggestion. The button changes to Selected. Click Dismiss if you don't want it.

2. To work faster, use the Select menu (All performance, All configuration, Everything or Clear selection), or select a group heading and use Accept all, Dismiss all or Restore.

   

3. Click Apply selected. The accepted values are written into Pattern Details, and each changed row is marked with a robot icon.

4. Click Save on the Mask Configuration.

Important

Applying suggestions does not save them. You must click Save on the Mask Configuration to keep the changes.

Keyboard shortcuts: use ↑ and ↓ to move between columns, Space to accept and Del to dismiss.

Dismissed Suggestions

A dismissed suggestion is struck through in the list. Its detail pane notes that it comes back only if the recommendation changes; click Undo to bring it back.

Dismissed suggestions are not shown again in later reviews of the same data source unless the suggestion changes. To see them, click Show dismissed earlier in the toolbar. They are marked Dismissed before, and you can still accept them.

Execute Masking - Single Data Source

To execute a new masking request, navigate to Data Management Hub > Execution Console.

Here, you can select your data source then press View. Scroll down to the Execution Details section, select Mask in the first dropdown menu, and your chosen mask configuration in the second dropdown menu. The Execute button can be used to then run a mask job or View Results to simply view the results from the previously ran masking jobs.

Execute Masking - Connection Groups

In order to run a mask scan on a connection group, firstly ensure TDM Queue Manager is turned on. Then, navigate to Data Management Hub > Execution Console > Connection Group.

Here, select your Connection Group from the first dropdown menu, then select Mask for your scan type. After selecting all the fields, click on View.

You can then select the mask configurations from the Data Configuration column for the specific Data Sources.

In order to run a bulk mask scan, select all the data sources that you want to execute the scan on and click on Execute.

Then, scroll to the bottom of the page to the Execution Details section to look at the scan status, and access the logs and reports.

Masking Log

In the table below, Log Button can be used to view logs for the execution.

Note: The Log window refreshes every 15 seconds if the mask is currently executing.

A currently executing Mask can also be Cancelled by clicking the Cancel button.

Masking Report - Single Data Source

Successfully completed masks will generate a Mask Report which can be viewed by clicking the Report button.

This can also be searched on or exported out to Excel or CSV.

Masking Report Table

Type	Description
Connection Name	The name of the data connection.
Schema	The schema under which the table resides.
Table Name	The name of the table containing the data.
Column	The specific column within the table.
Data Type	The type of data stored in the column (e.g., string, integer, date).
Data Length	The maximum length of the data in the column.
Masking Pattern Used	The pattern applied to mask the data in the column.
Masking Lookup Used	Lookup table or source used during the masking process.
Masking Parameters Used	Parameters applied during the masking process. TDM 3.4 or earlier
Flag Column	The column used to indicate a specific condition.
Flag Condition	Condition that determines when the flag is applied.
Flag String	The string value that represents the flag.
Masking Status	The current status of the masking operation (e.g., completed, in progress).
Scan Duration (Seconds)	The duration taken to complete the scan, measured in seconds.
Error Details	Details of any errors encountered for that specific column.

Masking Report - Connection Groups

For grouped connections, each data source in the group will generate a Mask Report if successful, which can be viewed by clicking the Report button.

In order export the reports in bulk, navigate to the Execution Console section in the Connection Group tab, and select the data sources to include in the bulk report.

Click Generate Report, then select the required date range. For option details and step-by-step instructions, see Generating Single TDM Report.

Important: Databases and files have different report structures, therefore their reports cannot be generated together in bulk. When selecting the data sources to include in the bulk report, make sure that files and database connections are not selected together.

Feature Support Table

The table below outlines the masking features supported by various data source connection types:

Connection Type	Conditional Masking	Cross Table Masking	Chunking
Amazon Redshift	✅	✅	✅
CouchDB	❌	❌	❌
DB2	✅	❌	✅
Informix	✅	❌	✅*
MariaDB	✅	✅	✅
MongoDB	❌	❌	❌
MSSQL	✅	✅	✅
MySQL	✅	✅	✅
Oracle	✅	✅	✅
PostgreSQL	✅	✅	✅
Sybase	✅	❌	❌
Teradata	✅	❌	✅
File - AVRO	✅	N/A	❌
File - Delimited	✅	N/A	✅
File - Fixed Width	✅	N/A	✅
File - JSON	✅	N/A	✅
File - ORC	✅	N/A	❌
File - Parquet	✅	N/A	❌
File - XML	✅	N/A	❌

Note: Normal masking is supported across ALL data source connection types.

*Informix Chunking: Chunking relies on the rowid pseudo-column, which is only available on non-fragmented tables by default. Fragmented tables must be created or altered WITH ROWIDS to support chunked masking; otherwise the tool falls back to a single-statement update. See Chunking and ROWID Awareness for details.