Deduplicate between an uncertain number of columns：SQL VS SPL #15

A certain database table has multiple fields, each storing an email address, which may be null or duplicated. Now we need to merge multiple fields into one, remove null and duplicate values, and merge them with commas. In a database/computing platform that supports functions such as array, for example, in azure-databricks, SQL can be written as follows: SELECT concat_ws(',', array_distinct(array(Mail_1, Mail_2, Mail_3, Mail_4, Mail_5)), NULL) AS Mail FROM my_data SQL Server and other databases do not have an array function, and generally require indirect implementation using statements such as cross apply: SELECT STRING_AGG(DistinctEmails.Mail, ',') AS Mail FROM my_data CROSS APPLY ( SELECT DISTINCT Mail FROM (VALUES (Mail_1), (Mail_2), (Mail_3), (Mail_4), (Mail_5)) AS EmailList(Mail) WHERE Mail IS NOT NULL ) AS DistinctEmails GROUP BY my_data.Mail_1, my_data.Mail_2, my_data.Mail_3, my_data.Mail_4, my_data.Mail_5 SQL must write column names, which is inflexible. To deduplicate between an uncertain number of columns, it is necessary to dynamically generate column names using stored procedures and then execute SQL, which will make the architecture more complex. SPL does not need to write column names, and the code is the same for different data sources: Try.Demo A1： Load data. A2： Take the current record, convert it into a set, take unique values and remove null, and merge it with commas. ~ indicates the current record, function ID is used for deduplication, @0 means null is removed. esProc SPL is free,Download

Apr 7, 2025 - 09:45

Deduplicate between an uncertain number of columns：SQL VS SPL #15

A certain database table has multiple fields, each storing an email address, which may be null or duplicated.

Now we need to merge multiple fields into one, remove null and duplicate values, and merge them with commas.

In a database/computing platform that supports functions such as array, for example, in azure-databricks, SQL can be written as follows:

SELECT concat_ws(',', array_distinct(array(Mail_1, Mail_2, Mail_3, Mail_4, Mail_5)), NULL) AS Mail
FROM my_data

SQL Server and other databases do not have an array function, and generally require indirect implementation using statements such as cross apply:

SELECT 
    STRING_AGG(DistinctEmails.Mail, ',') AS Mail
FROM my_data
CROSS APPLY (
    SELECT DISTINCT Mail
    FROM (VALUES (Mail_1), (Mail_2), (Mail_3), (Mail_4), (Mail_5)) AS EmailList(Mail)
    WHERE Mail IS NOT NULL
) AS DistinctEmails
GROUP BY my_data.Mail_1, my_data.Mail_2, my_data.Mail_3, my_data.Mail_4, my_data.Mail_5

SQL must write column names, which is inflexible. To deduplicate between an uncertain number of columns, it is necessary to dynamically generate column names using stored procedures and then execute SQL, which will make the architecture more complex. SPL does not need to write column names, and the code is the same for different data sources:
Try.Demo

A1： Load data.

A2： Take the current record, convert it into a set, take unique values and remove null, and merge it with commas. ~ indicates the current record, function ID is used for deduplication, @0 means null is removed.

esProc SPL is free,Download