Skillfully Using Output Variables to Enhance DolphinScheduler Workflow Flexibility and Maintainability

Output variables are a core mechanism in DolphinScheduler task scheduling for achieving data flow and task collaboration. By explicitly defining and passing parameters, they solve issues such as cross-node data sharing and priority conflicts while supporting complex process orchestration (such as sub-processes and conditional branches). Proper use of output variables can significantly improve workflow flexibility and maintainability. This article will introduce important output variables in DolphinScheduler and how to use them. 1. Use of Single Quotes ('), Double Quotes ("), and Backticks (`) in Shell Scripts In Shell scripts, single quotes ('), double quotes ("), and backticks (`) serve different purposes. Understanding their differences and applications is crucial for writing and debugging Shell scripts. 1.1 Single Quotes (') Purpose: Full quoting, used to protect all characters within the string, preventing variable substitution or command substitution. Characteristics: All characters inside the quotes are output as-is without interpretation. VAR="world" echo 'Hello, $VAR' # Output: Hello, $VAR 1.2 Double Quotes (") Purpose: Partial quoting, used to protect most characters within the string while allowing variable and command substitution. Characteristics: Variables and commands inside the quotes are interpreted, while other characters remain unchanged. VAR="world" echo "Hello, $VAR" # Output: Hello, world 1.3 Backticks (`) Purpose: Command substitution, used to execute the command inside the quotes and return its output as the result. Characteristics: The command inside the quotes is executed, and its result replaces the backtick content. Note: Backticks are an older method of command substitution. It is recommended to use $() instead. 1.4 Recommended Use of $() for Command Substitution More modern and readable Supports nesting, whereas backticks make nesting difficult 1.5 Use Cases Single Quotes: Use when you do not want any part of the string to be interpreted, such as for regular expressions or special characters. Double Quotes: Use when the string contains variables or command substitutions. Backticks / $(): Use when executing commands and using their output. 2. Output Variables and File Transfer in DolphinScheduler 2.1 Shell Output Variables 2.1.1 Process Overview 2.1.2 Task Configuration taskA Note: Here, ${lines_num} is directly replaced by the Worker. taskB 2.1.3 Output Results Focus on taskB output 2.2 Shell File Transfer 2.2.1 Process Overview 2.2.2 Task Configuration fileUploadTask fileDownloadTask 2.2.3 Output Results 2.3 SQL Task Output Variables The mode involves outputting results from an SQL task and using them in a Shell task. 2.3.1 Process Overview 2.3.2 Task Configuration sqlOutVarTask readOutVarTask 2.3.3 Output Results Conclusion Output variables in DolphinScheduler are a powerful mechanism for improving workflow flexibility and maintainability. By mastering their usage, users can create more dynamic, efficient, and scalable workflows. By applying these techniques, you can maximize the potential of DolphinScheduler in handling complex data integration and process automation tasks.

Mar 20, 2025 - 03:31

Skillfully Using Output Variables to Enhance DolphinScheduler Workflow Flexibility and Maintainability

Output variables are a core mechanism in DolphinScheduler task scheduling for achieving data flow and task collaboration. By explicitly defining and passing parameters, they solve issues such as cross-node data sharing and priority conflicts while supporting complex process orchestration (such as sub-processes and conditional branches). Proper use of output variables can significantly improve workflow flexibility and maintainability. This article will introduce important output variables in DolphinScheduler and how to use them.

1. Use of Single Quotes ('), Double Quotes ("), and Backticks (`) in Shell Scripts

In Shell scripts, single quotes ('), double quotes ("), and backticks (`) serve different purposes. Understanding their differences and applications is crucial for writing and debugging Shell scripts.

1.1 Single Quotes (')

Purpose: Full quoting, used to protect all characters within the string, preventing variable substitution or command substitution.
Characteristics: All characters inside the quotes are output as-is without interpretation.

VAR="world"
echo 'Hello, $VAR'  # Output: Hello, $VAR

1.2 Double Quotes (")

Purpose: Partial quoting, used to protect most characters within the string while allowing variable and command substitution.
Characteristics: Variables and commands inside the quotes are interpreted, while other characters remain unchanged.

VAR="world"
echo "Hello, $VAR"  # Output: Hello, world

1.3 Backticks (`)

Purpose: Command substitution, used to execute the command inside the quotes and return its output as the result.
Characteristics: The command inside the quotes is executed, and its result replaces the backtick content.
Note: Backticks are an older method of command substitution. It is recommended to use $() instead.

1.4 Recommended Use of `$()` for Command Substitution

More modern and readable
Supports nesting, whereas backticks make nesting difficult

1.5 Use Cases

Single Quotes: Use when you do not want any part of the string to be interpreted, such as for regular expressions or special characters.
Double Quotes: Use when the string contains variables or command substitutions.
Backticks / $(): Use when executing commands and using their output.

2. Output Variables and File Transfer in DolphinScheduler

2.1 Shell Output Variables

2.1.1 Process Overview

2.1.2 Task Configuration

taskA

Note: Here, ${lines_num} is directly replaced by the Worker.

taskB

2.1.3 Output Results

Focus on taskB output

2.2 Shell File Transfer

2.2.1 Process Overview

2.2.2 Task Configuration

fileUploadTask

fileDownloadTask

2.2.3 Output Results

2.3 SQL Task Output Variables

The mode involves outputting results from an SQL task and using them in a Shell task.

2.3.1 Process Overview

2.3.2 Task Configuration

sqlOutVarTask

readOutVarTask

2.3.3 Output Results

Conclusion

Output variables in DolphinScheduler are a powerful mechanism for improving workflow flexibility and maintainability. By mastering their usage, users can create more dynamic, efficient, and scalable workflows.

By applying these techniques, you can maximize the potential of DolphinScheduler in handling complex data integration and process automation tasks.