Funnel Stage Example

The Funnel stage is one of the processing stage. It copies multiple input data sets to a single output data set. This operation is useful for combining separate data sets into a single large data set. The stage can have any number of input links and a single output link.

The Funnel stage can operate in one of three modes:

· Continuous Funnel combines the records of the input data in the order it arrives. It takes one record from each input link in turn. If data is not available on an input link, the stage skips to the next link rather than waiting.

· Sort Funnel combines the input records in the order defined by the value(s) of one or more key columns and the order of the output records is determined by these sorting keys.

· Sequence Funnel copies all records from the first input data set to the output data set, then all the records from the second input data set, and so on.

Note: Metadata for all the inputs must be identical.

Example:

Here we are having Employee data as an input in 3 different files and we are going to perform Funnel operation on them as below.

Design the job as below:

DS job - Funnel Stage Example

Input Data:

1. Employee Input1:
input1 - Funnel Stage Example

2. Employee Input2:

input2 - Funnel Stage Example

3. Employee Input3:

input3 - Funnel Stage Example

Open Funnel properties window by doble click on Funnel stage or by Right clickàselecct properties. Under Stage tab, select Properties tab. Here we can select the option ‘Funnel Type’, select Continous Funnel option as shown below:

property Continous - Funnel Stage Example

Next step àUnder Output tab provide the mapping from source to target.

funnel output mapping - Funnel Stage Example

Next, configure the target dataset file to capture the result of various funnel operations.

Save and compile the job. Run the job to see the results.

Funnel Operations:

1. Continuous Funnel:

property Continous 1 - Funnel Stage Example

Output Data:

continous FunnelResult - Funnel Stage Example

     2. Sequence Funnel:

sequence funnel - Funnel Stage Example

Output Data:

sequence funnel output - Funnel Stage Example

     3. Sort Funnel Ascending:

sort funnel property - Funnel Stage Example

Output Data:

sort funnel output - Funnel Stage Example
4. Sort Funnel Descending:
sort funnel property desc - Funnel Stage Example

Output Data:

sort funnel desc output - Funnel Stage Example