Funnel Stage Example
The Funnel stage is one of the processing stage. It copies multiple input data sets to a single output data set. This operation is useful for combining separate data sets into a single large data set. The stage can have any number of input links and a single output link.
The Funnel stage can operate in one of three modes:
· Continuous Funnel combines the records of the input data in the order it arrives. It takes one record from each input link in turn. If data is not available on an input link, the stage skips to the next link rather than waiting.
· Sort Funnel combines the input records in the order defined by the value(s) of one or more key columns and the order of the output records is determined by these sorting keys.
· Sequence Funnel copies all records from the first input data set to the output data set, then all the records from the second input data set, and so on.
Note: Metadata for all the inputs must be identical.
Here we are having Employee data as an input in 3 different files and we are going to perform Funnel operation on them as below.
Design the job as below:
Input Data:1. Employee Input1:
2. Employee Input2:
3. Employee Input3:
Open Funnel properties window by doble click on Funnel stage or by Right clickàselecct properties. Under Stage tab, select Properties tab. Here we can select the option ‘Funnel Type’, select Continous Funnel option as shown below:
Next step àUnder Output tab provide the mapping from source to target.
Next, configure the target dataset file to capture the result of various funnel operations.
Save and compile the job. Run the job to see the results.
1. Continuous Funnel:
2. Sequence Funnel:
3. Sort Funnel Ascending: