DataStage: What is Transformer Stage?

DataStage provides several stages to load the data into the data warehouse or data marts. The stages classified as General, Database, Development and Debugging, File, Processing, Real Time, etc. and the transforming stage is a processing stage. We will explore the different options available such as execution mode, Preserve partition.

Advanced tab of scenario properties:

On the Advanced tab, the following options are available to configure.

• run mode. The stage can be executed in parallel mode or in sequential mode. In parallel mode, the data is processed by the available nodes as specified in the configuration file and by any node constraints specified in the Advanced tab. In sequential mode, the data is processed by the driver node.
• combinability mode. This is Auto by default, which allows WebSphere DataStage to combine the operators underlying parallel stages to be executed in the same process if appropriate for this type of stage.
• keep partition. This is set to Spread By default, it establishes or deletes the partition according to what was established in the previous stage. You can also select Place gold Clear. If you select Placethe scenario will request that the next stage preserves the partition how is.
• Node pool and resource constraints. Select this option to restrict parallel execution to the node group(s) or resource group(s) specified in the grid. The grid allows you to choose options from populated dropdown lists from the configuration file.
• Node Map Constraint. Select this option to restrict parallel execution to nodes in a defined node map. You can define a node map by typing node numbers in the text box or by clicking the browse button to open the Available Nodes dialog and select nodes from there. In effect, you are defining a new nodegroup for this stage (in addition to any nodegroups defined in the configuration file).

Properties Surrogate Key Tab:

Select the Font Type field as Flat file gold database sequence

Transformation stage: entry page

Partition tab:

The Partitioning tab allows you to specify details about how the incoming data was partitioned or collected when it was entered into the Transformer stage. It also allows you to specify that the data should be sorted in the input.

By default, the Transformer stage will attempt to preserve partitioning of the incoming data or use its own partitioning method as dictated by the previous job stage.

The Partitioning tab also allows you to specify that data arriving at the input link must be sorted. Sorting is always performed within data partitions.. If the stage is partitioning the incoming data, the sorting occurs after the partition. If the stage is data collection, the sorting occurs before the collection. The availability of sorting depends on the chosen partitioning method.

• perform classification. Select this to specify that data entering the link should be sorted. Select the column or columns to sort from the Available list.
• Stable. Select this if you want to keep previously sorted data sets. This is the default value.
• Only. Select this to specify that if multiple records have identical sort key values, only one record is retained. If stable sorting is also set, the first record is retained.

Preserves sort order:

Select this if you know that the rows being input to the Transformer stage have been sorted and you want to preserve the sort order.

The transformer stage is a very important stage and we use it very often in the design of DataStage jobs. The various options available in the stage along with the options available for defining local variables, calling routines, and transformations and derivations make this stage unique among the stages available in DataStage.