---

Note

Data flow systems involve a series of transformation on sets of data. Data is transformed from one form into another using different types of operations.

This section will explore designing a data flow system through the pipe and filter architectural style, a specific type of data flow architecture.

Pipes and Filters

A pipe and filter architectural style has independent entities called filters, which perform transformations on data input they receive, and pipes, which serve as connectors for the stream of data being transformed.

Below is a simple diagram outlining how a pipe and filter architecture works. Data flows in one direction, beginning at the data source and moving through a series of pipes and filters, to end at the data target. Changes to data are done sequentially from filter to filter. Each pipe serves as a connector for the stream of data. It transmits the output of one filter as input for the next filter. Each filter performs its local transformation on the data input it receives from the previous pipe and then sends the output on to the next pipe.

The order in which the filters transform data may change the end result. This is similar to mathematics, the order in which addition or multiplication occurs in an expression can change the end result.

UML component diagrams are a good option for showing the details of pipe and filter architecture.

The data source can be connected to a number of different filters. Each filter has a required interface to receive input from data sources and other filters. They also have a provided interface that will output transformed data. The data target is connected to receive the fully transformed result.

Advantages to using the pipe and filter architecture include:

• Loose and flexible coupling of components. In particular, this applies to filters. Each filter runs independently of other filters, which allows the easy addition of new filters, of moving filters around, or of changing individual filters in a system.

• Filters can be treated as black boxes. Users of the system do not need to know the inner workings of each filter.

• Reusability. Each filter can be called and used over and over again with different inputs. Filters can be repeatedly used in many different applications for different purposes.

Disadvantages to using the pipe and filter architecture include:

• Reduced performance due to excessive overheads in filters. Each filter must receive input, parse that input into some data structure, perform transformations, and send data out.

  If every filter must do this, it is a lot of overhead, and the performance of the system will be affected as more and more filters are added.

• Filters may get overloaded with massive amounts of data to process. If one filter is working to process large amounts of data, other filters may be waiting for inputs.

• This architecture cannot be used for interactive applications. Applications that require rapid responses will not find this architecture suitable, as data transfers and transformations take time depending on the amount of data being transmitted.

Pipe and filter architectures are popular in use, such as in the UNIX operating system for text-based utilities. If different data sets need to be manipulated in a system, the pipe and filter architecture is a good choice.