Pipeline (Unix)

pipespipepipelinepipelinespipingUnix pipepipedUnix pipespipeliningUnix pipelines
In Unix-like computer operating systems, a pipeline is a mechanism for inter-process communication using message passing.wikipedia
159 Related Articles

Douglas McIlroy

Doug McIlroyM. Douglas McIlroyMcIlroy
The concept of pipelines was championed by Douglas McIlroy at Unix's ancestral home of Bell Labs, during the development of Unix, shaping its toolbox philosophy. The pipeline concept was invented by Douglas McIlroy and first described in the man pages of Version 3 Unix.
McIlroy is best known for having originally developed Unix pipelines, software componentry and several Unix tools, such as spell, diff, sort, join, graph, speak, and tr.

Unix

UNIX operating systemAT&T UnixUnix-like
The concept of pipelines was championed by Douglas McIlroy at Unix's ancestral home of Bell Labs, during the development of Unix, shaping its toolbox philosophy. This feature of Unix was borrowed by other operating systems, such as Taos and MS-DOS, and eventually became the pipes and filters design pattern of software engineering.
Unix systems are characterized by various concepts: the use of plain text for storing data; a hierarchical file system; treating devices and certain types of inter-process communication (IPC) as files; and the use of a large number of software tools, small programs that can be strung together through a command-line interpreter using pipes, as opposed to using a single monolithic program that includes all of the same functionality.

Named pipe

named pipesmkfifoFIFO
This differs from named pipes, where messages are passed to or from a pipe that is named by making it a file, and remains after the processes are completed.
In computing, a named pipe (also known as a FIFO for its behavior) is an extension to the traditional pipe concept on Unix and Unix-like systems, and is one of the methods of inter-process communication (IPC).

Standard streams

standard outputstandard inputstdout
A pipeline is a set of processes chained together by their standard streams, so that the output text of each process (stdout) is passed directly as input (stdin) to the next one.
When a command is executed via an interactive shell, the streams are typically connected to the text terminal on which the shell is running, but can be changed with redirection or a pipeline.

Vertical bar

pipe|vertical line
In all usage one writes the commands in sequence, separated by the ASCII vertical bar character " " (which, for this reason, is often called "pipe character").
A pipe is an inter-process communication mechanism originating in Unix, which directs the output (standard out and, optionally, standard error) of one process to the input (standard in) to another.

C shell

cshChistory
In the csh shell, for instance, using " " instead of " " signifies that the standard error stream should also be merged with the standard output and fed to the next process.
Like all Unix shells, it supports filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration.

Unix philosophy

core architectural difference between Windows and Linuxdesign philosophy of Unixdo one thing, do it well" philosophy
The concept of pipelines was championed by Douglas McIlroy at Unix's ancestral home of Bell Labs, during the development of Unix, shaping its toolbox philosophy.
McIlroy, then head of the Bell Labs Computing Sciences Research Center, and inventor of the Unix pipe, summarized the Unix philosophy as follows:

File descriptor

file handlefile handlesfilehandle
The pipe ends appear to be normal, anonymous file descriptors, except that they have no ability to seek.
In Unix and related computer operating systems, a file descriptor (FD, less frequently fildes) is an abstract indicator (handle) used to access a file or other input/output resource, such as a pipe or network socket.

Anonymous pipe

pipe unnamed
This article is about anonymous pipes, where data written by one process is buffered by the operating system until it is read by the next process, and this uni-directional channel disappears when the processes are completed.
Pipelines are supported in most popular operating systems, from Unix and DOS onwards, and are created using the " " character in many shells.

Unix shell

shellshellssh
McIlroy noticed that much of the time command shells passed the output file from one program as input to another.
All Unix shells provide filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration.

Netcat

ncsocatNcat
Tools like netcat and socat can connect pipes to TCP/IP sockets.
That is because pipes are unidirectional.

Ken Thompson

Kenneth Lane ThompsonK. ThompsonKen L. Thompson
His ideas were implemented in 1973 when ("in one feverish night", wrote McIlroy) Ken Thompson added the system call and pipes to the shell and several utilities in Version 3 Unix.
Eventually, the tools developed by Thompson became the Unix operating system: Working on a PDP-7, a team of Bell Labs researchers led by Thompson and Ritchie, and including Rudd Canaday, developed a hierarchical file system, the concepts of computer processes and device files, a command-line interpreter, pipes for easy inter-process communication, and some small utility programs.

Research Unix

Version 10 UnixVersion 5 UnixVersion 3 Unix
The pipeline concept was invented by Douglas McIlroy and first described in the man pages of Version 3 Unix.

Dartmouth Time Sharing System

Dartmouth Time-Sharing SystemDTSSDartmouth College Timesharing System
Although developed independently, Unix pipes are related to, and were preceded by, the 'communication files' developed by Ken Lochner in the 1960s for the Dartmouth Time Sharing System.
They significantly antedated Unix pipes, as design documents put their conceptual origin sometime in 1967, and were described briefly in a 1969 conference:

Tee (command)

tee
It is primarily used in conjunction with pipes and filters.

Redirection (computing)

redirectionredirectredirecting
executes command1, using its output as the input for command2 (commonly called piping, with the "|" character being known as "pipe").

Pipeline (software)

pipelinepipelinespipes and filters
This feature of Unix was borrowed by other operating systems, such as Taos and MS-DOS, and eventually became the pipes and filters design pattern of software engineering.

Iteratee

An enumerator in the context of iterateesIteratees
In a sense, a distant predecessor of the notion of an enumerator pushing data into a chain of one or more iteratees, was the pipeline concept in operating systems.

Unix-like

*nixUnixlike
In Unix-like computer operating systems, a pipeline is a mechanism for inter-process communication using message passing.

Operating system

operating systemsOScomputer operating system
In Unix-like computer operating systems, a pipeline is a mechanism for inter-process communication using message passing.

Inter-process communication

interprocess communicationIPCinterapplication communication
In Unix-like computer operating systems, a pipeline is a mechanism for inter-process communication using message passing.

Process (computing)

processprocessesprocessing
A pipeline is a set of processes chained together by their standard streams, so that the output text of each process (stdout) is passed directly as input (stdin) to the next one.

Concurrency (computer science)

concurrencyconcurrentconcurrently
The first process is not completed before the second is started, but they are executed concurrently.