Redirection (computing): Difference between revisions

Revision as of 07:08, 14 October 2017

Revision as of 00:38, 12 October 2018

Line 10:

===Basic===

Typically, the [[syntax]] of these characters is as follows, using <code>&lt;</code> to redirect input, and <code>&gt;</code> to redirect output. <syntaxhighlight lang="bash" inline>command1 > file1</syntaxhighlight> executes <tt>command1</tt>, placing the output in <tt>file1</tt>, as opposed to displaying it at the terminal, which is the usual destination for standard output. This will [[Clobbering|clobber]] any existing data in <tt>file1</tt>.

Using <syntaxhighlight lang="bash" inline>command1 < file1</syntaxhighlight> executes <tt>command1</tt>, with <tt>file1</tt> as the source of input, as opposed to the [[Computer keyboard|keyboard]], which is the usual source for standard input.

<source lang=bash>

command1 > file1

</source>

<syntaxhighlight lang="bash" inline>command1 < infile > outfile</syntaxhighlight> combines the two capabilities: <tt>command1</tt> reads from <tt>infile</tt> and writes to <tt>outfile</tt>

executes <tt>command1</tt>, placing the output in <tt>file1</tt>, as opposed to displaying it at the terminal, which is the usual destination for standard output. This will [[Clobbering|clobber]] any existing data in <tt>file1</tt>.

Using

<source lang=bash>

command1 < file1

</source>

executes <tt>command1</tt>, with <tt>file1</tt> as the source of input, as opposed to the [[Computer keyboard|keyboard]], which is the usual source for standard input.

<source lang=bash>

command1 < infile > outfile

</source>

combines the two capabilities: <tt>command1</tt> reads from <tt>infile</tt> and writes to <tt>outfile</tt>

===Variants===

To append output to the end of the file, rather than clobbering it, the <code>&gt;&gt;</code> operator is used: <syntaxhighlight lang="bash" inline>command1 >> file1</syntaxhighlight>.

<source lang=bash>

command1 >> file1

</source>

To read from a stream literal (an inline file, passed to the standard input), one can use a [[here document]], using the <code>&lt;&lt;</code> operator:

<sourcesyntaxhighlight lang="bash">

tr a-z A-Z << END_TEXT

one two three

uno dos tres

END_TEXT

</syntaxhighlight>

</source>

To read from a string, one can use a [[here string]], using the <code>&lt;&lt;&lt;</code> operator: <syntaxhighlight lang="bash" inline>tr a-z A-Z <<< "one two three"</syntaxhighlight>, or:

<syntaxhighlight lang="bash">

To read from a string, one can use a [[here string]], using the <code>&lt;&lt;&lt;</code> operator:

<source lang=bash>

tr a-z A-Z <<< "one two three"

</source>

or:

<source lang=bash>

NUMBERS="one two three"

tr a-z A-Z <<< "$NUMBERS"

</syntaxhighlight>

</source>

==Piping==

Programs can be run together such that one program reads the output from another with no need for an explicit intermediate file:

[[Image:Pipeline.svg|thumb|A pipeline of three programs run on a text terminal]]

Programs can be run together such that one program reads the output from another with no need for an explicit intermediate file. <syntaxhighlight lang="bash" inline>command1 | command2</syntaxhighlight> executes <tt>command1</tt>, using its output as the input for <tt>command2</tt> (commonly called [[Pipeline (Unix)|piping]], with the "|" character being known as "pipe").

<source lang="bash">

command1 | command2

</source>

executes <tt>command1</tt>, using its output as the input for <tt>command2</tt> (commonly called [[Pipeline (Unix)|piping]], with the "|" character being known as "pipe").

The two programs performing the commands may run in parallel with the only storage space being working buffers (Linux allows up to 64K for each buffer) plus whatever work space each command's processing requires. For example, a "sort" command is unable to produce any output until all input records have been read, as the very last record received just might turn out to be first in sorted order. Dr. Alexia Massalin's experimental operating system, [[Synthesis kernel#Massalin.27s Synthesis kernel|Synthesis]], would adjust the priority of each task as they ran according to the fullness of their input and output buffers.

Line 71 ⟶ 42:

This produces the same end result as using two redirects and a temporary file, as in:

<sourcesyntaxhighlight lang="bash">

command1 > tempfile

command2 < tempfile

rm tempfile

</syntaxhighlight>

</source>

But here, <tt>command2</tt> does not start executing until <tt>command1</tt> has finished, and a sufficiently large scratch file is required to hold the intermediate results as well as whatever work space each task required. As an example, although DOS allows the "pipe" syntax, it employs this second approach. Thus, suppose some long-running program "Worker" produces various messages as it works, and that a second program, TimeStamp copies each record from ''stdin'' to ''stdout'', prefixed by the system's date and time when the record is received. A sequence such as <syntaxhighlight inline lang="bash">Worker | TimeStamp > LogFile.txt</syntaxhighlight> would produce timestamps only when Worker had finished, merely showing how swiftly its output file could be read and written.

Worker | TimeStamp > LogFile.txt

Would produce timestamps only when Worker had finished, merely showing how swiftly its output file could be read and written.

A good example for command piping is combining <code>[[echo (command)|echo]]</code> with another command to achieve something interactive in a non-interactive shell, e.g. <syntaxhighlight lang="bash" inline>echo -e 'user\npass' | ftp localhost</syntaxhighlight>. This runs the [[File Transfer Protocol|ftp]] client with input <tt>user</tt>, press <tt>return</tt>, then <tt>pass</tt>.

<source lang=bash>

echo -e 'user\npass' | ftp localhost

</source>

This runs the [[File Transfer Protocol|ftp]] client with input <tt>user</tt>, press <tt>return</tt>, then <tt>pass</tt>.

In casual use, the initial step of a pipeline is often <code>cat</code> or <code>echo</code>, reading from a file or string. This can often be replaced by input indirection or a [[here string]], and use of cat and piping rather than input redirection is known as [[useless use of cat]]. For example, the following commands:

<sourcesyntaxhighlight lang="bash">

cat infile | cmd

echo $string | cmd

echo -e 'user\npass' | ftp localhost

</syntaxhighlight>

</source>

can be replaced by:

<sourcesyntaxhighlight lang="bash">

cmd < infile

cmd <<< $string

ftp localhost <<< $'user\npass'

</syntaxhighlight>

</source>

As <code>echo</code> is often a shell-internal command, its use is not as criticized as cat, which is an external command.

Line 119 ⟶ 82:

|}

For example, <syntaxhighlight lang="bash" inline>command1 2> file1</syntaxhighlight> executes <tt>command1</tt>, directing the [[Standard streams#Standard error (stderr)|standard error]] stream to <tt>file1</tt>.

For example:

<source lang="bash">

command1 2> file1

</source>

executes <tt>command1</tt>, directing the [[Standard streams#Standard error (stderr)|standard error]] stream to <tt>file1</tt>.

In shells derived from [[C shell|csh]] (the [[C shell]]), the syntax instead appends the <tt>&</tt> (ampersand) character to the redirect characters, thus achieving a similar result. The reason for this is to distinguish between a file named '1' and stdout, i.e. 'cat file 2>1' vs 'cat file 2>&1'. In the first case, stderr is redirected to a file named '1' and in the second, stderr is redirected to stdout.

Another useful capability is to redirect one standard file handle to another. The most popular variation is to merge [[Standard streams#Standard error (stderr)|standard error]] into [[Standard streams#Standard output (stdout)|standard output]] so error messages can be processed together with (or alternately to) the usual output. Example:For example, <syntaxhighlight lang="bash" inline>find / -name .profile > results 2>&1</syntaxhighlight> will try to find all files named <tt>.profile</tt>. Executed without redirection, it will output hits to [[stdout]] and errors (e.g. for lack of privilege to traverse protected directories) to [[stderr]]. If standard output is directed to file <tt>results</tt>, error messages appear on the console. To see both hits and error messages in file <tt>results</tt>, merge [[stderr]] (handle 2) into [[stdout]] (handle 1) using '''<tt>2>&1</tt>'''.

<source lang="bash">

find / -name .profile > results 2>&1

</source>

will try to find all files named <tt>.profile</tt>. Executed without redirection, it will output hits to [[stdout]] and errors (e.g. for lack of privilege to traverse protected directories) to [[stderr]]. If standard output is directed to file <tt>results</tt>, error messages appear on the console. To see both hits and error messages in file <tt>results</tt>, merge [[stderr]] (handle 2) into [[stdout]] (handle 1) using '''<tt>2>&1</tt>''' .

<!-- *** This is nonsense. Redirection works perfectly fine, just not as you may have expected. ***

It's possible use <tt>2>&1</tt> before "<tt>></tt>" but it doesn't work. In fact, when the interpreter reads <tt>2>&1</tt>, it doesn't know yet where standard output is redirected and then standard error isn't merged.

-->

If the merged output is to be piped into another program, the file merge sequence '''<tt>2>&1</tt>''' must precede the pipe symbol, thus:

<source lang="bash">

find / -name .profile 2>&1 | less

</source>

A simplified but non-POSIX conforming form of the command:

<source lang="bash">

command > file 2>&1

</source>

is (not available in Bourne Shell prior to version 4, final release, or in the standard shell [[Debian Almquist shell]] used in Debian/Ubuntu):

<source lang="bash">

command &>file

</source>

If the merged output is to be piped into another program, the file merge sequence '''<tt>2>&1</tt>''' must precede the pipe symbol, thus, <syntaxhighlight lang="bash" inline>find / -name .profile 2>&1 | less</syntaxhighlight>

or:

A simplified but non-POSIX conforming form of the command, <syntaxhighlight lang="bash" inline>command > file 2>&1</syntaxhighlight> is (not available in Bourne Shell prior to version 4, final release, or in the standard shell [[Debian Almquist shell]] used in Debian/Ubuntu): <syntaxhighlight lang="bash" inline>command &>file</syntaxhighlight> or <syntaxhighlight lang="bash" inline>command >&file</syntaxhighlight>.

<source lang="bash">

command >&file

</source>

<u>'''NOTE:'''</u> It is possible to use <code>2>&1</code> before "<code>></code>" but the result is commonly misunderstood.

The rule is that any redirection sets the handle to the output stream independently.

So "<code>2>&1</code>" sets handle <code>2</code> to whatever handle <code>1</code> points to, which at that point usually is ''stdout''.

Then "<code>></code>" redirects handle <code>1</code> to something else, e.g. a file, but it does '''not''' change handle <code>2</code>, which still points to ''stdout''.

In the following example. standard output is written to ''file'', but errors are redirected from stderr to stdout, i.e. sent to the screen.

In the following example, standard output is written to ''file'', but errors are redirected from stderr to stdout, i.e. sent to the screen: <syntaxhighlight lang="bash" inline>command 2>&1 > file</syntaxhighlight>.

<source lang="bash">

command 2>&1 > file

</source>

To write both errors and standard output to ''file'', the order should be reversed. Standard output would first be redirected to the file, then stderr would additionally be redirected to the stdout handle that has already been changed to point at the file: <syntaxhighlight lang="bash" inline>command > file 2>&1</syntaxhighlight>.

<source lang="bash">

command > file 2>&1

</source>

==Chained pipelines==

The redirection and piping tokens can be chained together to create complex commands. For example, <syntaxhighlight lang="bash" inline>sort infile | uniq -c | sort -n > outfile</syntaxhighlight> sorts the lines of <tt>infile</tt> in lexicographical order, writes unique lines prefixed by the number of occurrences, sorts the resultant output numerically, and places the final output in <tt>outfile</tt>. This type of construction is used very commonly in [[shell script]]s and [[batch file]]s.

The redirection and piping tokens can be chained together to create complex commands. For example:

<source lang="bash">

sort infile | uniq -c | sort -n > outfile

</source>

sorts the lines of <tt>infile</tt> in lexicographical order, writes unique lines prefixed by the number of occurrences, sorts the resultant output numerically, and places the final output in <tt>outfile</tt>. This type of construction is used very commonly in [[shell script]]s and [[batch file]]s.

==Redirect to multiple outputs==

The standard command <tt>[[tee (command)|tee]]</tt> can redirect output from a command to several destinations:<syntaxhighlight lang="bash" inline>ls -lrt | tee xyz</syntaxhighlight>. This directs the file list output to both standard output and the file <tt>xyz</tt>.

<source lang="bash">

ls -lrt | tee xyz

</source>

This directs the file list output to both standard output and the file <tt>xyz</tt>.

==See also==