Kolmogorov complexity: Difference between revisions

Revision as of 02:58, 7 September 2004

Revision as of 21:57, 23 September 2004

Line 1:

[[zh:算法信息论]]

'''Algorithmic information theory''' is a field of study which attempts to capture the concept of complexity by using tools from theoretical computer science. The chief idea is to define the complexity (or '''Kolmogorov complexity''') of a [[string]] as the length of the shortest program which, when run without any input, outputs that string. Strings that can be produced by short programs are considered to be not very complex. This notion is surprisingly deep and can be used to state and prove impossibility results akin to [[Gödel's incompleteness theorem]] and [[halting problem|Turing's halting problem]].

The field was developed by [[Andrey Kolmogorov]], [[Ray Solomonoff]] and [[Gregory Chaitin]] starting in the late [[1960s]]. There are several

Line 7:

To formalize the above definition of complexity, one has to specify exactly what types of programs are allowed. Fortunately, it doesn't really matter: one could take a particular notation for [[Turing machine|Turing machines]], or [[Lisp programming language|LISP]] programs, or [[Pascal programming language|Pascal]] programs, or [[Java virtual machine]] bytecode.
If we agree to measure the lengths of all objects consistently in [[bit|bits]], then the resulting notions of complexity will only differ by a constant factorterm: if ''IK''₁(''s'') and ''IK''₂(''s'') are the complexities of the string ''s'' according to two different programming languages ''L''₁ and ''L''₂, then there areis constantsa ''C'' andconstant ''Dc'' (which only depend on the languages chosen, but not on ''s'') such that

:<math>I_1K_1(s) \le C \cdot I_2K_2 (s) + Dc </math>

Here, ''Dc'' is the length in bits of an interpreter for ''L''₂ written in ''L''₁, and ''C'' describes the overhead of storing programs written in ''L''₂ as part of programs in ''L''₁.

(One technical requirement is that it must be possible to embed arbitrary binary data into programs without

delimeters, e.g. by providing such data on "standard input" and considering all bits read

from this stream as part of the program.)

In the following, we will fix one definition and simply write ''IK''(''s'') for the complexity of the string ''s''.

The first surprising result is that ''IK''(''s'') cannot be computed: there is no general [[algorithm]] which takes a string ''s'' as input and produces the number ''IK''(''s'') as output. The proof is a formalization of the amusing [[Berry paradox]]: "Let ''n'' be the smallest number that cannot be defined in fewer than twenty English words. Well, I just defined it in fewer than twenty English words."

It is however straightforward to compute upper bounds for ''IK''(''s''): simply [[data compression|compress]] the string ''s'' with some method, implement the corresponding decompressor in the chosen language, concatenate the decompressor to the compressed string, and measure the resulting string's length.

The next important result is about the [[randomness]] of strings. Most strings are complex in the sense that they cannot be significantly compressed: ''IK''(''s'') is not much smaller than |''s''|, the length of ''s'' in bits. The precise statement is as follows: there is a constant ''K'' (which depends only on the particular specification of "program" used in the definition of complexity) such that for every ''n'',
the [[probability]] that a random string of length ''sn'' has complexity less than |''sn''| - ''nk'' is smaller than ''K'' 2^-''nk''. The proof is a [[counting argument]]: you count the programs and the strings, and compare. This theorem is the justification for Mike Goldman's challenge in the [http://www.faqs.org/faqs/compression-faq/ comp.compression FAQ]:

:''I will attach a prize of $5,000 to anyone who successfully meets this challenge. First, the contestant will tell me HOW LONG of a data file to generate. Second, I will generate the data file, and send it to the contestant. Last, the contestant will send me a decompressor and a compressed file, which will together total in size less than the original data file, and which will be able to restore the compressed file to the original state.''

Line 23 ⟶ 28:

Now for Chaitin's incompleteness result: though we know that most strings are complex in the above sense, the fact that a specific string is complex can never be proven (if the string's length is above a certain threshold). The precise formalization is as follows. Suppose we fix a particular consistent [[axiomatic system]] for the [[natural number|natural numbers]], say [[Peanos axioms|Peano's axioms]]. Then there exists a constant ''L'' (which only depends on the particular axiomatic system and the choice of definition of complexity) such that there is no string ''s'' for which the statement

:<math> IK(s) \ge L </math>

can be proven within the axiomatic system (even though, as we know, the vast majority of those statements must be true). Again, the proof of this result is a formalization of Berry's paradox.