Content deleted Content added
Realbadwolf (talk | contribs) No edit summary |
Link suggestions feature: 3 links added. |
||
| (115 intermediate revisions by 26 users not shown) | |||
Line 1:
{{Short
{{Infobox programming language
| name = Cuneiform
| logo =
| screenshot =
| caption = Screenshot of the Cuneiform editor and command line shell
| paradigm = [[functional programming|functional]], [[Scientific workflow system|scientific workflow]]
Line 8 ⟶ 9:
| founder =
| status = Active
| latest release version = 3.0.
| latest release date = {{release date|2018|
| latest preview version =
| latest preview date =
| typing =
| implementations =
| dialects =
| influenced_by = [[
| influenced =
| operating system = [[Linux]], [[
| programming language = [[Erlang (programming language)|Erlang]]
| license = [[Apache License]] 2.0
| website = {{URL|
| file_ext = .cfl
| year = 2013
Line 26 ⟶ 27:
'''Cuneiform''' is an [[open source software|open-source]] [[Scientific workflow system|workflow language]]
for large-scale scientific data analysis.<ref>{{Cite web|url=https://github.com/joergen7/cuneiform|title = Joergen7/Cuneiform|website = [[GitHub]]|date = 14 October 2021}}</ref><ref>{{Cite journal
| last1 = Brandt | first1 = Jörgen
| last2 = Bux | first2 = Marc N.
Line 37 ⟶ 38:
| url = http://ceur-ws.org/Vol-1330/paper-03.pdf
}}</ref>
It is a
Cuneiform is implemented in distributed [[Erlang (programming language)|Erlang]]. If run in distributed mode it drives a [[POSIX]]-compliant distributed file system like [[Gluster]] or [[Ceph (software)#CephFS|Ceph]] (or a [[Filesystem in Userspace|FUSE]] integration of some other file system, e.g., [[Apache Hadoop#HDFS|HDFS]]). Alternatively, Cuneiform scripts can be executed on top of [[HTCondor]] or [[Apache Hadoop|Hadoop]].<ref>{{cite web|title=Scalable Multi-Language Data Analysis on Beam: The Cuneiform Experience by Jörgen Brandt|url=http://beta.erlangcentral.org/videos/scalable-multi-language-data-analysis-on-beam-the-cuneiform-experience-by-jorgen-brandt/#.WBLlE2hNzIU|website=Erlang Central|access-date=28 October 2016|archive-url=https://web.archive.org/web/20161002222350/http://beta.erlangcentral.org/videos/scalable-multi-language-data-analysis-on-beam-the-cuneiform-experience-by-jorgen-brandt/#.WBLlE2hNzIU|archive-date=2 October 2016|url-status=dead}}</ref><ref>
{{Cite journal
| last1 = Bux | first1 = Marc
Line 52 ⟶ 55:
| year = 2015
| url = http://www.vldb.org/pvldb/vol8/p1892-bux.pdf
| doi = 10.14778/2824032.2824094
}}</ref><ref>
{{Cite journal
Line 79 ⟶ 83:
| url = http://www.di.fc.ul.pt/~bessani/publications/dmah15-bbc.pdf
}}
</ref><ref>{{cite web|title=Scalable Multi-Language Data Analysis on Beam: The Cuneiform Experience|url=http://www.erlang-factory.com/euc2016/jorgen-brandt|website=Erlang-factory.com|
Cuneiform is influenced by the work of Peter Kelly who proposes functional programming as a model for scientific workflow execution.<ref>{{cite journal
| last1 = Kelly | first1 = Peter M.
| last2 = Coddington | first2 = Paul D.
| last3 = Wendelborn | first3 = Andrew L.
| year = 2009
| title = Lambda calculus as a workflow model
| journal = Concurrency and Computation: Practice and Experience
| volume = 21
| issue = 16
| pages = 1999–2017
| doi = 10.1002/cpe.1448| s2cid = 10833434
}}</ref><ref>
{{cite journal
| title = Workflows and extensions to the Kepler scientific workflow system to support environmental sensor data access and analysis
| last1 = Barseghian | first1 = Derik
| last2 = Altintas | first2 = Ilkay
| last3 = Jones | first3 = Matthew B.
| last4 = Crawl | first4 = Daniel
| last5 = Potter | first5 = Nathan
| last6 = Gallagher | first6 = James
| last7 = Cornillon | first7 = Peter
| last8 = Schildhauer | first8 = Mark
| last9 = Borer | first9 = Elizabeth T.
| last10 = Seabloom | first10 = Eric W.
| journal = Ecological Informatics
| volume = 5
| number = 1
| pages = 42–50
| year = 2010
| doi = 10.1016/j.ecoinf.2009.08.008 | s2cid = 16392118 | url = https://escholarship.org/content/qt2q46n1tp/qt2q46n1tp.pdf?t=nivnuu
}}
</ref>
In this, Cuneiform is distinct from related workflow languages based on [[dataflow programming]] like [[Swift (parallel scripting language)|Swift]].<ref>
{{cite journal
| title = Nextflow enables reproducible computational workflows
| last1 = Di Tommaso | first1 = Paolo
| last2 = Chatzou | first2 = Maria
| last3 = Floden | first3 = Evan W
| last4 = Barja | first4 = Pablo Prieto
| last5 = Palumbo | first5 = Emilio
| last6 = Notredame | first6 = Cedric
| journal = Nature Biotechnology
| volume = 35
| number = 4
| pages = 316–319
| year = 2017
| doi = 10.1038/nbt.3820 | pmid = 28398311 | s2cid = 9690740 }}
</ref>
==External software integration==
External tools and libraries (e.g., [[R (programming language)|R]] or [[Python (programming language)|Python]] libraries) are integrated
Currently supported foreign programming languages are:
{{div col}}
* [[Bash (Unix shell)|Bash]]
* [[Elixir (programming language)|Elixir]]
* [[Erlang (programming language)|Erlang]]
* [[Java (programming language)|Java]]
* [[JavaScript]]
* [[MATLAB]]
* [[GNU Octave]]
Line 95 ⟶ 151:
* [[R (programming language)|R]]
* [[Racket (programming language)|Racket]]
{{div col end}}
Foreign language support for [[AWK]] and [[gnuplot]] are planned additions.
==Type system==
Cuneiform provides a simple, statically checked type system.<ref>
{{ cite journal
| title = Computation semantics of the functional scientific workflow language Cuneiform
| last1 = Brandt | first1 = Jörgen
| last2 = Reisig | first2 = Wolfgang
| last3 = Leser | first3 = Ulf
| journal = [[Journal of Functional Programming]]
| volume = 27
| year = 2017
| doi = 10.1017/S0956796817000119 | s2cid = 6128299 }}
</ref> While Cuneiform provides lists as [[compound data type]]s it omits traditional list accessors (head and tail) to avoid the possibility of runtime errors which might arise when accessing the empty list. Instead lists are accessed in an all-or-nothing fashion by only mapping or folding over them. Additionally, Cuneiform omits (at the organizational level) arithmetics which excludes the possibility of division by zero. The omission of any partially defined operation allows to guarantee that runtime errors can arise exclusively in foreign code.
===Base data types===
As base data types Cuneiform provides Booleans, strings, and files. Herein, files are used to exchange data in arbitrary format between foreign functions.
===Records and pattern matching===
Cuneiform provides [[Record_(computer_science)|record]]s (structs) as compound data types. The example below shows the definition of a variable <code>r</code> being a record with two fields <code>a1</code> and <code>a2</code>, the first being a string and the second being a Boolean.
<syntaxhighlight lang="swift">
let r : <a1 : Str, a2 : Bool> =
<a1 = "my string", a2 = true>;
</syntaxhighlight>
Records can be accessed either via projection or via [[pattern matching]]. The example below extracts the two fields <code>a1</code> and <code>a2</code> from the record <code>r</code>.
<syntaxhighlight lang="swift">
let a1 : Str = ( r|a1 );
let <a2 = a2 : Bool> = r;
</syntaxhighlight>
===Lists and list processing===
Furthermore, Cuneiform provides lists as compound data types. The example below shows the definition of a variable <code>xs</code> being a file list with three elements.
<syntaxhighlight lang="erlang">
let xs : [File] =
['a.txt', 'b.txt', 'c.txt' : File];
</syntaxhighlight>
Lists can be processed with the for and fold operators. Herein, the for operator can be given multiple lists to consume list element-wise (similar to <code>for/list</code> in [[Racket (programming language)|Racket]], <code>mapcar</code> in [[Common Lisp]] or <code>zipwith</code> in [[Erlang (programming language)|Erlang]]).
The example below shows how to map over a single list, the result being a file list.
<syntaxhighlight lang="ruby">
for x <- xs do
process-one( arg1 = x )
: File
end;
</syntaxhighlight>
The example below shows how to zip two lists the result also being a file list.
<syntaxhighlight lang="ruby">
for x <- xs, y <- ys do
process-two( arg1 = x, arg2 = y )
: File
end;
</syntaxhighlight>
Finally, lists can be aggregated by using the fold operator. The following example sums up the elements of a list.
<syntaxhighlight lang="text">
fold acc = 0, x <- xs do
add( a = acc, b = x )
end;
</syntaxhighlight>
==Parallel execution==
Cuneiform is a purely functional language, i.e., it does not support [[Reference (computer science)|mutable references]]. In the consequence, it can use subterm-independence to divide a program into parallelizable portions. The Cuneiform scheduler distributes these portions to worker nodes. In addition, Cuneiform uses a [[Evaluation strategy#Call by name|Call-by-Name evaluation strategy]] to compute values only if they contribute to the computation result. Finally, foreign function applications are [[Memoization|memoized]] to speed up computations that contain previously derived results.
For example, the following Cuneiform program allows the applications of <code>f</code> and <code>g</code> to run in parallel while <code>h</code> is dependent and can be started only when both <code>f</code> and <code>g</code> are finished.
{{pre|1=
let output-of-f : File = f();
let output-of-g : File = g();
h( f = output-of-f, g = output-of-g );
}}
The following Cuneiform program creates three parallel applications of the function <code>f</code> by mapping <code>f</code> over a three-element list:
{{pre|1=
let xs : [File] =
['a.txt', 'b.txt', 'c.txt' : File];
for x <- xs do
f( x = x )
: File
end;
}}
Similarly, the applications of <code>f</code> and <code>g</code> are independent in the construction of the record <code>r</code> and can, thus, be run in parallel:
{{sxhl|lang=erlang|1=
let r : <a : File, b : File> =
<nowiki><a = f(), b = g()></nowiki>;
}}
==Examples==
A hello-world script:
<syntaxhighlight lang="ruby">
in Bash *{
out="Hello $person"
}*
( greet( person
</syntaxhighlight>
This script defines a task <code>greet</code> in [[
The
Applying
Command line tools can be integrated by defining a task in [[Bash (Unix shell)|Bash]]:
<syntaxhighlight lang="ruby">
in Bash *{
sorted=sorted.bam
samtools sort -m 2G $bam -o $sorted
}*
</syntaxhighlight>
In this example a task <code>
It calls the tool [[SAMtools]], consuming an input file, in
==Release
{| class="wikitable"
Line 141 ⟶ 290:
! Version !! Appearance !! Implementation Language !! Distribution Platform !! Foreign Languages
|-
!
|
| [[
| [[Apache Hadoop]]
| Bash,
|-
! 2.0.x
Line 159 ⟶ 302:
| Bash, BeanShell, Common Lisp, MATLAB, GNU Octave, Perl, Python, R, Scala
|-
!
|
| [[
| [[HTCondor]], [[Apache Hadoop]]
| Bash
|-
! 3.0.x
| Feb. 2018
| [[Erlang (programming language)|Erlang]]
| Distributed Erlang
| Bash, Erlang, Java, MATLAB, GNU Octave, Perl, Python, R, Racket
|}
Cuneiform's surface syntax was revised
===Version 1===
In its first draft published in May 2014, Cuneiform was closely related to [[Make (software)|Make]] in that it constructed a static [[data dependency]] graph which the interpreter traversed during execution. The major difference to later versions was the lack of conditionals, recursion, or static type checking.
The following example script downloads a reference genome from an FTP server.
Line 199 ⟶ 348:
===Version 2===
[[File:Cf screenshot.jpg|thumb|Swing-based editor and REPL for Cuneiform 2.0.3]]
The second draft of the Cuneiform surface syntax, first published in March 2015, remained in use for three years outlasting the transition from Java to Erlang as Cuneiform's implementation language. Evaluation differs from earlier approaches in that the interpreter reduces a query expression instead of traversing a static graph. During the time the surface syntax remained in use the interpreter was formalized and simplified which resulted in a first specification of Cuneiform's semantics. The syntax featured conditionals. However, Booleans were encoded as lists, recycling the empty list as Boolean false and the non-empty list as Boolean true. Recursion was added later as a byproduct of formalization. However, static type checking was introduced only in Version 3.
The following script decompresses a zipped file and splits it into evenly sized partitions.
<pre>
Line 217 ⟶ 369:
fileLst;
</pre>
===Version 3===
The current version of Cuneiform's surface syntax, in comparison to earlier drafts, is an attempt to close the gap to mainstream functional programming languages. It features a simple, statically checked type system and introduces records in addition to lists as a second type of compound [[data structure]]. Booleans are a separate base data type.
The following script untars a file resulting in a file list.
<pre>
def untar( tar : File ) -> <fileLst : [File]>
in Bash *{
tar xf $tar
fileLst=`tar tf $tar`
}*
let hg38Tar : File =
'hg38/hg38.tar';
let <fileLst = faLst : [File]> =
untar( tar = hg38Tar );
faLst;
</pre>
==References==
Line 229 ⟶ 402:
[[Category:Linux programming tools]]
[[Category:Hadoop]]
[[Category:Statically typed programming languages]]
[[Category:Cross-platform free software]]
| |||