Climate and Forecast Metadata Conventions: Difference between revisions

Revision as of 14:56, 23 June 2010

Latest revision as of 23:52, 19 March 2025

(46 intermediate revisions by 29 users not shown)

Line 1:

The Climate and Forecast (CF) [[metadata]] conventions are conventions for the description of [[Earth sciences]] data, intended to promote the processing and sharing of [[data file]]s. The metadata defined by the CF conventions canare begenerally included in the same file as the data, (thus making the file "self-describing"). The conventions provide a definitive description of what the data values found in each [[netCDF]] variable representsrepresent, and of the spatial and temporal properties of the data, (including information about grids, such as [[grid cell]] bounds and cell averaging methods). This enables users of datafiles from different sources to decide which datavariables are comparable, and is a basis for building software applications with powerful [[data extraction]], grid remapping, [[data analysis]], and [[data visualization]] capabilities.

{{Newpage}}

==History and Evolutionevolution==

The Climate and Forecast (CF) [[metadata]] conventions are conventions for the description of [[Earth sciences]] data, intended to promote the processing and sharing of [[data file]]s. The metadata defined by the CF conventions can be included in the same file as the data (thus making the file "self-describing"). The conventions provide a definitive description of what the data in each variable represents, and of the spatial and temporal properties of the data (including information about grids, such as grid cell bounds and cell averaging methods). This enables users of data from different sources to decide which data are comparable, and is a basis for building software applications with powerful [[data extraction]], grid remapping, [[data analysis]], and [[data visualization]] capabilities.

The CF conventions were introduced in 2003, after several years of development by a collaboration that included staff from U.S. and European climate and weather laboratories .<ref>{{cite articlenews

==History and Evolution==

The CF conventions were introduced in 2003, after several years of development by a collaboration that included staff from U.S. and European climate and weather laboratories <ref>{{cite article

| author = Gregory, Jonathan

| year = 2003

| title = The CF metadata standard

| url = http://cf-pcmdicfconventions.llnl.govorg/Data/cf-documents/otheroverview/cf_overview_articlearticle.pdf

}}</ref>. The conventions contained generalizations and extensions to the earlier Cooperative Ocean/Atmosphere Research Data Service (COARDS) conventions <ref>{{cite web|url=http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html|title=Conventions for the Standardization of NetCDF files|date=May, 1995|access-date=2010-06-23|archive-url=https://web.archive.org/web/20100527095818/http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html|archive-date=2010-05-27|url-status=dead}}</ref> and the Gregory/Drach/Tett (GDT) conventions .<ref>{{cite web|url=http://www-pcmdi.llnl.gov/drach/GDT_convention.html|title=GDT netCDF conventions for climate data, version 1.3|date=March 14, 1999|access-date=June 23, 2010|archive-url=https://web.archive.org/web/20100610102527/http://www-pcmdi.llnl.gov/drach/GDT_convention.html|archive-date=June 10, 2010|url-status=dead}}</ref>. As the scope of the CF conventions grew along with its user base, the CF community adopted an [[open governance]] model .<ref>{{cite articlenews

| author = Lawrence, B.N.

| year = 2003

| title = Maintaining and Advancing the CF Standard for Earth System Science Community Data

| url = http://cf-pcmdi.llnl.gov/documents/white-papers/cf2_whitepaper_final.html

| url-status = dead

}}</ref>. The CF conventions are being considered by the [[NASA]] Standards Process Group (SPG) and others as more broadly applicable standards <ref>{{cite article

| archiveurl = https://web.archive.org/web/20100527133114/http://cf-pcmdi.llnl.gov/documents/white-papers/cf2_whitepaper_final.html/

| archivedate = 2010-05-27

}}</ref> In December 2008 the trio of standards, netCDF+CF+[[OPeNDAP]], was adopted by [[IOOS]] as a recommended standard (number 08-012) for the representation and transport of gridded data. The CF conventions are being considered by the [[NASA]] Standards Process Group (SPG) and others as more broadly applicable standards .<ref>{{cite articlenews

| author = Rew, Russ

| year = 2010

| title = CF Metadata Conventions

| url = http://wwwearthdata.esdswgnasa.orggov/sites/default/files/esdswg/spg/rfc/esds-rfc-021/ESDS-RFC-021-v0.01.pdf

| url-status = dead

| archiveurl = https://web.archive.org/web/20140404101632/https://earthdata.nasa.gov/sites/default/files/esdswg/spg/rfc/esds-rfc-021/ESDS-RFC-021-v0.01.pdf

| archivedate = 2014-04-04

}}</ref>

<ref>{{cite articlenews

| author = Domenico, Ben

| year = 2009

| title = Web Coverage Service (WCS) 1.1 extension for CF-netCDF 3.0 encoding

| url = http://xml.coverpages.org/OGC-09-018-WCS11-ExtensionCF-netCDF.pdf

}}</ref>.

==Applications and Useruser Basebase==

The CF conventions have been adopted by a wide variety of national and international programs and activities in the Earth sciences .<ref>[http://cf-pcmdi.llnl.gov/projects-and-groups-adopting-the-cf-conventions-as-their-standard Projects and Groups Adopting or Encouraging the CF-Conventions as a Standard] {{webarchive |url=https://web.archive.org/web/20100823061034/http://cf-pcmdi.llnl.gov/projects-and-groups-adopting-the-cf-conventions-as-their-standard |date=August 23, 2010 }}</ref>. For example, they were required for the [[climate model]] output data collected for [[Coupled model intercomparison project]]s, which are thewidely basisused offor the [[Intergovernmental Panel on Climate Change]] assessment reports .<ref>{{cite articlenews

| year = 2009

| title = Report of the 12th Session of the JSC/CLIVAR Working Group on Coupled Modelling (WGCM)

| url = http://wcrp.ipsl.jussieu.fr/Documents/WMP/136_WGCM12.pdf

}}</ref>.

They are promoted as an important element of scientific community coordination by the [[World Climate Research Programme]] .<ref>{{cite articlenews

| author = Kinter, James L. III

| year = 2005

| title = Data Issues for WCRP Weather and Climate Modeling

| url = http://wcrp.ipsl.jussieu.fr/Documents/WMP/WMP1_KinterTaylor.pdf

}}</ref> <ref>{{cite articlenews

| author = Taylor, Karl

| year = 2005

| title = Issues Related to the Dissemination of Climate Model Output

| url = http://wcrp.ipsl.jussieu.fr/Documents/WMP/Presentations/WMP1_2-7Taylor.pdf

}}</ref>. They are also leveragedused as a technical foundation for a number of software packages and data systems, including the Climate Model Output Rewriter (CMOR), which is post processing software for climate model data, and the [[Earth System Grid]], which distributes climate and other data .<ref>{{cite articlenews

| author = Doutriaux, Charles

| year = 2010

| title = Climate Model Output Rewriter

| url = http://esg-repo.llnl.gov/gitweb/?p=cmor.git;a=blob;f=Doc/cmor_users_guide.pdf

}}{{dead link|date=August 2017 |bot=InternetArchiveBot |fix-attempted=yes }}</ref><ref>{{cite news

}}</ref><ref>{{cite article

| author = Drach, Bob

| year = 2002

| title = Metadata Standards for Gridded Climate Data in the Earth System Grid

| url = http://archive.niees.ac.uk/talks/metadata/Bob_Drach_talk.ppt

| archive-url = https://web.archive.org/web/20040912230406/http://archive.niees.ac.uk/talks/metadata/Bob_Drach_talk.ppt

}}</ref> <ref>{{cite article

| url-status = dead

| archive-date = 2004-09-12

}}</ref><ref name="crwr.utexas">

{{cite news

| year = 2006

| title = NetCDF in ArcGIS 9.2

| url = http://www.crwr.utexas.edu/gis/gishydro06/Introduction/Presentations/netCDF.ppt

| access-date = 2010-06-23

}}</ref>. The CF conventions have also been used to describe the physical fields transferred between individual Earth system model [[software components]], such as atmosphere and ocean components, as the model runs

| archive-url = https://web.archive.org/web/20100614235111/http://www.crwr.utexas.edu/gis/gishydro06/Introduction/Presentations/netCDF.ppt

<ref>{{cite article

| archive-date = 2010-06-14

| url-status = dead

}}

}}</ref>. The CF conventions have also been used to describe the physical fields transferred between individual Earth system model [[software components]], such as atmosphere and ocean components, as the model runs

}}</ref>.<ref>{{cite articlejournal

| doi = 10.1007/s12145-008-0016-1

| author = Dunlap, R. L. Mark, S. Rugaber, V. Balaji, J. Chastang, L. Cinquini, C. DeLuca, D. Middleton, and S. Murphy

| year = 2008

| title = Earth system curator: metadata infrastructure for climate modeling

| url = http://www.springerlink.com/content/t781174802510364/fulltext.pdf

| journal = Earth Science Informatics

| volume = 1

| pagepages = 131-149131–149

| issue = 3–4

}}</ref>.

| doi-access = free

}}</ref>.

===Supported Datadata Typestypes===

CF is intended for use with [[state estimation]] and [[forecasting]] data, in the atmosphere, ocean, and other physical domains. It was designed primarily to address gridded data types such as [[numerical weather prediction]] model outputs and [[climatology]] data in which [[data binning]] is used to impose a regular structure.<ref name="crwr.utexas" /><ref>{{cite articlenews

| author = V. Balaji

| year = 2008

| title = Metadata Standards for Gridded Climate Data in the Earth System Grid

| url = http://esdswg.eosdis.nasa.gov/WG/SPG/pdf/esdswg-cf-balaji.pdf

| url-status = dead

}}</ref>

| archiveurl = https://web.archive.org/web/20100528100546/http://esdswg.eosdis.nasa.gov/WG/SPG/pdf/esdswg-cf-balaji.pdf

<ref>{{cite article

| archivedate = 2010-05-28

| year = 2006

}}</ref>. However, the CF conventions are also applicable to many classes of [[observational data]] and have been adopted by a number of groups for such applications.

| title = NetCDF in ArcGIS 9.2

| url = http://www.crwr.utexas.edu/gis/gishydro06/Introduction/Presentations/netCDF.ppt

}}</ref>. However, the CF conventions are also applicable to many classes of [[observational data]] and have been adopted by a number of groups for such applications.

===Supported Datadata Formatsformats===

CF originated as a standard for data written in [[NetCDFnetCDF]], but its structure is general and it has been adapted for use with other data formats. For example, using the CF conventions with [[Hierarchical Data Format]] data has been explored .<ref>{{cite articlenews

| author author1= Yang, M., |author2=R. Duerr, |author3=C. Lee | year = 2009

| year = 2009

| title = Investigation of using HDF5 archival information packages (AIP) to store NASA ECS data

| url = http://ams.confex.com/ams/89annual/techprogram/paper_148062.htm

| publisher = 25th Conference on International Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, 89th American Meteorological Society Meeting

}}</ref>.

==Design Principlesprinciples==

Several principles guide the development of CF conventions:

* Data should be self-describing, without external tables needed for interpretation.

Line 98 ⟶ 110:

* Redundancy should be avoided to prevent inconsistencies when writing data.

Specific CF metadata descriptors use values of attributes to represent

* [[Data provenance]]: <code>title</code>, <code>institution</code>, <code>contact</code>, <code>source</code> (e.g. model), <code>history</code> ([[audit trail]] of operations), <code>references</code>, <code>comment</code>

* Description of associated activity: <code>project</code>, <code>experiment</code>

* Description of data: <code>units</code>, <code>standard_name</code>, <code>long_name</code>, <code>auxiliary_variables</code>, <code>missing_value</code>, <code>valid_range</code>, <code>flag_values</code>, <code>flag_meanings</code>

Line 104 ⟶ 116:

* Meaning of grid cells: <code>cell_methods</code>, <code>cell_measures</code>, and climatological statistics.

A central element of the CF Conventions is the CF Standard Name Table. The CF Standard Name Table uniquely associates a standard name with each geophysical parameter in a [[data set]], where each name provides a precise description of physical quantities being represented. Note that this is the string value of the <code>standard_name</code> attribute, not the name of the parameter. The CF standard name table identifies over 1,000 physical quantities, each with a precise description and associated [[canonical units]]. Guidelines for construction of CF standard names are documented on the conventions web site.

As an example of the information provided by CF standard names, the entry for sea-level atmospheric pressure includes:

Line 110 ⟶ 122:

* description: sea_level means mean [[sea level]], which is close to the [[geoid]] in sea areas. [[Air pressure]] at sea level is the quantity often abbreviated as MSLP or PMSL.

* canonical units: Pa

==Software==

*[https://ncas-cms.github.io/cf-python/ CF-Python] is a data analysis package built on a complete implementation ([https://ncas-cms.github.io/cfdm/ CFDM]) of the CF conventions. The authors of the CFDM and CF-Python currently assert a desire to fully support all aspects of the CF conventions.

*[https://docs.unidata.ucar.edu/netcdf-java/current/userguide/ NetCDF-Java Library] parses CF Conventions and creates [https://web.archive.org/web/20141213021852/http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/CDM/index.html#CoordSys Coordinate System] objects from them

* [[Origin (data analysis software)|OriginPro]] version 2021b supports<ref> {{cite web|url=https://www.originlab.com/doc/Origin-Help/NetCDF-Importing-Processing| title=NetCDF Importing and Processing |publisher=originlab.com |access-date=2021-05-11}}</ref> netCDF CF Convention. Averaging can be performed during import to allow handling of large datasets in a GUI software.

* The [https://xarray.pydata.org xarray] Python library parses and decodes data stored according to CF Conventions.

* The [https://scitools-iris.readthedocs.io/en/stable/index.html Iris] Python library "draws heavily from the NetCDF CF Metadata Conventions as a source for its data model".<ref> {{cite web| url=https://scitools-iris.readthedocs.io/en/stable/further_topics/metadata.html| title=Metadata - Iris Documentation| access-date=2023-02-10}}</ref>

==References==

Line 115 ⟶ 134:

==External links==

*[httphttps://cf-pcmdiconvention.llnlgithub.govio/ CF Metadata Home Page]

*[https://web.archive.org/web/20110606124657/http://badc.nerc.ac.uk/help/formats/netcdf/index_cf.html The CF Metadata Convention (BADC page)]

*[https://web.archive.org/web/20110726034739/http://www.esdswg.org/spg/docindexfolder/standards-track-rfcs/ NASA Standards Process Group]

*[https://web.archive.org/web/20110717124506/http://marinemetadata.org/references/esdscfconventions Standard for the CF Metadata Conventions (Marine Metadata InteroperabiltyInteroperability Project page)]

*[https://web.archive.org/web/20110727140821/http://www.oceandatastandards.org/index.php?option=com_content&task=view&id=4&Itemid=8 Ocean Data Standards on Metadata]

<!--

[[Category:Metadata]]

[[Category:Earth Sciencessciences Metadatametadata Conventionsconventions]]

[[Category:Meteorological data and networks]]

[[Category:Science software]]

-->