High Efficiency Image File Format

High Efficiency Image File Format (HEIF) specifies the storage of individual images as well as image sequences in a single container file. It is an MPEG standard and defined by part 12 of MPEG-H (ISO/IEC 23008-12).

High Efficiency Image File Format (HEIF)
Filename extension	.heif, .heic
Internet media type	image/heif, image/heic, image/heif-sequence, image/heic-sequence
Developed by	Moving Picture Experts Group (MPEG)
Type of format	Image Container Format
Standard	ISO/IEC CD 23008-12
Website	http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format

HEIF specification defines the means of storing High Efficiency Video Codec (HEVC) encoded intra images and HEVC encoded image sequences in which inter prediction is applied in a constrained manner.

HEIF files are compatible with the ISO Base Media File Format (ISOBMFF, ISO/IEC 14496-12) and can also include other media streams, such as timed text and audio.

History

The requirements and main use cases of HEIF were defined in 2013 ^[1][2].The technical development of the specification took about 1.5 years and was finalized in summer 2015^[3].

Use Cases

Some main use cases supported by HEIF are as follows^[1][2]:

• Storage of single or multiple images and their thumbnails,
• Storage of burst photos,
• Support for simultaneous capture of video and still images, i.e. storing still images and timed image sequences into the same file,
• Efficient representation of animations and cinemagraphs,
• Storage of focal and exposure stacks into the same container file, 
• Storage of images derived from other images of the file, either as derived images represented by non-destructive image editing operations or as pre-computed derived images,
• Support for storing other media, such as audio and timed text, into the same container file with timed image sequences and synchronizing their playback. 

As HEVC provides support for various chroma formats and sample fidelities up to lossless coding, the format can serve the whole spectrum of use cases from today's consumer devices storing images typically at 8 bits per sample to high-end professional devices with sample fidelity and dynamic range requirements going all the way up to 16 bits per sample. Some usage scenarios that are facilitated by HEIF are described below:

Digital cameras and smartphones:

In order to save storage space [TODO: x-ref to compression results], HEIF-encapsulated HEVC-coded images can be used for compressing the full-resolution images while keeping a lower-resolution JPEG copy (e.g. at 4K resolution or below) for on-screen displaying purposes.

Digital cameras and smartphones can use HEIF to achieve convenient one-file packaging of burst photos, focal stacks, and exposure stacks. Similarly, simultaneously captured video and still images can be stored in the same HEIF file. HEIF also enables storage of any image collections into a single file, hence they can be shared easily.

Web pages and Internet-connected image applications:

The picture element of HTML5.1 provides the capability of indicating multiple alternatives for the same image, out of which the web browser can select the one that best suits its purpose. A motivation for web pages and connected applications to start using HEIF is to reduce the web page and/or image content download times [TODO: x-ref to compression results].

Image editing:

Changing of the orientation and cropping are basic features of HEIF and require no re-encoding of the images. Additionally, HEIF introduces a framework for non-destructive editing operations, which can be specified by external specifications. This feature can be used by image editing applications so that the editing instructions are kept in the same file as the original image.

Features

In ISOBMFF, a continuous or timed media/metadata stream forms a track, whereas static media or metadata are stored as items. Consequently, HEIF has the following basic design:

• Still images are stored as items. All image items are independently encoded and do not depend on any other item in their decoding. Any number of image items can be included in the same file.
• Image sequences are stored as tracks. An image sequence track can be indicated to be displayed either as a timed sequence or in a non-timed manner, such as a gallery or album of images. When there is coding dependency between images, an image sequence track is used instead of image items.
• A file may contain both image items and image sequence tracks along with other media. For example, it is possible to create a HEIF file, which includes image items or image sequence tracks conforming to HEIF, and audio and/or timed text tracks conforming to any derivative format of the ISOBMFF

Keeping the above-mentioned design principles in mind, HEIF files can store the following types of data^[4]:

• Image Items: storage of individual images, image properties and thumbnail(s).
• Image Derivations: derived images are generated during run-time based on descriptions such as rotation, grid and overlay. These images depend on other images stored in the HEIF file. The storage overhead of derived images is small.
• Image Sequences: storage of multiple time-related and/or temporally predicted images (like a burst-photo shot or cinemagraph animation), their properties and thumbnails. Different prediction options can be used in order to exploit the temporal and spatial similarities between the images. Hence, file sizes can be drastically reduced even when tens of images are stored in the same HEIF file.
• Auxiliary Image Items: storage of an image data which complements another image item. An alpha plane or a depth map are examples for such images. This data is not displayed as such, but used in various forms to complement another image item.
• Image Metadata: storage of EXIF, XMP and similar metadata which accompany the images stored in the HEIF file.

Examples of above-mentioned entities can be found here.

Comparison

Below table shows a comparison of different image formats^[4]:

Comparison of the features of some image file formats

	HEIF (.heic)	JPEG/Exif	PNG	GIF (89a)	WebP	JPEG-XR / TIFF	JPEG-XR / JPX	BPG
Formats and extensibility
Base container file format	ISOBMFF	TIFF	-	-	RIFF	TIFF	- 4	-
Lossy compression	Yes (HEVC)	Yes (JPEG)	No	No	Yes (VP8)	Yes	Yes	Yes (HEVC10)
Lossless compression	Yes (HEVC)	Yes (TIFF Rev 6.0)	Yes (PNG)1	Yes (GIF)1	Yes (VP8L)	Yes	Yes	Yes (HEVC10)
Extensible to other coding formats	Yes	Yes8	No	No	No	Yes8	Yes5	No
Metadata format (on top of internal)	Exif, XMP, MPEG-7	Exif	-	-	Exif, XMP	Exif, XMP	JPX, (XMP)6	Exif, XMP
Extensible to other metadata formats	Yes	No	No	No	No	No	Yes (XML-based)	Yes
Other media types (audio, text, etc.)	Yes	Audio2	No	No	No	No	Yes7	No
Multi-picture features
Multiple images in the same file	Yes	No11	No	Yes3	Yes3	No	Yes	Yes9
Image sequences / animations	Yes	No	No	Yes	Yes	No	Yes	Yes
Image coding	Yes	No	No	No	No	No	No	Yes
Derived images
Multiple-of-90-degree rotations	Yes	Yes	No	No	No	Yes	Yes	No
Cropping	Yes	No	No	No	No	No	Yes	No
Tiling/overlaying	Yes	No	No	No	Yes	No	Yes	No
Extensible to other editing operations	Yes	No	No	No	No	No	No	No
Auxiliary picture information
Transparency (alpha plane)	Yes	No	Yes	No12	Yes	Yes	Yes	Yes
Thumbnail image	Yes	Yes	No	No	No	Yes	Yes	Yes

1 In GIF and indexed color PNG encoding, lossy color quantization is applied while the color-quantized image is losslessly compressed.

2 PCM, µ-Law PCM and ADPCM encapsulated in RIFF WAV

3 Only for animations and tiling/overlaying

4 JPX is a box-structured format compatible with ISOBMFF. However, only the File Type box is common in JPX and ISOBMFF.

5 Encapsulation of JPEG-2000 and JPEG-XR have been specified for JPX container. Mappings for other codecs could be similarly specified.

6 JPX (ITU-T T.800 and T.801) specifies an own metadata schema, but is capable of carrying an XML formatted metadata, such as XMP.

7 JPX can contain media complying with ISOBMFF (or derivatives thereof). No accurate synchronization between JPX animations and other media.

8 TIFF as a container format facilitates extensions to other coding formats.

9 Only for animations, thumbnails, and alpha planes. Non-timed image collections not supported.

10 HEVC Main 4:4:4 16 Still Picture profile, Level 8.5, with additional constraints

11 Can be enabled through the MP extension

12 A palette index for fully transparency can be specified.

The following references are used in the comparison:

Image format	Version or date	Reference and/or URL
HEIF (.heic)	03/2015	ISO/IEC 23008-12 http://mpeg.chiariglione.org/standards/mpeg-h/image-file-format/draft-text-isoiec-fdis-23008-12-carriage-still-image-and-image
JPEG		ISO/IEC 10918-1 | ITU-T Rec. T.81 http://www.w3.org/Graphics/JPEG/itu-t81.pdf
Exif		http://www.cipa.jp/std/documents/e/DC-008-2012_E.pdf
PNG		http://www.w3.org/TR/PNG/
GIF	89a	http://www.w3.org/Graphics/GIF/spec-gif89a.txt
WebP		https://developers.google.com/speed/webp/docs/riff_containerVP8L: https://developers.google.com/speed/webp/docs/webp_lossless_bitstream_specification
JPEG-XR		ISO/IEC 29199-2 | ITU-T Rec. T.832 ISO/IEC 15444-2 | ITU-T Rec. T.801 (for JPX)
BPG	0.9.5	http://bellard.org/bpg/bpg_spec.txt

Note: It is acknowledged that a summary such as that in the comparison table might be somewhat incomplete when it comes to features of different formats. For example, the table does not cover some of the extensions of JPEG.

TODO: Add compression performance.

Syntax and Structure

HEIF conforms to the ISO Base Media File Format (ISO/IEC 14496-12). Image items are stored in root level meta box and image sequences are stored as video media tracks.

A file may contain both image items and image sequence tracks along with other media. For example, it is possible to create a file, which includes image items or image sequence tracks conforming to HEIF, and audio and timed text tracks conforming to any derivative format of the ISOBMFF.

File:HEIF and ISO base media file.png

HEIF and ISO Base Media File

Files conforming to ISOBMFF consist of a sequence of data structures called boxes, each containing a four-character code (4CC) indicating the type of the box, the size of the box in terms of bytes, and the payload of the box. Boxes may be nested, i.e. a box may contain other boxes. ISOBMFF and HEIF specify constraints on the allowed box order and hierarchy.

Image Items

The roles of image items as specified in HEIF are listed and described below. Note that a single image can be associated with more than one role.

Role	Description
coded image	A coded representation of an image.
derived image	An image that is represented in a file by an indicated operation to indicated input images and can be obtained by performing the operation to the input images.
cover image (a.k.a. primary item)	A representative image of the image items and image sequence tracks of the file. The cover image should be displayed when no other information is available on the preference to display the image items of the file. The file can have only one cover image.
thumbnail image	A smaller-resolution representation of a master image. A master image can have multiple thumbnail images (e.g. different resolutions).
auxiliary image	An image that complements a master image. For example, an alpha plane or a depth map. Can assist in displaying the master image but is not typically displayed as such.
master image	An image that is not a thumbnail image or an auxiliary image. It typically represents a full-resolution displayable image.
hidden image	An image that should never be displayed. Can be present in the file for example as an input image for a derived image.
pre-derived coded image	A coded image that has been derived from other images. For example, a high dynamic range image derived from an exposure-bracketed set of images. The input images used for derivation are linked to the final pre-derived coded image by referencing. No derivation operation is defined.

HEIF allows the storage of image properties which are shared among different image items. There are two types of properties: descriptive and transformative. Descriptive properties provide information about the image item without applying modifications on the image itself. Transformative properties provide information about the transformative modification that needs to be done on the image item. The HEIF standard specifies transformative properties for rotation by 90, 180, and 270 degrees and cropping. In addition to descriptive image properties, image items can optionally be characterized with metadata items, the format of which follows e.g. Exif or XMP metadata.

Derived Images

Derived images enable non-destructive image editing, where the original coded images are kept in the file, while new images, called derived images, can be introduced by specifying a transformation operation that is applied to one or more input images. HEIF specifies the generic structures used for storing derived images as items as well as a few specific types of derived images. The item type of a derived image item indicates the transformation operation, while the item payload contains the input parameters to the operation. The input images can be coded images or derived images. The derived image types specified in the HEIF standard are listed below. Other types may be specified in other documents or later versions of the HEIF standard.

Derived Image Items
Name	Item 4CC	Description
Identity transformation	iden	Cropping and/or rotation by 90, 180, or 270 degrees, imposed through the respective transformative properties.
Image overlay	iovl	Overlaying any number of input images in indicated order and locations onto the canvas of the output image.
Image grid	grid	Reconstructing a grid of input images of the same width and height.

References

1. "Requirements for still image coding using HEVC | MPEG". mpeg.chiariglione.org. Retrieved 2015-11-25.
2. "Requirements for HEVC image sequences | MPEG". mpeg.chiariglione.org. Retrieved 2015-11-25.
3. "Evidence motivates MPEG to launch new standardization effort for HDR".
4. M. M. Hannuksela, E. B. Aksu, V. K. Malamal Vadakital, J. Lainema. "Overview of the High Efficiency Image File Format". JCT-VC document JCTVC-V0072, Oct. 2015.