# On-disk format ```{note} These docs are written for anndata 0.8+. Files written before this version may differ in some conventions, but will still be read by newer versions of the library. ``` AnnData objects are saved on disk to hierarchical array stores like [HDF5] (via {doc}`H5py `) and {mod}`zarr`. This allows us to have very similar structures in disk and on memory. As an example we’ll look into a typical `.h5ad`/ `.zarr` object that’s been through an analysis. The structures are largely equivalent, though there are a few minor differences when it comes to type encoding. (elements)= ## Elements `````{tab-set} ````{tab-item} HDF5 :sync: hdf5 ```python >>> import h5py >>> store = h5py.File("for-ondisk-docs/cart-164k-processed.h5ad", mode="r") >>> list(store.keys()) ['X', 'layers', 'obs', 'obsm', 'obsp', 'uns', 'var', 'varm', 'varp'] ``` ```` ````{tab-item} Zarr :sync: zarr ```python >>> import zarr >>> store = zarr.open("for-ondisk-docs/cart-164k-processed.zarr", mode="r") >>> list(store.keys()) ['X', 'layers', 'obs', 'obsm', 'obsp', 'uns', 'var', 'varm', 'varp'] ``` ```` ````` In general, `AnnData` objects are comprised of various types of elements. Each element is encoded as either an Array (or Dataset in hdf5 terminology) or a collection of elements (e.g. Group) in the store. We record the type of an element using the `encoding-type` and `encoding-version` keys in its attributes. For example, we can see that this file represents an `AnnData` object from its metadata: ```python >>> dict(store.attrs) {'encoding-type': 'anndata', 'encoding-version': '0.1.0'} ``` Using this information, we're able to dispatch onto readers for the different element types that you'd find in an anndata. (element)= ### Element Specification * An element can be any object within the storage hierarchy (typically an array or group) with associated metadata * An element MUST have a string-valued field `"encoding-type"` in its metadata * An element MUST have a string-valued field `"encoding-version"` in its metadata that can be evaluated to a version (anndata)= ### AnnData specification (v0.1.0) * An `AnnData` object MUST be a group. * The group's metadata MUST include entries: `"encoding-type": "anndata"`, `"encoding-version": "0.1.0"`. * An `AnnData` group MUST contain entries `"obs"` and `"var"`, which MUST be dataframes (though this may only have an index with no columns). * The group MAY contain an entry `X`, which MUST be either a dense or sparse array and whose shape MUST be (`n_obs`, `n_var`) * The group MAY contain a mapping `layers`. Entries in `layers` MUST be dense or sparse arrays which have shapes (`n_obs`, `n_var`) * The group MAY contain a mapping `obsm`. Entries in `obsm` MUST be sparse arrays, dense arrays, or dataframes. These entries MUST have a first dimension of size `n_obs` * The group MAY contain a mapping `varm`. Entries in `varm` MUST be sparse arrays, dense arrays, or dataframes. These entries MUST have a first dimension of size `n_var` * The group MAY contain a mapping `obsp`. Entries in `obsp` MUST be sparse or dense arrays. The entries first two dimensions MUST be of size `n_obs` * The group MAY contain a mapping `varp`. Entries in `varp` MUST be sparse or dense arrays. The entries first two dimensions MUST be of size `n_var` * The group MAY contain a mapping `uns`. Entries in `uns` MUST be an anndata encoded type. (dense-arrays)= ## Dense arrays Dense numeric arrays have the most simple representation on disk, as they have native equivalents in H5py {doc}`h5py:high/dataset` and {class}`zarr.Array`\ s. We can see an example of this with dimensionality reductions stored in the `obsm` group: `````{tab-set} ````{tab-item} HDF5 :sync: hdf5 ```python >>> store["obsm/X_pca"] ``` ```` ````{tab-item} Zarr :sync: zarr ```python >>> store["obsm/X_pca"] ``` ```` ````` ```python >>> dict(store["obsm"]["X_pca"].attrs) {'encoding-type': 'array', 'encoding-version': '0.2.0'} ``` (array)= ### Dense arrays specification (v0.2.0) * Dense arrays MUST be stored in an Array object * Dense arrays MUST have the entries `'encoding-type': 'array'` and `'encoding-version': '0.2.0'` in their metadata (sparse-arrays)= ## Sparse arrays Sparse arrays don’t have a native representations in HDF5 or Zarr, so we've defined our own based on their in-memory structure. Currently two sparse data formats are supported by `AnnData` objects, CSC and CSR (corresponding to {class}`scipy.sparse.csc_matrix` and {class}`scipy.sparse.csr_matrix` respectively). These formats represent a two-dimensional sparse array with three one-dimensional arrays, `indptr`, `indices`, and `data`. ```{note} A full description of these formats is out of scope for this document, but are [easy to find]. ``` We represent a sparse array as a `Group` on-disk, where the kind and shape of the sparse array is defined in the `Group`'s attributes: ```python >>> dict(store["X"].attrs) {'encoding-type': 'csr_matrix', 'encoding-version': '0.1.0', 'shape': [164114, 40145]} ``` The group contains three arrays: `````{tab-set} ````{tab-item} HDF5 :sync: hdf5 ```python >>> store["X"].visititems(print) data indices indptr ``` ```` ````{tab-item} Zarr :sync: zarr ```python >>> store["X"].visititems(print) data indices indptr ``` ```` ````` (csr_matrix)= (csc_matrix)= ### Sparse array specification (v0.1.0) * Each sparse array MUST be its own group * The group MUST contain arrays `indices`, `indptr`, and `data` * The group's metadata MUST contain: * `"encoding-type"`, which is set to `"csr_matrix"` or `"csc_matrix"` for compressed sparse row and compressed sparse column, respectively. * `"encoding-version"`, which is set to `"0.1.0"` * `"shape"` which is an integer array of length 2 whose values are the sizes of the array's dimensions (dataframes)= ## DataFrames DataFrames are saved as a columnar format in a group, so each column of a DataFrame is saved as a separate array. We save a little more information in the attributes here. ```python >>> dict(store["var"].attrs) {'_index': 'ensembl_id', 'column-order': ['highly_variable', 'means', 'variances', 'variances_norm', 'feature_is_filtered', 'feature_name', 'feature_reference', 'feature_biotype', 'mito'], 'encoding-type': 'dataframe', 'encoding-version': '0.2.0'} ``` These attributes identify the index of the dataframe, as well as the original order of the columns. Each column in this dataframe is encoded as its own array. `````{tab-set} ````{tab-item} HDF5 :sync: hdf5 ```python >>> store["var"].visititems(print) ensembl_id feature_biotype feature_biotype/categories feature_biotype/codes feature_is_filtered ... ``` ```` ````{tab-item} Zarr :sync: zarr ```python >>> store["var"].visititems(print) ensembl_id feature_biotype feature_biotype/categories feature_biotype/codes feature_is_filtered ... ``` ```` ````` ```python >>> dict(store["var"]["feature_name"].attrs) {'encoding-type': 'categorical', 'encoding-version': '0.2.0', 'ordered': False} >>> dict(store["var"]["feature_is_filtered"].attrs) {'encoding-type': 'array', 'encoding-version': '0.2.0'} ``` (dataframe)= ### Dataframe Specification (v0.2.0) * A dataframe MUST be stored as a group * The group's metadata: * MUST contain the field `"_index"`, whose value is the key of the array to be used as an index/ row labels * MUST contain encoding metadata `"encoding-type": "dataframe"`, `"encoding-version": "0.2.0"` * MUST contain `"column-order"` an array of strings denoting the order of column entries * The group MUST contain an array for the index * Each entry in the group MUST correspond to an array with equivalent first dimensions * Each entry SHOULD share chunk sizes (in the HDF5 or zarr container) (mappings)= ## Mappings Mappings are simply stored as `Group`s on disk. These are distinct from DataFrames and sparse arrays since they don’t have any special attributes. A `Group` is created for any `Mapping` in the AnnData object, including the standard `obsm`, `varm`, `layers`, and `uns`. Notably, this definition is used recursively within `uns`: `````{tab-set} ````{tab-item} HDF5 :sync: hdf5 ```python >>> store["uns"].visititems(print) [...] pca pca/variance pca/variance_ratio [...] ``` ```` ````{tab-item} Zarr :sync: zarr ```python >>> store["uns"].visititems(print) [...] pca pca/variance pca/variance_ratio [...] ``` ```` ````` (dict)= (mapping)= ### Mapping specifications (v0.1.0) * Each mapping MUST be its own group * The group's metadata MUST contain the encoding metadata `"encoding-type": "dict"`, `"encoding-version": "0.1.0"` (scalars)= ## Scalars Zero dimensional arrays are used for scalar values (i.e. single values like strings, numbers or booleans). These should only occur inside of `uns`, and are commonly saved parameters: `````{tab-set} ````{tab-item} HDF5 :sync: hdf5 ```python >>> store["uns/neighbors/params"].visititems(print) method metric n_neighbors random_state ``` ```` ````{tab-item} Zarr :sync: zarr ```python >>> store["uns/neighbors/params"].visititems(print) method metric n_neighbors random_state ``` ```` ````` ```python >>> store["uns/neighbors/params/metric"][()] 'euclidean' >>> dict(store["uns/neighbors/params/metric"].attrs) {'encoding-type': 'string', 'encoding-version': '0.2.0'} ``` (numeric-scalar)= (string)= ### Scalar specification (v0.2.0) * Scalars MUST be written as a 0 dimensional array * Numeric scalars * MUST have `"encoding-type": "numeric-scalar"`, `"encoding-version": "0.2.0"` in their metadata * MUST be a single numeric value, including boolean, unsigned integer, signed integer, floating point, or complex floating point * String scalars * MUST have `"encoding-type": "string"`, `"encoding-version": "0.2.0"` in their metadata * In zarr, scalar strings MUST be stored as a fixed length unicode dtype * In HDF5, scalar strings MUST be stored as a variable length utf-8 encoded string dtype (categorical-arrays)= ## Categorical arrays ```python >>> categorical = store["obs"]["development_stage"] >>> dict(categorical.attrs) {'encoding-type': 'categorical', 'encoding-version': '0.2.0', 'ordered': False} ``` Discrete values can be efficiently represented with categorical arrays (similar to `factors` in `R`). These arrays encode the values as small width integers (`codes`), which map to the original label set (`categories`). Each entry in the `codes` array is the zero-based index of the encoded value in the `categories` array. To represent a missing value, a code of `-1` is used. We store these two arrays separately. `````{tab-set} ````{tab-item} HDF5 :sync: hdf5 ```python >>> categorical.visititems(print) categories codes ``` ```` ````{tab-item} Zarr :sync: zarr ```python >>> categorical.visititems(print) categories codes ``` ```` ````` (categorical)= ### Categorical array specification (v0.2.0) * Categorical arrays MUST be stored as a group * The group's metadata MUST contain the encoding metadata `"encoding-type": "categorical"`, `"encoding-version": "0.2.0"` * The group's metadata MUST contain the boolean valued field `"ordered"`, which indicates whether the categories are ordered * The group MUST contain an integer valued array named `"codes"` whose maximum value is the number of categories - 1 * The `"codes"` array MAY contain signed integer values. If so, the code `-1` denotes a missing value * The group MUST contain an array called `"categories"` (string-arrays)= ## String arrays Arrays of strings are handled differently than numeric arrays since numpy doesn't really have a good way of representing arrays of unicode strings. `anndata` assumes strings are text-like data, so it uses a variable length encoding. `````{tab-set} ````{tab-item} HDF5 :sync: hdf5 ```python >>> store["var"][store["var"].attrs["_index"]] ``` ```` ````{tab-item} Zarr :sync: zarr ```python >>> store["var"][store["var"].attrs["_index"]] ``` ```` ````` ```python >>> dict(categorical["categories"].attrs) {'encoding-type': 'string-array', 'encoding-version': '0.2.0'} ``` (string-array)= ### String array specifications (v0.2.0) * String arrays MUST be stored in arrays * The arrays's metadata MUST contain the encoding metadata `"encoding-type": "string-array"`, `"encoding-version": "0.2.0"` * In `zarr`, string arrays MUST be stored using `numcodecs`' `VLenUTF8` codec * In `HDF5`, string arrays MUST be stored using the variable length string data type, with a utf-8 encoding (nullable-arrays)= ## Nullable integers, booleans, and strings We support IO with Pandas nullable integer, boolean, and string arrays. We represent these on disk similar to `numpy` masked arrays, `julia` nullable arrays, or `arrow` validity bitmaps (see {issue}`504` for more discussion). That is, we store an indicator array (or mask) of null values alongside the array of all values. `````{tab-set} ````{tab-item} HDF5 :sync: hdf5 ```python >>> from anndata import write_elem >>> null_store = h5py.File("tmp.h5", mode="w") >>> int_array = pd.array([1, None, 3, 4]) >>> int_array [1, , 3, 4] Length: 4, dtype: Int64 >>> write_elem(null_store, "nullable_integer", int_array) >>> null_store.visititems(print) nullable_integer nullable_integer/mask nullable_integer/values ``` ```` ````{tab-item} Zarr :sync: zarr ```python >>> from anndata import write_elem >>> null_store = zarr.open() >>> int_array = pd.array([1, None, 3, 4]) >>> int_array [1, , 3, 4] Length: 4, dtype: Int64 >>> write_elem(null_store, "nullable_integer", int_array) >>> null_store.visititems(print) nullable_integer nullable_integer/mask nullable_integer/values ``` ```` ````` ```python >>> dict(null_store["nullable_integer"].attrs) {'encoding-type': 'nullable-integer', 'encoding-version': '0.1.0'} ``` (nullable-integer)= ### Nullable integer specifications (v0.1.0) * Nullable integers MUST be stored as a group * The group’s attributes MUST contain the encoding metadata `"encoding-type": "nullable-integer"`, `"encoding-version": "0.1.0"` * The group MUST contain an integer valued array under the key `"values"` * The group MUST contain an boolean valued array under the key `"mask"` (nullable-boolean)= ### Nullable boolean specifications (v0.1.0) * Nullable booleans MUST be stored as a group * The group’s attributes MUST contain the encoding metadata `"encoding-type": "nullable-boolean"`, `"encoding-version": "0.1.0"` * The group MUST contain an boolean valued array under the key `"values"` * The group MUST contain an boolean valued array under the key `"mask"` * The `"values"` and `"mask"` arrays MUST be the same shape (nullable-string-array)= ### Nullable string specifications (v0.1.0) * Nullable strings MUST be stored as a group * The group’s attributes MUST contain the encoding metadata `"encoding-type": "nullable-string-array"`, `"encoding-version": "0.1.0"` * The group’s attributes MAY contain `"na-value"` as an indicator for missing value semantics with the possible value `"NA"` or `"NaN"` described in [](#missing-value-semantics), and the default being `"NA"` * The group MUST contain a string valued array under the key `"values"` * The group MUST contain a boolean valued array under the key `"mask"` * The `"values"` and `"mask"` arrays MUST be the same shape (missing-value-semantics)= ### Missing value semantics If available in the runtime data model, the following values representing missing value semantics are defined. See the individual specification versions for elements that support them. `"NA"` means that a comparison between a missing value and a defined value produces a missing value (e.g. `"x"==NA` → `NA`). `"NaN"` means that a comparison between a missing value and a defined value produces a binary result (e.g. `"x"==NaN` → `false`). After reading an element supporting missing value semantics, it SHOULD behave according to the specified semantics if the runtime data model allows for it. If no semantics are specified in the representation, the default semantics from the element specification SHOULD be used. When the encoded semantics value is unknown to the implementation, reading the element SHOULD NOT produce an error, and the implementation may freely choose the semantics. For element specifications supporting missing value semantics, the semantics value most closely matching the source data model SHOULD be written. If the source data model instead behaves very differently from all currently defined ones, please reach out to get a new value for the behavior defined here. #### Examples Pandas’ string arrays support both `"NA"` and `"NaN"` semantics, while `R`’s `character` array only supports `"NA"` semantics. Therefore an R implementation SHOULD write a `character` array with `"na-value": "NA"`, and when reading a `"nullable-string-array"`, it MAY choose to read it into a `character` array, ignoring the `"na-value"` attribute (since `character` arrays can not be made to behave with `"NaN"` semantics). (awkward-array)= ## AwkwardArrays ```{warning} **Experimental** Support for ragged arrays via awkward array is considered experimental under the 0.9.0 release series. Please direct feedback on it's implementation to [https://github.com/scverse/anndata](https://github.com/scverse/anndata). ``` Ragged arrays are supported in `anndata` through the [Awkward Array](https://awkward-array.org/) library. For storage on disk, we break down the awkward array into it’s constituent arrays using [`ak.to_buffers`](https://awkward-array.readthedocs.io/en/latest/_auto/ak.to_buffers.html) then writing these arrays using `anndata`’s methods. `````{tab-set} ````{tab-item} HDF5 :sync: hdf5 ```python >>> store["varm/transcript"].visititems(print) node1-mask node10-data node11-mask node12-offsets node13-mask node14-data node16-offsets node17-data node2-offsets node3-data node4-mask node5-offsets node6-data node7-mask node8-offsets node9-mask ``` ```` ````{tab-item} Zarr :sync: zarr ```python >>> store["varm/transcript"].visititems(print) node1-mask node10-data node11-mask node12-offsets node13-mask node14-data node16-offsets node17-data node2-offsets node3-data node4-mask node5-offsets node6-data node7-mask node8-offsets node9-mask ``` ```` ````` The length of the array is saved to it’s own `"length"` attribute, while metadata for the array structure is serialized and saved to the `“form”` attribute. ```python >>> dict(store["varm/transcript"].attrs) {'encoding-type': 'awkward-array', 'encoding-version': '0.1.0', 'form': '{"class": "RecordArray", "fields": ["tx_id", "seq_name", ' '"exon_seq_start", "exon_seq_end", "ensembl_id"], "contents": ' '[{"class": "BitMaskedArray", "mask": "u8", "valid_when": true, ' '"lsb_order": true, "content": {"class": "ListOffsetArray", ' '"offsets": "i64", "content": {"class": "NumpyArray", "primitive": ' '"uint8", "inner_shape": [], "parameters": {"__array__": "char"}, ' '"form_key": "node3"}, "parameters": {"__array__": "string"}, ' '"form_key": "node2"}, "parameters": {}, "form_key": "node1"}, ' ... 'length': 40145} ``` These can be read back as awkward arrays using the [`ak.from_buffers`](https://awkward-array.readthedocs.io/en/latest/_auto/ak.from_buffers.html) function: ```python >>> import awkward as ak >>> from anndata.io import read_elem >>> awkward_group = store["varm/transcript"] >>> ak.from_buffers( ... awkward_group.attrs["form"], ... awkward_group.attrs["length"], ... {k: read_elem(v) for k, v in awkward_group.items()} ... ) >>> transcript_models[:5] [{tx_id: 'ENST00000450305', seq_name: '1', exon_seq_start: [...], ...}, {tx_id: 'ENST00000488147', seq_name: '1', exon_seq_start: [...], ...}, {tx_id: 'ENST00000473358', seq_name: '1', exon_seq_start: [...], ...}, {tx_id: 'ENST00000477740', seq_name: '1', exon_seq_start: [...], ...}, {tx_id: 'ENST00000495576', seq_name: '1', exon_seq_start: [...], ...}] ----------------------------------------------------------------------- type: 5 * { tx_id: ?string, seq_name: ?string, exon_seq_start: option[var * ?int64], exon_seq_end: option[var * ?int64], ensembl_id: ?string } >>> transcript_models[0] {tx_id: 'ENST00000450305', seq_name: '1', exon_seq_start: [12010, 12179, 12613, 12975, 13221, 13453], exon_seq_end: [12057, 12227, 12697, 13052, 13374, 13670], ensembl_id: 'ENSG00000223972'} ------------------------------------------------------------ type: { tx_id: ?string, seq_name: ?string, exon_seq_start: option[var * ?int64], exon_seq_end: option[var * ?int64], ensembl_id: ?string } ``` [easy to find]: https://en.wikipedia.org/wiki/Sparse_matrix#Compressed_sparse_row_(CSR,_CRS_or_Yale_format) [hdf5]: https://en.wikipedia.org/wiki/Hierarchical_Data_Format