esmvalcore.io.local#

Find files on the local filesystem.

Example configuration to find CMIP6 data on a personal computer:

Listing 20 Contents of data-local.yml#

projects:
  CMIP7:
    data:
      local:
        type: esmvalcore.io.local.LocalDataSource
        rootpath: ~/climate_data
        # Directory structure and file name template from https://wcrp-cmip.github.io/cmip7-guidance/docs/CMIP7/Global_Attributes/
        dirname_template: "{drs_specs}/{project}/{activity}/{institute}/{dataset}/{exp}/{ensemble}/{region}/{frequency}/{short_name}/{branding_suffix}/{grid}/{version}"
        filename_template: "{short_name}_{branding_suffix}_{frequency}_{region}_{grid}_{dataset}_{exp}_{ensemble}*.nc"
  CMIP6:
    data:
      local:
        type: esmvalcore.io.local.LocalDataSource
        rootpath: ~/climate_data
        dirname_template: "{project}/{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}"
        filename_template: "{short_name}_{mip}_{dataset}_{exp}_{ensemble}_{grid}*.nc"

The module will find files matching the glob.glob() pattern formed by rootpath/dirname_template/filename_template, where the facets defined inside the curly braces of the templates are replaced by their values from the Dataset or the recipe plus any facet-value pairs that can be automatically added using augment_facets(). Note that the name of the data source, local-data in the example above, must be unique within each project but can otherwise be chosen freely.

To start using this module on a personal computer, copy the example configuration file into your configuration directory by running the command:

esmvaltool config copy data-local.yml

and tailor it for your own system if needed.

Example configuration files for popular HPC systems and some supported climate models are also available. View the list of available files by running the command:

esmvaltool config list

Further information is available in Data sources.

Data:

GRIB_FORMATS

GRIB file extensions.

Classes:

`LocalDataSource`(name, project, priority, ...)	Data source for finding files on a local filesystem.
`LocalFile`(args, *kwargs)	File on the local filesystem.

esmvalcore.io.local.GRIB_FORMATS = ('.grib2', '.grib', '.grb2', '.grb', '.gb2', '.gb')#: GRIB file extensions.

class esmvalcore.io.local.LocalDataSource(name: str, project: str, priority: int, rootpath: ~pathlib.Path, dirname_template: str, filename_template: str, ignore_warnings: list[dict[str, ~typing.Any]] | None = <factory>, ignore_datetimes_in_filename: bool = False)[source]#

Bases: DataSource

Data source for finding files on a local filesystem.

Attributes:

`debug_info`	A string containing debug information when no data is found.
`dirname_template`	The template for the directory names.
`filename_template`	The template for the file names.
`ignore_datetimes_in_filename`	Ignore date times specified in file names.
`ignore_warnings`	Warnings to ignore when loading the data.
`name`	A name identifying the data source.
`priority`	The priority of the data source.
`project`	The project that the data source provides data for.
`rootpath`	The path where the directories are located.

Methods:

find_data(**facets)

Find data locally.

Parameters:

name (str)
project (str)
priority (int)
rootpath (Path)
dirname_template (str)
filename_template (str)
ignore_warnings (list[dict[str, Any]] | None)
ignore_datetimes_in_filename (bool)

debug_info: str = ''#: A string containing debug information when no data is found.

dirname_template: str#: The template for the directory names.

filename_template: str#: The template for the file names.

find_data(**facets: FacetValue) → list[LocalFile][source]#

Find data locally.

Parameters:: **facets (FacetValue) – Find data matching these facets.
Returns:: A list of files.
Return type:: list[LocalFile]

ignore_datetimes_in_filename: bool = False#

Ignore date times specified in file names.

By default, if possible, the time range spanned by a file is determined by datetimes given in its file name. For example, the file my-model_20000101-20051231.nc will be assigned a start date of 2000-01-01 and an end date of 2005-12-31. Only if reading datetimes from the file name fails, the actual file will be opened to determine the time range from the file contents.

If this option is set to True, date times in file names are ignored, and the time range is always determined from the contents of the file.

Note that in the vast majority of cases, reading datetimes from file names works very well. In addition, opening files to determine time range is much slower than just reading file names. Thus, this option should only be set to True if absolutely necessary.

ignore_warnings: list[dict[str, Any]] | None#

Warnings to ignore when loading the data.

The list should contain dict`s with keyword arguments that will be passed to the :func:`warnings.filterwarnings function when calling LocalFile.to_iris().

name: str#: A name identifying the data source.

priority: int#: The priority of the data source. Lower values have priority.

project: str#: The project that the data source provides data for.

rootpath: Path#: The path where the directories are located.

class esmvalcore.io.local.LocalFile(*args, **kwargs)[source]#

Bases: PosixPath, DataElement

File on the local filesystem.

Attributes:

`attributes`	Attributes are key-value pairs describing the data.
`facets`	Facets are key-value pairs that were used to find this data.
`ignore_warnings`	Warnings to ignore when loading the data.

Methods:

`prepare`()	Prepare the data for access.
`to_iris`()	Load the data as Iris cubes.

property attributes: dict[str, Any]#: Attributes are key-value pairs describing the data.

property facets: Facets#: Facets are key-value pairs that were used to find this data.

property ignore_warnings: list[dict[str, Any]] | None#

Warnings to ignore when loading the data.

The list should contain dict`s with keyword arguments that will be passed to the :func:`warnings.filterwarnings function when calling the to_iris method.

prepare() → None[source]#

Prepare the data for access.

Return type:: None

to_iris() → CubeList[source]#

Load the data as Iris cubes.

Returns:: The loaded data.
Return type:: iris.cube.CubeList