Find files on the local filesystem#

Find files on the local filesystem.

Deprecated since version 2.14.0: This module has been moved to esmvalcore.io.local. Importing it as esmvalcore.local is deprecated and will be removed in version 2.16.0.

Classes:

LocalDataSource(name, project, priority, ...)

Data source for finding files on a local filesystem.

LocalFile(*args, **kwargs)

File on the local filesystem.

Functions:

find_files(*[, debug])

Find files on the local filesystem.
class esmvalcore.local.LocalDataSource(name: str, project: str, priority: int, rootpath: ~pathlib.Path, dirname_template: str, filename_template: str, ignore_warnings: list[dict[str, ~typing.Any]] | None = <factory>, ignore_datetimes_in_filename: bool = False)[source]#

Bases: DataSource

Data source for finding files on a local filesystem.

Attributes:

debug_info

A string containing debug information when no data is found.

dirname_template

The template for the directory names.

filename_template

The template for the file names.

ignore_datetimes_in_filename

Ignore date times specified in file names.

ignore_warnings

Warnings to ignore when loading the data.

name

A name identifying the data source.

priority

The priority of the data source.

project

The project that the data source provides data for.

rootpath

The path where the directories are located.

Methods:

find_data(**facets)

Find data locally.
Parameters:

  • name (str)

  • project (str)

  • priority (int)

  • rootpath (Path)

  • dirname_template (str)

  • filename_template (str)

  • ignore_warnings (list[dict[str, Any]] | None)

  • ignore_datetimes_in_filename (bool)

debug_info: str = ''#

A string containing debug information when no data is found.

dirname_template: str#

The template for the directory names.

filename_template: str#

The template for the file names.

find_data(**facets: FacetValue) → list[LocalFile][source]#

Find data locally.

Parameters:

**facets (FacetValue) – Find data matching these facets.

Returns:

A list of files.

Return type:

list[LocalFile]

ignore_datetimes_in_filename: bool = False#

Ignore date times specified in file names.

By default, if possible, the time range spanned by a file is determined by datetimes given in its file name. For example, the file my-model_20000101-20051231.nc will be assigned a start date of 2000-01-01 and an end date of 2005-12-31. Only if reading datetimes from the file name fails, the actual file will be opened to determine the time range from the file contents.

If this option is set to True, date times in file names are ignored, and the time range is always determined from the contents of the file.

Note that in the vast majority of cases, reading datetimes from file names works very well. In addition, opening files to determine time range is much slower than just reading file names. Thus, this option should only be set to True if absolutely necessary.

ignore_warnings: list[dict[str, Any]] | None#

Warnings to ignore when loading the data.

The list should contain dict`s with keyword arguments that will be passed to the :func:`warnings.filterwarnings function when calling LocalFile.to_iris().

name: str#

A name identifying the data source.

priority: int#

The priority of the data source. Lower values have priority.

project: str#

The project that the data source provides data for.

rootpath: Path#

The path where the directories are located.

class esmvalcore.local.LocalFile(*args, **kwargs)[source]#

Bases: PosixPath, DataElement

File on the local filesystem.

Attributes:

attributes

Attributes are key-value pairs describing the data.

facets

Facets are key-value pairs that were used to find this data.

ignore_warnings

Warnings to ignore when loading the data.

Methods:

prepare()

Prepare the data for access.

to_iris()

Load the data as Iris cubes.
property attributes: dict[str, Any]#

Attributes are key-value pairs describing the data.

property facets: Facets#

Facets are key-value pairs that were used to find this data.

property ignore_warnings: list[dict[str, Any]] | None#

Warnings to ignore when loading the data.

The list should contain dict`s with keyword arguments that will be passed to the :func:`warnings.filterwarnings function when calling the to_iris method.

prepare() → None[source]#

Prepare the data for access.

Return type:

None

to_iris() → CubeList[source]#

Load the data as Iris cubes.

Returns:

The loaded data.

Return type:

iris.cube.CubeList

esmvalcore.local.find_files(*, debug: bool = False, **facets: FacetValue) → list[LocalFile] | tuple[list[LocalFile], list[Path]][source]#

Find files on the local filesystem.

Deprecated since version 2.14.0: This function is deprecated and will be removed in version 2.16.0. Please use esmvalcore.local.LocalDataSource.find_data() instead.

The directories that are searched for files are defined in esmvalcore.config.CFG under the 'rootpath' key using the directory structure defined under the 'drs' key. If esmvalcore.config.CFG['rootpath'] contains a key that matches the value of the project facet, those paths will be used. If there is no project specific key, the directories in esmvalcore.config.CFG['rootpath']['default'] will be searched.

See Input data for extensive instructions on configuring ESMValCore so it can find files locally.

Parameters:

  • debug (bool) – When debug is set to True, the function will return a tuple with the first element containing the files that were found and the second element containing the glob.glob() patterns that were used to search for files.

  • **facets (FacetValue) – Facets used to search for files. An '*' can be used to match any value. By default, only the latest version of a file will be returned. To select all versions use version='*'. It is also possible to specify multiple values for a facet, e.g. exp=['historical', 'ssp585'] will match any file that belongs to either the historical or ssp585 experiment. The timerange facet can be specified in ISO 8601 format.

Return type:

list[LocalFile] | tuple[list[LocalFile], list[Path]]

Note

A value of timerange='*' is supported, but combining a '*' with a time or period as supported in the recipe is currently not supported and will return all found files.

Examples

Search for files containing surface air temperature from any CMIP6 model for the historical experiment:

>>> esmvalcore.local.find_files(
...     project='CMIP6',
...     activity='CMIP',
...     mip='Amon',
...     short_name='tas',
...     exp='historical',
...     dataset='*',
...     ensemble='*',
...     grid='*',
...     institute='*',
... )
[LocalFile('/home/bandela/climate_data/CMIP6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/tas/gn/v20181214/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc')]
Returns:

The files that were found.

Return type:

list[LocalFile]

Parameters:

  • debug (bool)

  • facets (FacetValue)