{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# Using and understanding Catalogs\n", "\n", "
INFO\n", "\n", "Catalogs in `xscen` are built upon Datastores in `intake_esm`. For more information on basic usage, such as the `search()` function, [please consult their documentation](https://intake-esm.readthedocs.io/en/stable/).\n", "
\n", "\n", "Catalogs are made of two files:\n", "\n", "- JSON file containing metadata such as the catalog's title, description etc. It also contains an attribute *catalog_file* that points towards the CSV. Most xscen catalog will have very similar JSON files.\n", "- CSV file containing the catalog itself. This file can be zipped.\n", "\n", "Two types of catalogs have been implemented in `xscen`.\n", "\n", "- __Static catalogs:__ A [`DataCatalog`](../xscen.rst#xscen.catalog.DataCatalog) is a *read-only* `intake-esm` catalog that contains information on all available data. Usually, this type of catalog should only be consulted at the start of a new project.\n", "\n", "- __Updatable catalogs:__ A [`ProjectCatalog`](../xscen.rst#xscen.catalog.ProjectCatalog) is a *DataCatalog* with additional *write* functionalities. This kind of catalog should be used to keep track of the new data created during the course of a project, such as regridded or bias-corrected data, since it can `update` itself and append new information to the associated CSV file.\n", "\n", "__NOTE:__ As to not accidentally lose data, both catalogs currently have no function to remove data from the CSV file. However, upon initialisation and when updating or refreshing itself, the catalog validates that all entries still exist and, if files have been manually removed, deletes their entries from the catalog.\n", "\n", "Catalogs in `xscen` are made to follow a nomenclature that is as close as possible to the Python Earth Science Standard Vocabulary : [https://github.com/ES-DOC/pyessv](https://github.com/ES-DOC/pyessv). The columns are listed below but for more details and concrete examples about the entries, consult [the relevant page in the documentation](../columns.rst):\n", "\n", "| Column name | Description |\n", "| :- | :- |\n", "| id | Unique DatasetID generated by `xscen` based on a subset of columns. |\n", "| type | Type of data: [forecast, station-obs, gridded-obs, reconstruction, simulation] |\n", "| processing_level | Level of post-processing reached: [raw, extracted, regridded, biasadjusted] |\n", "| bias_adjust_institution | Institution that computed the bias adjustment. |\n", "| bias_adjust_project | Name of the project that computed the bias adjustment. |\n", "| mip_era | CMIP Generation associated with the data. |\n", "| activity | Model Intercomparison Project (MIP) associated with the data. |\n", "| driving_model | Name of the driver. |\n", "| institution | Institution associated with the source. |\n", "| source | Name of the model or the dataset. |\n", "| experiment | Name of the experiment of the model. |\n", "| member | Name of the realisation (or of the driving realisation in the case of RCMs). |\n", "| xrfreq | Pandas/xarray frequency. |\n", "| frequency | Frequency in letters (CMIP6 format). |\n", "| variable | Variable(s) in the dataset. |\n", "| domain | Name of the region covered by the dataset. |\n", "| date_start | First date of the dataset. |\n", "| date_end | Last date of the dataset. |\n", "| version | Version of the dataset. |\n", "| format | Format of the dataset. |\n", "| path | Path to the dataset. |\n", "\n", "Individual projects may use a different set of columns, but those will always be present in the official Ouranos internal catalogs. Some parts of `xscen` will however expect certain column names, so diverging from the official list is to be done with care.\n", "\n", "## Basic Catalog Usage\n", "\n", "If an official catalog already exists, it should be opened using `xs.DataCatalog` by pointing it to the JSON file:" ] }, { "cell_type": "code", "execution_count": null, "id": "1", "metadata": { "tags": [] }, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "from xscen import DataCatalog, ProjectCatalog\n", "\n", "# Prepare a dummy folder where data will be put\n", "output_folder = Path().absolute() / \"_data\"\n", "output_folder.mkdir(exist_ok=True)\n", "\n", "DC = DataCatalog(f\"{Path().absolute()}/samples/pangeo-cmip6.json\")\n", "DC" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "The content of the catalog can be accessed by a call to `df`, which will return a `pandas.DataFrame`." ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": {}, "outputs": [], "source": [ "# Access the catalog\n", "DC.df[0:3]" ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "The `unique` function allows listing unique elements for either all the catalog or a subset of columns. It can be called in a few various ways, listed below:" ] }, { "cell_type": "code", "execution_count": null, "id": "5", "metadata": {}, "outputs": [], "source": [ "# List all unique elements in the catalog, returns a pandas.Series\n", "DC.unique()" ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [], "source": [ "# List all unique elements in a subset of columns, returns a pandas.Series\n", "DC.unique([\"variable\", \"frequency\"])" ] }, { "cell_type": "code", "execution_count": null, "id": "7", "metadata": {}, "outputs": [], "source": [ "# List all unique elements in a single columns, returns a list\n", "DC.unique(\"id\")[0:5]" ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "### Basic .search() commands\n", "\n", "The `search` function comes from `intake-esm` and allows searching for specific elements in the catalog's columns. It accepts both wildcards and regular expressions (except for *variable*, which must be exact due to being in *tuples*).\n", "\n", "While regex isn't great at inverse matching (\"does not contain\"), it is possible. Here are a few useful commands:\n", "\n", " - ^string : Starts with string\n", "\n", " - string$ : Ends with string\n", "\n", " - ^(?!string).*$ : Does not start with string\n", "\n", " - .*(? NOTE: Excepting fixed fields, where *'fx'* should be used, frequencies here use the `pandas` nomenclature ('D', 'H', '6H', 'MS', etc.).\n", "- `other_search_criteria` is used to search for specific entries in other columns of the catalog, such as *activity*. `require_all_on` can also be passed here.\n", "- `exclusions` is used to exclude certain simulations or keywords from the results.\n", "- `match_hist_and_fut` is used to indicate that RCP/SSP simulations should be matched with their *historical* counterparts.\n", "- `periods` is used to search for specific time periods.\n", "- `allow_resampling` is used to allow searching for data at higher frequencies than requested.\n", "- `allow_conversion` is used to allow searching for calculable variables, in the case where the requested variable would not be available.\n", "- `restrict_resolution` is used to limit the results to the finest or coarsest resolution available for each source.\n", "- `restrict_members` is used to limit the results to a maximum number of realizations for each source.\n", "- `restrict_warming_level` is used to limit the results to only datasets that are present in the csv used for calculating warming levels. You can also pass a dict to verify that a given warming level is reached.\n", "\n", "Note that compared to `search`, the result of `search_data_catalog` is a dictionary with one entry per unique ID. A given unique ID might contain multiple datasets as per `intake-esm`'s definition, because it groups catalog lines per *id - domain - processing_level - xrfreq*. Thus, it would separate model data that exists at different frequencies.\n", "\n", "\n", "#### Example 1: Multiple variables and frequencies + Historical and future\n", "\n", "Let's start by searching for CMIP6 data that has subdaily precipitation, daily minimum temperature and the land fraction data. The main difference compared to searching for reference datasets is that in most cases, `match_hist_and_fut` will be required to match *historical* simulations to their future counterparts. This works for both CMIP5 and CMIP6 nomenclatures." ] }, { "cell_type": "code", "execution_count": null, "id": "20", "metadata": { "tags": [] }, "outputs": [], "source": [ "import xscen as xs\n", "\n", "variables_and_freqs = {\"tasmin\": \"D\", \"pr\": \"3h\", \"sftlf\": \"fx\"}\n", "other_search_criteria = {\"institution\": [\"NOAA-GFDL\"]}\n", "\n", "cat_sim = xs.search_data_catalogs(\n", " data_catalogs=[f\"{Path().absolute()}/samples/pangeo-cmip6.json\"],\n", " variables_and_freqs=variables_and_freqs,\n", " other_search_criteria=other_search_criteria,\n", " match_hist_and_fut=True,\n", ")\n", "\n", "cat_sim" ] }, { "cell_type": "markdown", "id": "21", "metadata": {}, "source": [ "If required, at this stage, a dataset can be looked at in more details. If we examine the results (look at the 'date_start' and 'date_end' columns), we'll see that it successfully found historical simulations in the *CMIP* activity and renamed both their *activity* and *experiment* to match the future simulations." ] }, { "cell_type": "code", "execution_count": null, "id": "22", "metadata": { "tags": [] }, "outputs": [], "source": [ "cat_sim[\"ScenarioMIP_NOAA-GFDL_GFDL-CM4_ssp585_r1i1p1f1_gr1\"].df" ] }, { "cell_type": "markdown", "id": "23", "metadata": {}, "source": [ "#### Example 2: Restricting results\n", "\n", "The two previous search results were the same simulation, but on 2 different grids (`gr1` and `gr2`). If desired, `restrict_resolution` can be called to choose the finest or coarsest grid." ] }, { "cell_type": "code", "execution_count": null, "id": "24", "metadata": { "tags": [] }, "outputs": [], "source": [ "variables_and_freqs = {\"tasmin\": \"D\", \"pr\": \"3h\", \"sftlf\": \"fx\"}\n", "other_search_criteria = {\"institution\": [\"NOAA-GFDL\"], \"experiment\": [\"ssp585\"]}\n", "\n", "cat_sim = xs.search_data_catalogs(\n", " data_catalogs=[f\"{Path().absolute()}/samples/pangeo-cmip6.json\"],\n", " variables_and_freqs=variables_and_freqs,\n", " other_search_criteria=other_search_criteria,\n", " match_hist_and_fut=True,\n", " restrict_resolution=\"finest\",\n", ")\n", "\n", "cat_sim" ] }, { "cell_type": "markdown", "id": "25", "metadata": {}, "source": [ "Similarly, if we search for historical NorESM2-MM data, we'll find that it has 3 members. If desired, `restrict_members` can be called to choose a maximum number of realization per model." ] }, { "cell_type": "code", "execution_count": null, "id": "26", "metadata": {}, "outputs": [], "source": [ "variables_and_freqs = {\"tasmin\": \"D\"}\n", "other_search_criteria = {\"source\": [\"NorESM2-MM\"], \"experiment\": [\"historical\"]}\n", "\n", "cat_sim = xs.search_data_catalogs(\n", " data_catalogs=[f\"{Path().absolute()}/samples/pangeo-cmip6.json\"],\n", " variables_and_freqs=variables_and_freqs,\n", " other_search_criteria=other_search_criteria,\n", " restrict_members={\"ordered\": 2},\n", ")\n", "\n", "cat_sim" ] }, { "cell_type": "markdown", "id": "27", "metadata": {}, "source": [ "Finally, `restrict_warming_level` can be used to be sure that the results either exist in `xscen`'s warming level database (if a boolean), or reach a given warming level." ] }, { "cell_type": "code", "execution_count": null, "id": "28", "metadata": {}, "outputs": [], "source": [ "variables_and_freqs = {\"tasmin\": \"D\"}\n", "\n", "cat_sim = xs.search_data_catalogs(\n", " data_catalogs=[f\"{Path().absolute()}/samples/pangeo-cmip6.json\"],\n", " variables_and_freqs=variables_and_freqs,\n", " match_hist_and_fut=True,\n", " restrict_warming_level={\n", " \"wl\": 2\n", " }, # SSP126 gets eliminated, since it doesn't reach +2°C by 2100.\n", ")\n", "\n", "cat_sim" ] }, { "cell_type": "markdown", "id": "29", "metadata": {}, "source": [ "#### Example 3: Search for data that can be computed from what's available\n", "\n", "`allow_resampling` and `allow_conversion` are powerful search tools to find data that doesn't explicitly exist in the catalog, but that can easily be computed." ] }, { "cell_type": "code", "execution_count": null, "id": "30", "metadata": { "tags": [] }, "outputs": [], "source": [ "cat_sim_adv = xs.search_data_catalogs(\n", " data_catalogs=[f\"{Path().absolute()}/samples/pangeo-cmip6.json\"],\n", " variables_and_freqs={\"evspsblpot\": \"D\", \"tas\": \"YS\"},\n", " other_search_criteria={\"source\": [\"NorESM2-MM\"], \"processing_level\": [\"raw\"]},\n", " match_hist_and_fut=True,\n", " allow_resampling=True,\n", " allow_conversion=True,\n", ")\n", "cat_sim_adv" ] }, { "cell_type": "markdown", "id": "31", "metadata": {}, "source": [ "If we examine the SSP5-8.5 results, we'll see that while it failed to find *evspsblpot*, it successfully understood that *tasmin* and *tasmax* can be used to compute it. It also understood that daily *tasmin* and *tasmax* is a valid search result for `{tas: YS}`, since it can be computed first, then aggregated to a yearly frequency." ] }, { "cell_type": "code", "execution_count": null, "id": "32", "metadata": { "tags": [] }, "outputs": [], "source": [ "cat_sim_adv[\"ScenarioMIP_NCC_NorESM2-MM_ssp585_r1i1p1f1_gn\"].unique()" ] }, { "cell_type": "markdown", "id": "33", "metadata": {}, "source": [ "It's also possible to search for multiple frequencies at the same time by using a list of xrfreq." ] }, { "cell_type": "code", "execution_count": null, "id": "34", "metadata": { "tags": [] }, "outputs": [], "source": [ "cat_sim_adv_multifreq = xs.search_data_catalogs(\n", " data_catalogs=[f\"{Path().absolute()}/samples/pangeo-cmip6.json\"],\n", " variables_and_freqs={\"tas\": [\"D\", \"MS\", \"YS\"]},\n", " other_search_criteria={\n", " \"source\": [\"NorESM2-MM\"],\n", " \"processing_level\": [\"raw\"],\n", " \"experiment\": [\"ssp585\"],\n", " },\n", " match_hist_and_fut=True,\n", " allow_resampling=True,\n", " allow_conversion=True,\n", ")\n", "print(\n", " cat_sim_adv_multifreq[\n", " \"ScenarioMIP_NCC_NorESM2-MM_ssp585_r1i1p1f1_gn\"\n", " ]._requested_variable_freqs\n", ")" ] }, { "cell_type": "markdown", "id": "35", "metadata": {}, "source": [ "#### Derived variables\n", "\n", "The `allow_conversion` argument is built upon `xclim`'s virtual indicators module and `intake-esm`'s [DerivedVariableRegistry](https://ncar.github.io/esds/posts/2021/intake-esm-derived-variables/) in a way that should be seamless to the user. It works by using the methods defined in `xscen/xclim_modules/conversions.yml` to add a registry of *derived* variables that exist virtually through computation methods.\n", "\n", "In the example above, we can see that the search failed to find *evspsblpot* within *NorESM2-MM*, but understood that *tasmin* and *tasmax* could be used to estimate it using `xclim`'s `potential_evapotranspiration`.\n", "\n", "Most use cases should already be covered by the aforementioned file. The preferred way to add new methods is to [submit a new indicator to xclim](https://xclim.readthedocs.io/en/stable/contributing.html), and then to add a call to that indicator in `conversions.yml`. In the case where this is not possible or where the transformation would be out of scope for `xclim`, the calculation can be implemented into `xscen/xclim_modules/conversions.py` instead.\n", "\n", "Alternatively, if other functions or other parameters are required for a specific use case (e.g. using `relative_humidity` instead of `relative_humidity_from_dewpoint`, or using a different formula), then a custom YAML file can be used. This custom file can be referred to using the `conversion_yaml` argument of `search_data_catalogs`.\n", "\n", "`.derivedcat` can be called on a catalog to obtain the list of DerivedVariable and the function associated to them. In addition, `._requested_variables` will display the list of variables that will be opened by the `to_dataset_dict()` function, including *DerivedVariables*.\n", "\n", "
WARNING\n", "\n", "`_requested_variables` should NOT be modified under any circumstance, as it is likely to make `to_dataset_dict()` fail. To add some transparency on which variables have been **requested** and which are the **dependent** ones, `xscen` has added `_requested_variables_true` and `_dependent_variables`. This is very likely to be changed in the future.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "36", "metadata": { "tags": [] }, "outputs": [], "source": [ "cat_sim_adv[\"ScenarioMIP_NCC_NorESM2-MM_ssp585_r1i1p1f1_gn\"].derivedcat" ] }, { "cell_type": "code", "execution_count": null, "id": "37", "metadata": {}, "outputs": [], "source": [ "print(cat_sim_adv[\"ScenarioMIP_NCC_NorESM2-MM_ssp585_r1i1p1f1_gn\"]._requested_variables)\n", "print(\n", " f\"Requested: {cat_sim_adv['ScenarioMIP_NCC_NorESM2-MM_ssp585_r1i1p1f1_gn']._requested_variables_true}\"\n", ")\n", "print(\n", " f\"Dependent: {cat_sim_adv['ScenarioMIP_NCC_NorESM2-MM_ssp585_r1i1p1f1_gn']._dependent_variables}\"\n", ")" ] }, { "cell_type": "markdown", "id": "38", "metadata": {}, "source": [ "
INFO\n", "\n", "`allow_conversion` currently fails if:\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "39", "metadata": {}, "source": [ "## Creating a New Catalog from a Directory\n", "\n", "### Initialisation\n", "\n", "The `create` argument of `ProjectCatalog` can be called to create an empty *ProjectCatalog* and a new set of JSON and CSV files.\n", "\n", "By default, `xscen` will populate the JSON with generic information, defined in `catalog.esm_col_data`. That metadata can be changed using the `project` argument with entries compatible with the ESM Catalog Specification (refer to the link above). Usually, the most useful and common entries will be:\n", "\n", "- title\n", "- description\n", "\n", "`xscen` will also instruct `intake_esm` to group catalog lines per *id - domain - processing_level - xrfreq*. This should be adequate for most uses. In the case that it is not, the following can be added to `project`:\n", "\n", "- \"aggregation_control\": {\"groupby_attrs\": [list_of_columns]}\n", "\n", "Other attributes and behaviours of the project definition can be modified in a similar way." ] }, { "cell_type": "code", "execution_count": null, "id": "40", "metadata": {}, "outputs": [], "source": [ "project = {\n", " \"title\": \"tutorial-catalog\",\n", " \"description\": \"Catalog for the tutorial NetCDFs.\",\n", "}\n", "\n", "PC = ProjectCatalog(\n", " str(output_folder / \"tutorial-catalog.json\"),\n", " create=True,\n", " project=project,\n", " overwrite=True,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "41", "metadata": {}, "outputs": [], "source": [ "# The metadata is stored in PC.esmcat\n", "PC.esmcat" ] }, { "cell_type": "markdown", "id": "42", "metadata": { "tags": [] }, "source": [ "### Appending new data to a ProjectCatalog\n", "\n", "At this stage, the CSV is still empty. There are two main ways to populate a catalog with data:\n", "\n", "- Using `xs.ProjectCatalog.update_from_ds` to append a Dataset and populate the catalog columns using metadata.\n", "\n", "- Using `xs.catutils.parse_directory` to parse through existing NetCDF or Zarr data and decode their information based on file and directory names.\n", "\n", "This tutorial will focus on `catutils.parse_directory`, as `update_from_ds` is more so a function that will be called during a climate-scenario-generation workflow. See the [Getting Started](2_getting_started.ipynb#Updating-the-catalog) tutorial for more details on `update_from_ds`.\n", "\n", "#### Parsing a directory\n", "\n", "
INFO\n", "\n", "If you are an Ouranos employee, this section should be of limited use (unless you need to retroactively parse a directory containing exiting datasets). Please consult the existing Ouranos catalogs using `xs.search_data_catalogs` instead.\n", "
\n", "\n", "The [`parse_directory`](../xscen.rst#xscen.catutils.parse_directory) function relies on analyzing patterns to adequately decode the filenames to store that information in the catalog.\n", "\n", "- Patterns are a succession of column names in curly brackets. See below for examples. The pattern starts where the directory path stops.\n", "- If necessary, `read_from_file` can be used to open the files and read metadata from global attributes. Refer to the API for Docstrings and usage.\n", "- In cases where some column information is the same across all data, `homogenous_info` can be used to explicitly give an attribute to the datasets being processed.\n", "- Anything that isn't filled will be marked as `None`.\n", "\n", "The following example will search through the samples folder and infer information from the folder names. The filename is ignored, except its extension. The variable name and time bounds are read from the file itself." ] }, { "cell_type": "code", "execution_count": null, "id": "43", "metadata": {}, "outputs": [], "source": [ "from xscen.catutils import parse_directory\n", "\n", "df = parse_directory(\n", " directories=[f\"{Path().absolute()}/samples/tutorial/\"],\n", " patterns=[\n", " \"{activity}/{domain}/{institution}/{source}/{experiment}/{member}/{frequency}/{?:_}.nc\"\n", " ],\n", " homogenous_info={\n", " \"mip_era\": \"CMIP6\",\n", " \"type\": \"simulation\",\n", " \"processing_level\": \"raw\",\n", " },\n", " read_from_file=[\"variable\", \"date_start\", \"date_end\"],\n", ")\n", "df" ] }, { "cell_type": "markdown", "id": "44", "metadata": {}, "source": [ "#### Unique Dataset ID\n", "\n", "In addition to the parse itself, `parse_directory` will create a unique Dataset ID that can be used to easily determine one simulation from another. This can be edited with the `id_columns` argument of `parse_directory`, but by default, IDs are based on CMIP6's ID structure with additions related to regional models and bias adjustment:\n", "\n", "- `{bias_adjust_project} _ {mip_era} _ {activity} _ {driving_model} _ {institution} _ {source} _ {experiment} _ {member} _ {domain}`\n", "\n", "This utility can also be called by itself through `xs.catalog.generate_id()`.\n", "\n", "
INFO\n", "\n", "When constructing IDs, empty columns will be skipped.\n", "
" ] }, { "cell_type": "code", "execution_count": null, "id": "45", "metadata": {}, "outputs": [], "source": [ "df.iloc[0][\"id\"]" ] }, { "cell_type": "markdown", "id": "46", "metadata": {}, "source": [ "#### Appending data using ProjectCatalog.update()\n", "\n", "At this stage, `df` is a `pandas.DataFrame`. `ProjectCatalog.update` is used to append this data to the CSV file and save the results on disk." ] }, { "cell_type": "code", "execution_count": null, "id": "47", "metadata": {}, "outputs": [], "source": [ "PC.update(df)\n", "\n", "PC" ] }, { "cell_type": "markdown", "id": "48", "metadata": {}, "source": [ "#### More on patterns and advanced features\n", "\n", "The `patterns` argument acts as a reverse format string.\n", "\n", "- The \"\\_\" format specifier (like in `{field:_}` allows matching a name containing underscores for this field. The path separators (/, \\\\) are still excluded. Any format specifier supported by [`parse` are usable](https://github.com/r1chardj0n3s/parse).\n", "- Fields starting with a \"?\" will be ignored in the output. This allows to have readable patterns to identify parts we know exist, but do not want to be included in the metadata\n", "- The `DATES` special field will match single dates or date bounds (see below).\n", "- `{?:_}` is useful in filenames as a \"wildcard\" matching. For exammple: `{?:_}_{DATES}.nc` will read in the last \"element\" of the filename into `date_start` and `date_end`, ignoring all previous parts." ] }, { "cell_type": "code", "execution_count": null, "id": "49", "metadata": {}, "outputs": [], "source": [ "# Create fake files for the example:\n", "root = Path(\".\").absolute() / \"_data\" / \"parser_examples\"\n", "root.mkdir(exist_ok=True)\n", "\n", "paths = [\n", " # Folder name includes underscore, single year implicitly means the full year\n", " \"CCCma/CanESM2/day/tg_mean/tg_mean_1950.nc\",\n", " # Fx frequency, no date bounds, strange model name\n", " \"CCCma/CanESM-2/fx/sftlf/sftlf_fx.nc\",\n", " # Bounds given as range at a monthly frequency\n", " \"MIROC/MIROC6/mon/uas/uas_199901-200011.nc\",\n", " # Version number included in the source name, range given a years\n", " \"ERA/ERA5_v2/yr/heat_wave_frequency/hwf_2100-2399.nc\",\n", "]\n", "for path in paths:\n", " (root / path).parent.mkdir(exist_ok=True, parents=True)\n", " with (root / path).open(\"w\") as f:\n", " f.write(\"example\")" ] }, { "cell_type": "markdown", "id": "50", "metadata": {}, "source": [ "##### Example 1 - wrong\n", "The `variable` field does not allow underscores, so the first and last files are not parsed correctly.\n", "\n", "Notice how the `DATES` field was parsed into `date_start` and `date_end`. It also matched with `fx`, returning `NaT` for both fields, as expected." ] }, { "cell_type": "code", "execution_count": null, "id": "51", "metadata": {}, "outputs": [], "source": [ "patt = \"{institution}/{source}/{frequency}/{variable}/{?var}_{DATES}.nc\"\n", "parse_directory(directories=[root], patterns=[patt])" ] }, { "cell_type": "markdown", "id": "52", "metadata": {}, "source": [ "##### Example 2 - wrong again\n", "We fixed the variable field by allowing underscores. We also modified the filename pattern to match any string, including underscores, except for the last element.\n", "\n", "Notice how the \"1950\" part of `tg_mean` has been converted to `date_start='1950-01-01'` and `date_end='1950-12-31'`.\n", "\n", "The `source` field does not allow underscores, so \"ERA5_v2\" is not parsed correctly. However, what we would want is rather to assign \"v2\" to the version field." ] }, { "cell_type": "code", "execution_count": null, "id": "53", "metadata": {}, "outputs": [], "source": [ "patt = \"{institution}/{source}/{frequency}/{variable:_}/{?:_}_{DATES}.nc\"\n", "parse_directory(directories=[root], patterns=[patt])" ] }, { "cell_type": "markdown", "id": "54", "metadata": {}, "source": [ "##### Example 3 - Correct!\n", "We added a second pattern that includes the `version` field." ] }, { "cell_type": "code", "execution_count": null, "id": "55", "metadata": {}, "outputs": [], "source": [ "patts = [\n", " \"{institution}/{source}_{version}/{frequency}/{variable:_}/{?:_}_{DATES}.nc\",\n", " \"{institution}/{source}/{frequency}/{variable:_}/{?:_}_{DATES}.nc\",\n", "]\n", "parse_directory(directories=[root], patterns=patts)" ] }, { "cell_type": "markdown", "id": "56", "metadata": {}, "source": [ "##### Example 4 - Filter on folder names\n", "We can filter the results to include only some folders with the `dirglob` argument." ] }, { "cell_type": "code", "execution_count": null, "id": "57", "metadata": {}, "outputs": [], "source": [ "parse_directory(directories=[root], patterns=patts, dirglob=\"*/CanESM*\")" ] }, { "cell_type": "markdown", "id": "58", "metadata": {}, "source": [ "##### Example 5 - Modifying metadata\n", "We use the `cvs` (Controlled VocabularieS) argument here to replace some terms found in the paths by others we prefer.\n", "\n", "Two replacement types are used :\n", " - simple : in the `source` column, all values of \"CanESM-2\" are replaced by \"CanESM2\"\n", " - complex : in the `institution` column, if the value \"MIROC\" is seen, it triggers the setting of \"global\" in this row's `domain` column, overriding whatever was already present in this field." ] }, { "cell_type": "code", "execution_count": null, "id": "59", "metadata": {}, "outputs": [], "source": [ "parse_directory(\n", " directories=[root],\n", " patterns=patts,\n", " cvs={\n", " \"source\": {\"CanESM-2\": \"CanESM2\"},\n", " \"institution\": {\"MIROC\": {\"domain\": \"global\"}},\n", " },\n", ")" ] }, { "cell_type": "markdown", "id": "60", "metadata": {}, "source": [ "##### Example 6 : Even more complex field processing\n", "In the preceding example, we used the `cvs` argument to replace values by others, or to trigger replacements based on values of other columns. The exact value must be matched and map to exact values. Another alternative to transform the parsed fields is to feed a function to the path parsing step. This is done by declaring a new \"type\" to the parser. In the following example, we'll implement a very useless transformation that reverses the letters of the institution." ] }, { "cell_type": "code", "execution_count": null, "id": "61", "metadata": {}, "outputs": [], "source": [ "from xscen.catutils import register_parse_type\n", "\n", "\n", "@register_parse_type(\"rev\")\n", "def _reverse_word(text):\n", " return \"\".join(reversed(text))\n", "\n", "\n", "patts_mod = [\n", " \"{institution:rev}/{source}_{version}/{frequency}/{variable:_}/{?:_}_{DATES}.nc\",\n", " \"{institution:rev}/{source}/{frequency}/{variable:_}/{?:_}_{DATES}.nc\",\n", "]\n", "parse_directory(directories=[root], patterns=patts_mod)" ] }, { "cell_type": "markdown", "id": "62", "metadata": {}, "source": [ "### Restructuring catalogued files on disk\n", "\n", "The opposite operation to `parse_directory` is also handled by `xscen.catutils`. In this section, we show how to create a Path from a xscen-extraced dataset or from a catalog entry.\n", "\n", "#### Simple : template string and attributes\n", "Given a dataset that was opened by `xs.extract_dataset` or `DataCatalog.to_dataset()`, we can easily construct a path from the xscen-added attributes." ] }, { "cell_type": "code", "execution_count": null, "id": "63", "metadata": {}, "outputs": [], "source": [ "# Open\n", "ds = PC.search(variable=\"tas\", experiment=\"ssp585\").to_dataset()\n", "\n", "path_template = \"{institution}/{source}/{experiment}_{frequency}.nc\"\n", "\n", "print(path_template.format(**xs.utils.get_cat_attrs(ds)))" ] }, { "cell_type": "markdown", "id": "64", "metadata": {}, "source": [ "While this method is simple, it can't handle neither the list-like `variable` field nor the `date_start` and `date_end` datetime fields.\n", "\n", "#### Complete : build_path\n", "The [`build_path`](../xscen.rst#xscen.catutils.build_path) function has a more complex interface to be used in more complex workflows.\n", "\n", "The default parameters has a pretty good folder structure that depends on the columns `type` (usually one of simulation, reconstruction or station-obs) and `processing_level` (often raw, biasadjusted or something else)." ] }, { "cell_type": "code", "execution_count": null, "id": "65", "metadata": {}, "outputs": [], "source": [ "xs.catutils.build_path(ds)" ] }, { "cell_type": "markdown", "id": "66", "metadata": {}, "source": [ "The folder schema can be passed explicitly, as a dictionary with two entries:\n", "- \"folders\" : a list of fields to build the folder hierarchy.\n", "- \"filename\" : a list of fields to build the filename.\n", "\n", "In both cases, a special \"DATES\" field can be given. It will be translated to the most efficient way to write the temporal bounds of the dataset." ] }, { "cell_type": "code", "execution_count": null, "id": "67", "metadata": {}, "outputs": [], "source": [ "custom_schema = {\n", " \"folders\": [\"type\", \"institution\", \"source\", \"experiment\"],\n", " \"filename\": [\"variable\", \"DATES\"],\n", "}\n", "xs.catutils.build_path(ds, schemas=custom_schema)" ] }, { "cell_type": "markdown", "id": "68", "metadata": {}, "source": [ "The function has more options:\n", "\n", "- A \"root\" folder can be specified\n", "- Other fields can be passed to override those in the data or fill for missing ones." ] }, { "cell_type": "code", "execution_count": null, "id": "69", "metadata": {}, "outputs": [], "source": [ "xs.catutils.build_path(ds, root=Path(\"/tmp\"), domain=\"REG\")" ] }, { "cell_type": "markdown", "id": "70", "metadata": {}, "source": [ "Above, we called the function with a dataset. In this case, the \"facets\" are extracted from various sources, with this priority (highest at the top):\n", "\n", "1. Facets passed explicitly to `build_path` as keyword arguments\n", "2. Attributes prefixed with \"cat:\"\n", "3. Other Attributes\n", "4. variable names, start and end date, and frequency, as extracted by `parse_from_date`.\n", "\n", "But the function can also take a single dataframe row:" ] }, { "cell_type": "code", "execution_count": null, "id": "71", "metadata": {}, "outputs": [], "source": [ "xs.catutils.build_path(PC.search(variable=\"tas\", experiment=\"ssp585\").df.iloc[0])" ] }, { "cell_type": "markdown", "id": "72", "metadata": {}, "source": [ "Or a full DataFrame/Catalog. In this case, the return value is a DataFrame, copy form the catalog, with a \"new_path\" column added." ] }, { "cell_type": "code", "execution_count": null, "id": "73", "metadata": {}, "outputs": [], "source": [ "# We show only three columns of the output catalog\n", "xs.catutils.build_path(PC.search(variable=\"tas\"))[[\"id\", \"path\", \"new_path\"]]" ] }, { "cell_type": "markdown", "id": "74", "metadata": {}, "source": [ "This can be used in a workflow that renames or copies the files to their new name, usually using `shutil`." ] }, { "cell_type": "code", "execution_count": null, "id": "75", "metadata": {}, "outputs": [], "source": [ "import shutil as sh\n", "\n", "# Create the destination folder\n", "root = Path(\".\").absolute() / \"_data\" / \"path_builder_examples\"\n", "root.mkdir(exist_ok=True)\n", "\n", "# Get new names:\n", "newdf = xs.catutils.build_path(PC, root=root)\n", "\n", "# Copy files\n", "for i, row in newdf.iterrows():\n", " Path(row[\"new_path\"]).parent.mkdir(parents=True, exist_ok=True)\n", " sh.copy(row[\"path\"], row[\"new_path\"])\n", " print(f\"Copied {row['path']}\\n\\tto {row['new_path']}\")\n", "\n", "# Update catalog:\n", "PC.df[\"path\"] = newdf[\"new_path\"]\n", "PC.update()" ] }, { "cell_type": "code", "execution_count": null, "id": "76", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "@webio": { "lastCommId": null, "lastKernelId": null }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.11" } }, "nbformat": 4, "nbformat_minor": 5 }