Pangeo Forge Recipes Versions Save

What's Changed

• Add SciPy video and slides to what_is_pangeo_forge.md by @rabernat in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/416
• Document how to access data with Globus by @rabernat in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/408
• Remove pre-commit action in favor of pre-commit.ci by @andersy005 in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/423
• Cancel Redundant CI jobs by @andersy005 in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/424
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/381
• Fix CI by @andersy005 in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/430
• Avoid serialization blues by computing + caching the hash. by @alxmrs in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/429
• Bump peter-evans/slash-command-dispatch from 2 to 3 by @dependabot in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/434
• Bump codecov/codecov-action from 2.0.2 to 3.1.1 by @dependabot in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/431
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/426
• Bump actions/setup-python from 3 to 4 by @dependabot in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/433
• Bump LouisBrunner/checks-action from 1.2.0 to 1.5.0 by @dependabot in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/432
• Update release notes in preparation for new release by @andersy005 in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/436

New Contributors

• @andersy005 made their first contribution in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/423
• @pre-commit-ci made their first contribution in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/381
• @dependabot made their first contribution in https://github.com/pangeo-forge/pangeo-forge-recipes/pull/434

Full Changelog: https://github.com/pangeo-forge/pangeo-forge-recipes/compare/0.9.1...0.9.2

• Persist Pangeo Forge execution context metadata in target datasets. This information, which includes the pangeo-forge-recipes version as well as recipe and input hashes, attaches execution provenance to the dataset itself. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/359
• File pattern support for .tiffs. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/393
• Improved HDFReferenceRecipe by passing target_options to the MultiZarrToZarr class.
• Typo fixes, documentation updates, and project health improvements. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/364, https://github.com/pangeo-forge/pangeo-forge-recipes/pull/365, https://github.com/pangeo-forge/pangeo-forge-recipes/pull/366, https://github.com/pangeo-forge/pangeo-forge-recipes/pull/388, https://github.com/pangeo-forge/pangeo-forge-recipes/pull/396, https://github.com/pangeo-forge/pangeo-forge-recipes/pull/394, https://github.com/pangeo-forge/pangeo-forge-recipes/pull/398, https://github.com/pangeo-forge/pangeo-forge-recipes/pull/407.
• Fixed XarrayZarrRecipe's finalize_target() stage by using bulk delete APIs for consolidating Zarr coordinates.

• Breaking changes: Deprecated XarrayZarrRecipe manual stage methods. Also deprecated FilePattern(..., is_opendap=True) kwarg, which is superseded by FilePattern(..., file_type="opendap"). https://github.com/pangeo-forge/pangeo-forge-recipes/pull/362
• Added serialization module along with BaseRecipe.sha256 and FilePattern.sha256 methods. Collectively, this provides for generation of deterministic hashes for both recipe and file pattern instances. Checking these hashes against those from a prior version of the recipe can be used to determine whether or not a particular recipe instance in a Python module (which may contain arbitrary numbers of recipe instances) has changed since the last time the instances in that module were executed. The file pattern hashes are based on a merkle tree built cumulatively from all of the index:filepath pairs yielded by the pattern's self.items() method. As such, in cases where a new pattern is intended to append to an existing dataset which was built from a prior version of that pattern, the pattern hash can be used to determine the index from which to begin appending. This is demonstrated in the tests. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/349
• Created new Prefect executor which wraps the Dask executor in a single Task. This should mitigate problems related to large numbers of Prefect Tasks (https://github.com/pangeo-forge/pangeo-forge-recipes/issues/347).
• Implemented feature to cap cached filename lengths at 255 bytes on local filesystems, to accommodate the POSIX filename length limit. Cached filename lengths are not truncated on any other filesystem. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/353

• Added .file_type attribute to pangeo_forge_recipes.patterns.FilePattern. This attribute will eventually supercede .is_opendap, which will be deprecated in 0.9.0. Until then, FilePattern(..., is_opendap=True) is supported as equivalent to FilePattern(..., file_type="opendap"). https://github.com/pangeo-forge/pangeo-forge-recipes/pull/322

• Removed click from dependencies and removed cli entrypoint.

• Fixed dependency issue with pip installation.
• Fixed bug where recipes would fail if the target chunks exceeded the full array length. https://github.com/pangeo-forge/pangeo-forge-recipes/issues/279

• Breaking change: Replace recipe classes' storage attibutes with .storage_config of type pangeo_forge_recipes.storage.StorageConfig. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/288
• Add setup_logging convenience function. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/287

• Apache Beam executor added. https://github.com/pangeo-forge/pangeo-forge-recipes/issues/169. By Alex Merose. ❤️
• Dask executor updates. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/260, https://github.com/pangeo-forge/pangeo-forge-recipes/pull/261
• Index type update. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/257
• Fix incompatibility with fsspec>=2021.11.1. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/247

• Major internal refactor of executors. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/219.
• Began deprecation cycle for recipe methods (e.g. recipe.prepare_target()) in favor of module functions.
• Addition of open_input_with_fsspec_reference option on pangeo_forge_recipes.recipes.XarrayZarrRecipe, permitting the bypassing of h5py when opening inputs. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/218.

• Added pangeo_forge_recipes.recipes.HDFReferenceRecipe class to create virtual Zarrs from collections of NetCDF / HDF5 files. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/174
• Limit output from logging. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/175
• Change documentation structure. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/178
• Move fsspec_open_kwargs and is_opendap parameters out of pangeo_forge_recipes.recipes.XarrayZarrRecipe and into pangeo_forge_recipes.patterns.FilePattern. Add query_string_secrets as attribute of pangeo_forge_recipes.patterns.FilePattern. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/167