# Changelog

All notable changes to this project are documented here.

---

## 2026-03-18 (4)

### Fixed
- **Root logger level `DEBUG` → `INFO`** — `setup_logging()` in `hls_utils.py` set the
  root logger to `DEBUG`, causing rasterio and GDAL internal trace messages to flood the
  log (thousands of `DEBUG [rasterio.env]` / `DEBUG [rasterio._io]` lines per run).
  Changed to `INFO` so only pipeline-level messages and library `WARNING`+ output appear.
- **Steps 04/05: `"Skipped"` results logged at `ERROR`** — the result-dispatch logic in
  the main loop of steps 04 and 05 routed any string not starting with `"OK"` or
  `"WARNING"` to `logger.error`. `"Skipped (Exists)"` (step 04) and
  `"Skipped (No outliers)"` (step 05) both fell into this branch. Added an explicit
  `Skipped` prefix check → `logger.info` before the error fallthrough.
- **Steps 09/10: `BLOCKXSIZE` without `TILED=YES` in temp tile writes** — `to_raster()`
  calls for intermediate temp GeoTIFFs in steps 09 and 10 passed `blockxsize`/`blockysize`
  without `tiled=True`, producing GDAL `CPLE_IllegalArg` warnings on every tile. Added
  `tiled=True` to all three affected `to_raster()` calls.
- **Step 09: dask chunk mismatch `UserWarning`** — `xr.open_dataset(nc_path, chunks={'time': 10})`
  produced a dask performance warning when the on-disk NetCDF chunk layout differed from
  the requested chunking. Changed to `chunks='auto'` to align with on-disk layout,
  consistent with how steps 04, 05, and 10 open datasets.
- **Step 03: CRS remap log message rewording** — the southern hemisphere CRS adjustment
  log line used `[CRS fix]` and `corrected to`, implying an error condition. Replaced
  with `[CRS]` and `southern hemisphere tile, remapped … → …` to reflect that this is
  routine, expected processing for any southern hemisphere tile.

---

## 2026-03-18 (3)

### Added
- **Structured logging across all Python steps and `hls_pipeline.sh`** — all ten
  Python pipeline scripts (steps 02–11) now use Python's `logging` module via a
  shared `setup_logging(step_name)` helper in `src/hls_utils.py`. Every log line
  carries a timestamp, level, and bracketed step label:
  `2026-03-18 20:55:49  INFO      [04_mean_reproject]  message`. Previously all
  diagnostic output used bare `print()` calls with no timestamps or severity levels.
  Key design points:
  - Single implementation in `hls_utils.py`; no logging boilerplate duplicated
    across scripts.
  - Root logger handler guard (`if not root.handlers`) makes `setup_logging`
    idempotent — calling it in `multiprocessing.Pool` or `ProcessPoolExecutor`
    child processes does not produce duplicate output.
  - Worker functions (steps 02–05, 09–10) are unchanged; they return status
    strings/dicts to the main process, which performs all `logger.*()` calls.
  - `StreamHandler` targets `sys.stdout` so `2>&1 | tee -a "$LOGFILE"` in the
    shell captures all output.
  - `hls_pipeline.sh` gains `log_info`, `log_warn`, and `log_error` helper
    functions that emit the same timestamp + level + `[pipeline]` format, making
    combined shell/Python log output visually consistent.

---

## 2026-03-18 (2)

### Fixed
- **Step 03 — southern hemisphere CRS stored as UTM North** — HLS v2.0 GeoTIFFs
  for tiles south of the equator embed a UTM North zone (EPSG:326xx) with negative
  northings instead of the standard UTM South convention (EPSG:327xx,
  false_northing=10,000,000). All southern Africa (BioSCape) and other southern
  hemisphere tiles were affected. `HLSNetCDFAggregator.run()` in
  `src/03_hls_netcdf_build.py` now detects this case after reading the first
  GeoTIFF: if `pyproj.to_epsg(min_confidence=20)` returns a UTM North code
  (32601–32660) and the pixel-center y mean is negative, the CRS WKT is replaced
  with the UTM South equivalent (EPSG + 100, e.g. 32634 → 32734) and
  y-coordinates are shifted by +10,000,000 m. This correction is applied before
  chunk dicts are built, so both single-chunk and merged tiles are written with
  the correct EPSG:327xx CRS and positive UTM South northings. Previously rebuilt
  tiles will need to be regenerated with step 03 to pick up the corrected CRS and
  coordinates; downstream steps 04–11 that reproject to `TARGET_CRS` are not
  affected because they perform a full reprojection from the source CRS.

---

## 2026-03-18

### Fixed
- **Step 03 — `_FillValue` lost in `merge_chunks`** — `process_netcdf_chunk` correctly
  creates the VI variable with `fill_value=np.nan`, but `merge_chunks` recreated the
  same variable without a `fill_value` argument. netCDF4 therefore fell back to its
  built-in default sentinel (`9.969209968386869e+36`) for all missing cells in merged
  files, and the `_FillValue` attribute was absent from the output. Any tile requiring
  chunk merging (virtually all multi-year tiles with more acquisitions than `CHUNK_SIZE`)
  was affected. Fixed by adding `fill_value=np.nan` to the `createVariable` call in
  `merge_chunks` (`src/03_hls_netcdf_build.py` line 231). Newly rebuilt tiles will store
  missing data as `NaN` and carry a proper `_FillValue = NaN` attribute.

---

## 2026-03-12

### Changed
- **Pipeline scripts moved to `src/`** — all 11 step scripts
  (`01_hls_download_query.sh` – `11_hls_outlier_gpkg.py`) and `hls_utils.py`
  relocated from the repository root into `src/`. `hls_pipeline.sh` remains at
  the root. All invocation paths in `hls_pipeline.sh`, `CLAUDE.md`, `README.md`,
  and `docs/` updated accordingly. Python `import hls_utils` statements are
  unaffected (Python resolves the import from the script's own directory).
- **Step 03 — improved CF-1.8 CRS metadata in NetCDF output** — the
  `spatial_ref` grid-mapping variable now carries both `crs_wkt` (CF-1.8
  standard) and `spatial_ref` (GDAL / rioxarray compatibility) attributes, plus
  `grid_mapping_name` (derived via pyproj) and `long_name`. The `x`/`y`
  coordinate variables now include `standard_name`, `long_name`, and `axis`
  attributes; the `time` variable now includes `standard_name`, `calendar`, and
  `axis`. A global `Conventions = "CF-1.8"` attribute is now written. The
  `merge_chunks` path mirrors all the same attributes. These changes make
  `da.rio.crs` (rioxarray path 1 in `detect_crs()`) reliably resolve without
  falling back to the global `crs` attribute. Existing NetCDF files built with
  the prior format remain readable via the `detect_crs()` fallback chain.
- **Step 03 — CRS WKT stored as pyproj WKT2 instead of GDAL WKT1** —
  `HLSNetCDFAggregator.run()` now generates the CRS WKT string via
  `ProjCRS.from_user_input(crs).to_wkt()` (pyproj WKT2) instead of
  rasterio's `crs.to_wkt()` (GDAL WKT1). GDAL WKT1 for some HLS tiles
  lacks a top-level `AUTHORITY["EPSG","XXXXX"]` node, causing
  `pyproj.CRS.from_wkt(wkt).to_epsg()` to return `None`. Downstream
  consumers that group tiles by EPSG code (e.g. cross-CRS reprojection
  checks) would treat same-zone tiles as different CRS groups. The pyproj
  WKT2 output always includes a resolvable authority node. Existing NetCDF
  files retain their original WKT; rebuilding with step 03 is recommended
  for tiles where EPSG grouping matters downstream.

---

## 2026-02-28

### Added
- **`NETCDF_COMPLEVEL`** — configurable zlib compression level (0–9, default `1`)
  for NetCDF time-series files written by step 03. Threaded through
  `HLSNetCDFAggregator` into `chunk_info` dicts (worker) and `merge_chunks`.
- **`GEOTIFF_COMPRESS`** — configurable compression codec (default `LZW`) for all
  GeoTIFF outputs in steps 02 and 04–10. Accepts any codec supported by the
  local GDAL build (`LZW`, `DEFLATE`, `ZSTD`, `NONE`).
- **`GEOTIFF_BLOCK_SIZE`** — configurable internal tile block dimension in pixels
  (default `512`) for all tiled GeoTIFF outputs in steps 04–10. `512` is
  standard for desktop GIS; `256` is preferred for Cloud-Optimized GeoTIFFs.
- **`reproject_resolution()` in `hls_utils.py`** — CRS-unit-aware resolution
  helper replacing all hardcoded `resolution=30` calls in steps 04, 05, 09, 10.
  Returns metres unchanged for projected CRS; converts to approximate degrees
  for geographic CRS and logs a warning.

### Fixed
- Steps 04, 05, 09, and 10 produced a 1×1 pixel output with no valid data when
  `TARGET_CRS` was set to a geographic CRS (e.g. `EPSG:4148`) because
  `resolution=30` was interpreted as 30 degrees per pixel instead of 30 metres.

---

## 2026-02-26

### Added
- Read the Docs configuration and Sphinx documentation scaffold (`docs/`)
- `docs/overview.md`: comprehensive pipeline guide (full user documentation)

### Changed
- README.md restructured as a GitHub landing page (elevator pitch, outputs
  table, key features, quick start, and link to RTD); full documentation
  moved to `docs/overview.md`
- `docs/index.md` updated to a hub toctree (overview, configuration,
  changelog); no longer uses `{include}` to pull README content

### Fixed
- System requirements table in README: added `gdalinfo` (called directly by
  step 01 for GeoTIFF validation; provided by the conda environment via
  rasterio's GDAL dependency); clarified that conda is required not just for
  Python packages but because it supplies native geospatial libraries (GDAL,
  PROJ, HDF5, GEOS)
- Per-file download validation with retry logic in step 01

### Changed
- `NUM_WORKERS` restored to `8` in `config.env`

### Removed
- Bulk download mode retired; tile-by-tile is now the only download mode,
  reducing peak disk usage to roughly one tile's worth of raw data at a time

---

## 2026-02-25

### Added
- **Step 09 — CountValid mosaic**: counts valid (unmasked, in-range) observations
  per pixel across all download cycles and mosaics the result into a single
  study-area-wide GeoTIFF. Reads from NetCDF files (step 03); independent of
  `TIMESLICE_WINDOWS` and the time-series step.

### Changed
- Steps renumbered to reflect execution order:
  - Former step 09 (time-series) → **Step 10**
  - Former step 10 (outlier GeoPackage) → **Step 11**

---

## 2026-02-22

### Added
- Initial release of the HLS Vegetation Index Pipeline
- 11-step end-to-end workflow: download → VI calculation → NetCDF →
  reprojection → mosaics → time-series → outlier export
- Support for NDVI, EVI2, and NIRv vegetation indices
- Bitwise Fmask quality masking with independently configurable flags for
  cirrus, cloud, adjacent cloud, shadow, snow/ice, water, and aerosol mode
- Tile-by-tile orchestration for steps 01–03, with optional space-saver
  flags to remove raw and/or VI intermediate files after each tile's NetCDF
  is built
- Configurable parallel processing via `NUM_WORKERS`
- Per-VI valid range outlier detection with configurable bounds
  (`VALID_RANGE_NDVI`, `VALID_RANGE_EVI2`, `VALID_RANGE_NIRv`)
- Multi-band seasonal composite stacks via `TIMESLICE_WINDOWS` (step 10)
- GeoPackage export of per-pixel outlier observations with WGS84 coordinates
  (step 11)
- Pre-flight band validation: the orchestrator checks that all bands required
  for the selected VIs are configured before any step executes
- `SKIP_APPROVAL` flag for automated / non-interactive pipeline runs