Precomputed Median Mosaics

[BillyDodds]

Hi all,

I have fixed some bugs present in the Sentinel-2_precomputed_mosaics notebook, replacing the sentinel_mosaic.py file with a revised SentinelMosaic.py file. This file contains a SentinelMosaic class that will assist in loading the precomputed mosaics. See the Sentinel-2_precomputed_mosaics notebook for how to use it.

This revised version is much faster than the previous code. The .get_mosaic function is by far the most effective way to download rich, cloudless Sentinel-2 data directly to your disk where it can be easily read and accessed for your models. When querying a new region, the method will download the required tiles to a specified location on your computer. Once the scene has been downloaded once, it does not need to be downloaded again! Instead, the second time it will be read from directly from your computer or VM storage, greatly speeding up your data pipeline.

Follow the example notebook here to take full advantage of the precomputed mosaics in your data pipeline. Don't miss this opportunity to make your model even better!

Please reply here if any issues arise surrounding the use of the precomputed mosaics.

Just 15 days to go! All the best!

Kind regards,
Billy Dodds

[jonasanmaru]

Hi @BillyDodds ,

I just tested out and it works well!

However, I am just curious why the following codes doesn't work when a dask GatewayCluster is created at the beginning of the script.
Note: Converting the Xarray to panda dataframe without Dask works fine.

*I am not sure if this have to do with how xr.open_mfdataset behave with dask.

P/s: I think I found the issues, reference: pydata/xarray#4213

About dask, indeed is helping a lot to create a delayed object of the tiles (consider that at least for S2 data are in jp2 and we are forced to use open_rasterio instead of open_mfdataset) so the solution should be compatible with this kind of approach.

from dask_gateway import GatewayCluster
cluster = GatewayCluster()  # Creates the Dask Scheduler. Might take a minute.
client = cluster.get_client()
cluster.adapt(minimum=1, maximum=100)
cluster

bbox = (152.4, -31.9, 152.9, -31.4)
median = mosaic_downloader.get_mosaic(bbox)
median.to_dataframe()

Error received:

---------------------------------------------------------------------------
RasterioIOError                           Traceback (most recent call last)
Input In [15], in <module>
----> 1 median.to_dataframe()

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/dataarray.py:2784, in DataArray.to_dataframe(self, name, dim_order)
   2781 else:
   2782     ordered_dims = ds._normalize_dim_order(dim_order=dim_order)
-> 2784 df = ds._to_dataframe(ordered_dims)
   2785 df.columns = [name if c == unique_name else c for c in df.columns]
   2786 return df

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/dataset.py:5362, in Dataset._to_dataframe(self, ordered_dims)
   5360 def _to_dataframe(self, ordered_dims: Mapping[Any, int]):
   5361     columns = [k for k in self.variables if k not in self.dims]
-> 5362     data = [
   5363         self._variables[k].set_dims(ordered_dims).values.reshape(-1)
   5364         for k in columns
   5365     ]
   5366     index = self.coords.to_index([*ordered_dims])
   5367     return pd.DataFrame(dict(zip(columns, data)), index=index)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/dataset.py:5363, in <listcomp>(.0)
   5360 def _to_dataframe(self, ordered_dims: Mapping[Any, int]):
   5361     columns = [k for k in self.variables if k not in self.dims]
   5362     data = [
-> 5363         self._variables[k].set_dims(ordered_dims).values.reshape(-1)
   5364         for k in columns
   5365     ]
   5366     index = self.coords.to_index([*ordered_dims])
   5367     return pd.DataFrame(dict(zip(columns, data)), index=index)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/variable.py:510, in Variable.values(self)
    507 @property
    508 def values(self):
    509     """The variable's data as a numpy.ndarray"""
--> 510     return _as_array_or_item(self._data)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/variable.py:250, in _as_array_or_item(data)
    236 def _as_array_or_item(data):
    237     """Return the given values as a numpy array, or as an individual item if
    238     it's a 0d datetime64 or timedelta64 array.
    239 
   (...)
    248     TODO: remove this (replace with np.asarray) once these issues are fixed
    249     """
--> 250     data = np.asarray(data)
    251     if data.ndim == 0:
    252         if data.dtype.kind == "M":

File /srv/conda/envs/notebook/lib/python3.8/site-packages/numpy/core/_asarray.py:102, in asarray(a, dtype, order, like)
     99 if like is not None:
    100     return _asarray_with_like(a, dtype=dtype, order=order, like=like)
--> 102 return array(a, dtype, copy=False, order=order)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/dask/array/core.py:1541, in Array.__array__(self, dtype, **kwargs)
   1540 def __array__(self, dtype=None, **kwargs):
-> 1541     x = self.compute()
   1542     if dtype and x.dtype != dtype:
   1543         x = x.astype(dtype)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/dask/base.py:288, in DaskMethodsMixin.compute(self, **kwargs)
    264 def compute(self, **kwargs):
    265     """Compute this dask collection
    266 
    267     This turns a lazy Dask collection into its in-memory equivalent.
   (...)
    286     dask.base.compute
    287     """
--> 288     (result,) = compute(self, traverse=False, **kwargs)
    289     return result

File /srv/conda/envs/notebook/lib/python3.8/site-packages/dask/base.py:571, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    568     keys.append(x.__dask_keys__())
    569     postcomputes.append(x.__dask_postcompute__())
--> 571 results = schedule(dsk, keys, **kwargs)
    572 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py:2725, in Client.get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   2723         should_rejoin = False
   2724 try:
-> 2725     results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   2726 finally:
   2727     for f in futures.values():

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py:1980, in Client.gather(self, futures, errors, direct, asynchronous)
   1978 else:
   1979     local_worker = None
-> 1980 return self.sync(
   1981     self._gather,
   1982     futures,
   1983     errors=errors,
   1984     direct=direct,
   1985     local_worker=local_worker,
   1986     asynchronous=asynchronous,
   1987 )

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py:868, in Client.sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    866     return future
    867 else:
--> 868     return sync(
    869         self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    870     )

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py:332, in sync(loop, func, callback_timeout, *args, **kwargs)
    330 if error[0]:
    331     typ, exc, tb = error[0]
--> 332     raise exc.with_traceback(tb)
    333 else:
    334     return result[0]

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py:315, in sync.<locals>.f()
    313     if callback_timeout is not None:
    314         future = asyncio.wait_for(future, callback_timeout)
--> 315     result[0] = yield future
    316 except Exception:
    317     error[0] = sys.exc_info()

File /srv/conda/envs/notebook/lib/python3.8/site-packages/tornado/gen.py:762, in Runner.run(self)
    759 exc_info = None
    761 try:
--> 762     value = future.result()
    763 except Exception:
    764     exc_info = sys.exc_info()

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py:1845, in Client._gather(self, futures, errors, direct, local_worker)
   1843         exc = CancelledError(key)
   1844     else:
-> 1845         raise exception.with_traceback(traceback)
   1846     raise exc
   1847 if errors == "skip":

File /srv/conda/envs/notebook/lib/python3.8/site-packages/dask/array/core.py:116, in getter()
    111     # Below we special-case `np.matrix` to force a conversion to
    112     # `np.ndarray` and preserve original Dask behavior for `getter`,
    113     # as for all purposes `np.matrix` is array-like and thus
    114     # `is_arraylike` evaluates to `True` in that case.
    115     if asarray and (not is_arraylike(c) or isinstance(c, np.matrix)):
--> 116         c = np.asarray(c)
    117 finally:
    118     if lock:

File /srv/conda/envs/notebook/lib/python3.8/site-packages/numpy/core/_asarray.py:102, in asarray()
     99 if like is not None:
    100     return _asarray_with_like(a, dtype=dtype, order=order, like=like)
--> 102 return array(a, dtype, copy=False, order=order)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/indexing.py:358, in __array__()
    357 def __array__(self, dtype=None):
--> 358     return np.asarray(self.array, dtype=dtype)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/numpy/core/_asarray.py:102, in asarray()
     99 if like is not None:
    100     return _asarray_with_like(a, dtype=dtype, order=order, like=like)
--> 102 return array(a, dtype, copy=False, order=order)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/indexing.py:522, in __array__()
    521 def __array__(self, dtype=None):
--> 522     return np.asarray(self.array, dtype=dtype)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/numpy/core/_asarray.py:102, in asarray()
     99 if like is not None:
    100     return _asarray_with_like(a, dtype=dtype, order=order, like=like)
--> 102 return array(a, dtype, copy=False, order=order)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/indexing.py:552, in __array__()
    551 def __array__(self, dtype=None):
--> 552     self._ensure_cached()
    553     return np.asarray(self.array, dtype=dtype)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/indexing.py:549, in _ensure_cached()
    547 def _ensure_cached(self):
    548     if not isinstance(self.array, NumpyIndexingAdapter):
--> 549         self.array = NumpyIndexingAdapter(np.asarray(self.array))

File /srv/conda/envs/notebook/lib/python3.8/site-packages/numpy/core/_asarray.py:102, in asarray()
     99 if like is not None:
    100     return _asarray_with_like(a, dtype=dtype, order=order, like=like)
--> 102 return array(a, dtype, copy=False, order=order)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/indexing.py:522, in __array__()
    521 def __array__(self, dtype=None):
--> 522     return np.asarray(self.array, dtype=dtype)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/numpy/core/_asarray.py:102, in asarray()
     99 if like is not None:
    100     return _asarray_with_like(a, dtype=dtype, order=order, like=like)
--> 102 return array(a, dtype, copy=False, order=order)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/indexing.py:423, in __array__()
    421 def __array__(self, dtype=None):
    422     array = as_indexable(self.array)
--> 423     return np.asarray(array[self.key], dtype=None)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/rioxarray/_io.py:301, in __getitem__()
    300 def __getitem__(self, key):
--> 301     return indexing.explicit_indexing_adapter(
    302         key, self.shape, indexing.IndexingSupport.OUTER, self._getitem
    303     )

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/indexing.py:712, in explicit_indexing_adapter()
    690 """Support explicit indexing by delegating to a raw indexing method.
    691 
    692 Outer and/or vectorized indexers are supported by indexing a second time
   (...)
    709 Indexing result, in the form of a duck numpy-array.
    710 """
    711 raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support)
--> 712 result = raw_indexing_method(raw_key.tuple)
    713 if numpy_indices.tuple:
    714     # index the loaded np.ndarray
    715     result = NumpyIndexingAdapter(np.asarray(result))[numpy_indices]

File /srv/conda/envs/notebook/lib/python3.8/site-packages/rioxarray/_io.py:284, in _getitem()
    282 else:
    283     with self.lock:
--> 284         riods = self.manager.acquire(needs_lock=False)
    285         if self.vrt_params is not None:
    286             riods = WarpedVRT(riods, **self.vrt_params)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/backends/file_manager.py:181, in acquire()
    166 def acquire(self, needs_lock=True):
    167     """Acquire a file object from the manager.
    168 
    169     A new file is only opened if it has expired from the
   (...)
    179         An open file object, as returned by ``opener(*args, **kwargs)``.
    180     """
--> 181     file, _ = self._acquire_with_cache_info(needs_lock)
    182     return file

File /srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/backends/file_manager.py:205, in _acquire_with_cache_info()
    203     kwargs = kwargs.copy()
    204     kwargs["mode"] = self._mode
--> 205 file = self._opener(*self._args, **kwargs)
    206 if self._mode == "w":
    207     # ensure file doesn't get overriden when opened again
    208     self._mode = "a"

File /srv/conda/envs/notebook/lib/python3.8/site-packages/rasterio/env.py:437, in wrapper()
    434     session = DummySession()
    436 with env_ctor(session=session):
--> 437     return f(*args, **kwds)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/rasterio/__init__.py:220, in open()
    216 # Create dataset instances and pass the given env, which will
    217 # be taken over by the dataset's context manager if it is not
    218 # None.
    219 if mode == 'r':
--> 220     s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
    221 elif mode == "r+":
    222     s = get_writer_for_path(path, driver=driver)(
    223         path, mode, driver=driver, sharing=sharing, **kwargs
    224     )

File rasterio/_base.pyx:263, in rasterio._base.DatasetBase.__init__()

RasterioIOError: /tmp/EastAustralia/S31_50_E152_75/red_median.tif: No such file or directory`

[BillyDodds]

Hi @jonasanmaru .

I'm not sure, but my suggestion would be to try and edit the class yourself. Like you said, maybe using something other than open_mfdataset will work. Let me know how you go!

Cheers,
Billy Dodds