Extracting dfs0 from large dfs2 files

Question

I want to extract a dfs0 from a large dfs2 file and try:

from mikeio import Dfs2
dfs = Dfs2(dfs2_filename)
ds = dfs.read()
(k,j) = dfs.find_nearest_element(lon,lat)
ds2 = ds.isel(k).isel(j)
ds2.to_dfs0(dfs0_filename)

This works fine most of the time, however, for very large dfs2 files I get a memory error. I can reduce memory cost with those options:

ds = dfs.read(items=items,time_steps=time_steps)

However, I am interested in all items and time steps. So my current work around is to take chunks in the time dimension, something like this:

n = dfs.n_timesteps;
dChunk = int(np.ceil(n/nChunks))
for i_ch in range(chunks):
   time_steps = list(range((i_ch*dChunk),np.min([(i_ch+1)*dChunk,n])))
   ds = dfs.read(items=items,time_steps=time_steps)
   (k,j) = dfs.find_nearest_element(lon,lat)
   ds2 = ds.isel(k).isel(j)

I then save the chunks as temporary files, followed by a concat.

What is the better way to do this?

Can I do some form of 'isel' before the 'read'? Similar to the 'element' option of Dfsu:

dfsu.read(items=None, time_steps=None, elements=None)

Thanks!

Answer 1

Currently, no elements or similar argument exists for Dfs2 read method (like it does for Dfsu). I think the read method should have such an argument, so please suggest this in https://github.com/DHI/mikeio/issues

I think your chunked version is a good way to handle the memory limitation.

Answer 2

It should be doable for any size dfs2 file using the mikecore python package,

There is an example for extracting dfs0 files from dfsu files, called ExtractDfs0FromDfsu in

https://github.com/DHI/mikecore-python/blob/master/tests/examples_dfsu.py

If knowing the index of the element(s) in the dfs2 to extract from, that should work very similarly.

That would be a very low-memory approach, only having one dfs2 timestep in memory at the time.

The mike core python package is available from pypi:

https://pypi.org/project/mikecore/