'Tensorflow dataset element shuffle within specified range
How do I shuffle the elements of tf.data.Dataset within a certain range. Having an input array, with shape = (10,), in the first 5 elements would be shuffled within the first 5 places and the next 5 withing the next 5 places, and so on.
ex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] ==> [3, 4, 2, 5, 1, 7, 10, 9, 6, 8]
Below is the function I implemented, which works on numpy arrays, but not on tensorflow data tensors.
Could someone please help me out with advice?
def indexShifting(my_array):
lim = my_array.shape[0]
shift = 5 # limits the range of shuffle
rng = np.random.default_rng()
for i in range(floor(lim/shift)): # 10/5
rng.shuffle(my_array[(i*shift):((i+1)*shift)])
return my_array
a = np.arange(1,11)
> array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
indexShifting(a)
> array([ 3, 4, 2, 5, 1, 7, 10, 9, 6, 8])
With tf.data.Dataset, there are a few errors I receive.
- len is not well defined for symbolic Tensors.
- 'tensorflow.python.framework.ops.EagerTensor' object does not support item assignment
This is dummy data.
a = np.random.permutation(10)
b = np.random.permutation(10)
c = np.random.permutation(10)
X = np.vstack((a,b,c))
y = np.expand_dims(np.ones(3), axis=1)
train_ds = tf.data.Dataset.from_tensor_slices((X, y))
train_ds = train_ds.map(lambda x, y: (indexShifting(x), y))
TF = 2.5.0 Python = 3.8.10
Any help and advice would help me out a lot. Thanks :)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|