How can I access the filenames gathered by tf.data.Dataset.list_files()?

Question

I am using

file_data = tf.data.Dataset.list_files("../*.png")

to collect image files for training in TensorFlow, but would like to access the list of gathered filenames so I can perform a label lookup.

Calling sess.run([file_data]) has been unsuccessful:

TypeError: Fetch argument <TensorSliceDataset shapes: (), types: tf.string> has invalid type <class 'tensorflow.python.data.ops.dataset_ops.TensorSliceDataset'>, must be a string or Tensor. (Can not convert a TensorSliceDataset into a Tensor or Operation.)

Are there any other methods I can use?

Answer 1

With some additional experimenting, I found a way to solve this:

First, turn the Dataset into an iterator:

iterator_helper = file_data.make_one_shot_iterator()

Then, iterate through the elements in a tf Session:

with tf.Session() as sess:
    filename_temp = iterator_helper.get_next()
    print(sess.run[filename_temp])

Answer 2

The Dataset.list_files() API uses the tf.matching_files() op to list the files matching the given pattern. You can also get the list of files as a tf.Tensor using that op, and pass it directly to sess.run():

filenames_as_tensor = tf.matching_files("../*.png")
filenames_as_array = sess.run(filenames_as_tensor)

for filename in filenames_as_array:
  print(filename)

Answer 3

Here is how I've done it in Tensorflow 2

def load_image_train(image_file):
  """ 
      a function to load image and return
      the image and it's address
  """
  my_image = load_image_func(image_file)    
  return my_image, image_file

Then use tf.data.Dataset.list_files to load the list of files we have in a folder:

PATH = "path_to_dataset_folder"
train_dataset_names = tf.data.Dataset.list_files(os.path.join(PATH , 'train/*.jpg'))  

Finally map them so you can have both "file address" and "data" as a tensor:

train_dataset = train_dataset_names.map(load_image_train,
                         num_parallel_calls=tf.data.AUTOTUNE)

Then you can seperate them as you wish and use the file name as a label or whatever.