Tensorflow1 tf.Session Ignore GPU certification

Question

When I limit GPU by os.environ["CUDA_VISIBLE_DEVICES"]="1" and load trained model like below, tacotron1.15.5[horovod] load model to all GPUs(8) with same process ID. GPU memory Usage

Have anyone suffered problem like me? I think it is related with version problem.

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
config.allow_soft_placement = True
        
self.session = tf.Session(config=config)

Tensorflow 1.15.5
Cuda : 11.0
driver version 450.172.01

Answer 1

You can do this with Tensorflow v2.x. Since TF v1.x is not actively supporting, you can use TF v2.8.0.

If you would like a particular operation to run on a device of your choice instead of what's automatically selected for you, you can use with tf.device to create a device context, and all the operations within that context will run on the same designated device.

import tensorflow as tf
tf.debugging.set_log_device_placement(True)

# Place tensors on the GPU
with tf.device('/GPU:0'):
  a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
  b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])

# Run on the GPU
c = tf.matmul(a, b)
print(c)

Output

Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0
Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0
Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0
tf.Tensor(
[[22. 28.]
 [49. 64.]], shape=(2, 2), dtype=float32)