Implement Relu derivative in python numpy

Question

I'm trying to implement a function that computes the Relu derivative for each element in a matrix, and then return the result in a matrix. I'm using Python and Numpy.

Based on other Cross Validation posts, the Relu derivative for x is 1 when x > 0, 0 when x < 0, undefined or 0 when x == 0

Currently, I have the following code so far:

def reluDerivative(self, x):
    return np.array([self.reluDerivativeSingleElement(xi) for xi in x])

def reluDerivativeSingleElement(self, xi):
    if xi > 0:
        return 1
    elif xi <= 0:
        return 0

Unfortunately, xi is an array because x is an matrix. reluDerivativeSingleElement function doesn't work on array. So I'm wondering is there a way to map values in a matrix to another matrix using numpy, like the exp function in numpy?

Thanks a lot in advance.

Answer 1

I guess this is what you are looking for:

>>> def reluDerivative(x):
...     x[x<=0] = 0
...     x[x>0] = 1
...     return x

>>> z = np.random.uniform(-1, 1, (3,3))
>>> z
array([[ 0.41287266, -0.73082379,  0.78215209],
       [ 0.76983443,  0.46052273,  0.4283139 ],
       [-0.18905708,  0.57197116,  0.53226954]])
>>> reluDerivative(z)
array([[ 1.,  0.,  1.],
       [ 1.,  1.,  1.],
       [ 0.,  1.,  1.]])

Answer 2

That's an exercise in vectorization.

This code

if x > 0:
  y = 1
elif xi <= 0:
  y = 0

Can be reformulated into

y = (x > 0) * 1

This is something that will work for numpy arrays, since boolean expressions involving them are turned into arrays of values of these expressions for elements in said array.

Answer 3

Basic function to return derivative of relu could be summarized as follows:

f'(x) = x > 0

So, with numpy that would be:

def relu_derivative(z):
    return np.greater(z, 0).astype(int)

Answer 4

def dRelu(z):
    return np.where(z <= 0, 0, 1)

Here z is a ndarray in my case.

Answer 5

def reluDerivative(self, x): 
    return 1 * (x > 0)

Answer 6

You are on a good track: thinking on vectorized operation. Where we define a function, and we apply this function to a matrix, instead of writing a for loop.

This threads answers your question, where it replace all the elements satisfy the condition. You can modify it into ReLU derivative.

https://stackoverflow.com/questions/19766757/replacing-numpy-elements-if-condition-is-met

In addition, python supports functional programming very well, try to use lambda function.

https://www.python-course.eu/lambda.php

Answer 7

This works:

def dReLU(x):
    return 1. * (x > 0)

Answer 8

As mentioned by Neil in the comments, you can use heaviside function of numpy.

def reluDerivative(self, x):
    return np.heaviside(x, 0)

Answer 9

If you want to use pure Python:

def relu_derivative(x):
    return max(sign(x), 0)

Answer 10

If you want it with the derivative you can use:

def relu(neta):
    relu = neta * (neta > 0)
    d_relu = (neta > 0)
    return relu, d_relu

Answer 11

When x is larger than 0, the slope is 1. When x is smaller than or equal to 0, the slope is 0.

if (x > 0):
    return 1
if (x <= 0):
    return 0

This can be written more compact:

return 1 * (x > 0)