'Optimized way to implement XOR between floats for huge 2D array data

I need to implement xor between floats in python for huge 2D array data (like thousand row by thousand column matrix). I use the following implementation:

import struct
def fxor(a, b):
  rtrn = []
  a = struct.pack('d', a)
  b = struct.pack('d', b)
  for ba, bb in zip(a, b):
    rtrn.append(ba ^ bb)
  return struct.unpack('d', bytes(rtrn))[0]
print(fxor(5.34, 5.34))               #0.0
print(fxor(10.23, 5.34))              #9.54764402360672e-308
print(fxor(10.23,fxor(10.23, 5.34)))  #5.34

The way I use fxor:


# for demo purpose I took 3 by 2 matrix
mat1 = np.random.random_sample((3, 2))
mat2 = np.random.random_sample((3, 2))
resultant = []
for i in range(3):
    row = []
    for j in range(2):
        row.append(fxor(mat1[i][j],mat2[i][j]))
    resultant.append(row)
resultant

Which work perfectly in my case. But when I check the time profile it seems the implementation is very slow for large array (60% of total time).

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
250000    1.438    0.000    1.926    0.000 2837056651.py:3(fxor)
.
.
.
500000    0.124    0.000    0.124    0.000 {built-in method _struct.pack}
250000    0.067    0.000    0.067    0.000 {built-in method _struct.unpack}

Is there any optimized way to do this like np.bitwise_xor does for int value?

Update

@jasonharper suggest me to use .view(np.int64) which work nice:

mat1 = np.random.random_sample((3, 2))
mat2 = np.random.random_sample((3, 2))
print(mat1)
mat3 = np.bitwise_xor(mat1.view(np.int64),mat2.view(np.int64))
print(np.bitwise_xor(mat2.view(np.int64),mat3).view(np.float64))
# output
#[[0.71297944 0.33048679]
# [0.82762999 0.26549565]
# [0.94499741 0.2570297 ]]
#[[0.71297944 0.33048679]
# [0.82762999 0.26549565]
# [0.94499741 0.2570297 ]]

But the issue is, sometimes it gives the following error:

ValueError: When changing to a larger dtype, its size must be a divisor of the total size in bytes of the last axis of the array.

How to handle this error?

Update 2

Everything works nice until the array size cross >10000. Because then I get two error for different different execution. This

ValueError: operands could not be broadcast together with shapes (10000,1250) (10000,10000) 

and this.

ValueError: When changing to a larger dtype, its size must be a divisor of the total size in bytes of the last axis of the array.

I can assure you that the dimension of those matrices are same because they are passed through

assert first_mat.shape == second_mat.shape

I was unable to predict the reason because sometimes the program run without any issue and sometimes it raise them for that huge 2D array. If you want to know how I generate those array then here is my another question where I showed how I generate those matrices.

The problem mostly depend on the numpy view

---> 46     return np.bitwise_xor(Matrix.view(np.int64),transformationMatrix.view(np.int64)).view(np.float64)

update 3

@JérômeRichard suggested to check shape of .view() for both matrices. I was surprise that my mat1 was int valued matrix which create the issue. I update that to return always float valued matrix and things are working nice until I got nan value for some cases.

a = np.array([[4.27666612,4.61512052],[0.19573934,0.82816473]])
b = np.array([[0.97597378,0.09191992],[0.32720493,0.86295611]])
np.bitwise_xor(a.view(np.uint8),b.view(np.uint8)).view(np.float64)
# gives
#array([[            nan, 7.72164724e+306],
#       [4.17041859e-308, 1.54832353e-309]])

Which is not feasible for my problem. I was surprise why nan was return as a result of xor kind operation. How to handle this infeasibility?

update 4

I still find the np.bitwise_xor problematic with narray.view(np.uint8) mode. because it gives the overflow value every times.

# overflow values are
np.finfo(np.double).min, np.finfo(np.double).max
# -1.79769313486e+308, 1.79769313486e+308

Even, its become hard to work with the resultant data. Is there no efficient solution at all?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source