How can I use queues in python and find the subset of matching values of two functions?

Question

import time
import multiprocessing as mp

start = time.time()

def square(i):
    return i**2

def add(i):
    return i

def sigma_sum(start, stop, expression):
    return sum(expression(i) for i in range(start, stop))

def foo(q):
    for n in range(1000):
        q.put(list(sigma_sum(1, n, add)))

def bar(r):
    for k in range(1000):
        r.put(list(sigma_sum(1, k, square)))

if __name__ == '__main__':
    q = mp.Queue()
    r = mp.Queue()
    mp1 = mp.Process(target=foo, args = (q,))
    mp2 = mp.Process(target=bar, args = (r,))
    mp1.start()
    mp2.start()
    print(set(q.get()) & set(r.get()))

end = time.time()

print((start-end)*-1)

I adjusted this code to try to work with queues but so far I have had absolutely no success. I'm getting errors like:

>File "/Users/crxunch/temp2.py", line 17, in foo
   >q.put(list(sigma_sum(1, n, add)))
>TypeError: 'int' object is not iterable 

and

>File "/Users/crxunch/temp2.py", line 21, in bar
>   r.put(list(sigma_sum(1, k, square)))
>TypeError: 'int' object is not iterable 

If anyone could provide any help oh this I would really appreciate it.

Answer 1

right now

def sigma_sum(start, stop, expression):
    return sum(expression(i) for i in range(start, stop))

returns a single value (int/float), and you call it like so

q.put(list(sigma_sum(1, n, add)))

where you try to create a list from this single element, that s there the error comes from.

to solve the problem eather change the sigma_sum function to return a list or change the put lines so you don't try to make a list of a single element

Answer 2

You have multiple issues:

1. First and foremost, you have q.put(list(sigma_sum(1, n, add))) but the list constructor requires an iterable yet sigma_sum computes a single int value, which is not an iterable.
2. Your main process is expecting a single value in each queue, but foo and bar are each attempting to write 1000 values to their respective queues.
3. You should move the definitions of start and end to within the if __name__ == '__main__': block.
4. Your computation of the running time can be simplified (requires no multiplication):

import time
import multiprocessing as mp

def square(i):
    return i**2

def add(i):
    return i

def sigma_sum(start, stop, expression):
    return sum(expression(i) for i in range(start, stop))

def foo(q):
    q.put([sigma_sum(1, n, add) for n in range(1000)])

def bar(r):
    r.put([sigma_sum(1, k, add) for k in range(1000)])

if __name__ == '__main__':
    start = time.time()

    q = mp.Queue()
    r = mp.Queue()
    mp1 = mp.Process(target=foo, args=(q,))
    mp2 = mp.Process(target=bar, args=(r,))
    mp1.start()
    mp2.start()
    print(set(q.get()) & set(r.get()))
    mp1.join()
    mp2.join()

    end = time.time()

    print(end - start)

Prints:

{0, 1, 391170, 3, 468996, 378885, 6, 28680, 10, 15, 491536, 143380, 21, 20503, 323610, 28, 14365, 362526, 2080, 348195, 36, 163878, 98346, 309291, 118828, 45, 430128, 22578, 53301, 55, 10296, 63546, 43071, 8256, 66, 452676, 440391, 6216, 258121, 397386, 92235, 180300, 78, 75855, 372816, 41041, 489555, 208981, 16471, 4186, 91, 110685, 45150, 18528, 2145, 278631, 105, 198765, 319600, 233586, 12403, 231540, 467061, 69751, 120, 235641, 145530, 352380, 229503, 32896, 411778, 292995, 153735, 136, 237705, 299151, 270480, 419986, 227475, 39060, 116886, 159895, 153, 487578, 30876, 51360, 239778, 2211, 34980, 65703, 171, 26796, 47278, 225456, 24753, 4278, 286903, 6328, 260281, 94395, 190653, 190, 73920, 8385, 403651, 241860, 207046, 14535, 10440, 366796, 104653, 184528, 210, 223446, 450775, 305371, 315615, 100576, 20706, 485605, 2278, 231, 465130, 356590, 243951, 147696, 428275, 438516, 28920, 170236, 253, 131328, 221445, 129286, 22791, 37128, 174345, 133386, 108811, 16653, 196878, 114960, 12561, 383250, 4371, 276, 389403, 127260, 57630, 84255, 166176, 18721, 246051, 135460, 59685, 82215, 6441, 2346, 280875, 300, 49455, 86320, 262450, 272691, 483636, 67896, 155961, 377146, 55611, 219453, 205120, 125250, 8515, 96580, 325, 178503, 80200, 334153, 72010, 338253, 137550, 61776, 395605, 10585, 88410, 330078, 351, 248160, 463203, 43365, 311655, 342378, 418155, 409965, 448878, 2415, 41328, 4465, 14706, 162165, 149878, 123256, 360825, 378, 53628, 217470, 295296, 33153, 78210, 188805, 481671, 139656, 45451, 326028, 24976, 371091, 27028, 31125, 406, 113050, 6555, 289180, 90525, 63903, 346528, 301476, 436645, 250278, 39340, 35245, 20910, 102831, 12720, 435, 264628, 2485, 182710, 195000, 426426, 121278, 401856, 203203, 16836, 8646, 215496, 106953, 4560, 465, 141778, 322003, 76245, 23005, 479710, 274911, 461280, 51681, 47586, 18915, 98790, 29161, 10731, 70125, 350703, 496, 252405, 283128, 92665, 158203, 2556, 307720, 446985, 152076, 6670, 528, 66066, 119316, 37401, 213531, 365085, 14878, 172578, 168490, 143916, 4656, 561, 318003, 111156, 387640, 477753, 381501, 266815, 74305, 2628, 416328, 8778, 176715, 254541, 201295, 12880, 595, 186966, 354903, 434778, 408156, 459361, 164451, 393828, 49770, 193131, 94830, 25200, 57970, 375411, 630, 211575, 117370, 21115, 17020, 27261, 10878, 60031, 6786, 33411, 424581, 297606, 55945, 291466, 43660, 2701, 31375, 41616, 4753, 277140, 146070, 105111, 475800, 666, 101025, 180901, 19110, 445096, 68265, 314028, 256686, 62128, 154290, 23220, 35511, 45753, 84666, 82621, 703, 400065, 303810, 53956, 39621, 72390, 15051, 160461, 285390, 8911, 86736, 269011, 2775, 359128, 80601, 369370, 29403, 209628, 109278, 336610, 199396, 741, 457446, 332520, 115440, 13041, 4851, 340725, 6903, 473851, 97020, 88831, 131841, 129795, 64261, 328455, 780, 78606, 133903, 148240, 11026, 432915, 127765, 47895, 258840, 232221, 234270, 344865, 2850, 52003, 230181, 191271, 236328, 414505, 37675, 135981, 185136, 820, 17205, 228150, 125751, 238395, 310078, 324415, 170820, 498501, 90951, 21321, 207690, 443211, 406351, 226128, 25425, 279378, 422740, 9045, 4950, 240471, 174936, 385881, 138075, 76636, 861, 166753, 471906, 70500, 349030, 27495, 156520, 123753, 19306, 379756, 7021, 2926, 455535, 271216, 363378, 224115, 103285, 113526, 392055, 15225, 496506, 242556, 66430, 293761, 197506, 33670, 903, 31626, 261003, 23436, 320400, 13203, 299925, 107416, 140185, 150426, 179101, 222111, 99235, 50086, 11175, 93096, 244650, 121771, 287661, 162735, 373680, 41905, 946, 494515, 43956, 5050, 3003, 205761, 35778, 74691, 353220, 398278, 58311, 469965, 29646, 431056, 220116, 56280, 60378, 9180, 990, 39903, 246753, 7140, 142311, 46056, 306153, 189420, 17391, 492528, 441330, 316410, 453628, 119805, 263175, 3081, 1035, 111628, 54285, 412686, 62481, 218130, 183315, 273430, 21528, 281625, 68635, 5151, 248865, 95266, 367653, 420903, 15400, 195625, 158766, 19503, 490545, 25651, 152628, 13366, 1081, 357435, 468028, 11325, 37950, 203841, 72771, 144453, 404550, 48205, 27730, 83028, 85078, 3160, 216153, 7260, 117855, 105570, 101475, 9316, 23653, 52326, 1128, 250986, 81003, 64620, 173166, 169071, 87153, 488566, 334971, 312445, 384126, 296065, 339076, 5253, 31878, 33930, 330891, 265356, 451725, 390286, 429201, 289941, 1176, 79003, 439453, 177310, 378015, 165025, 89253, 343206, 3240, 214185, 17578, 302253, 466095, 109746, 146611, 326836, 97461, 187578, 253116, 486591, 29890, 275653, 1225, 201930, 361675, 36046, 396495, 115921, 42195, 11476, 7381, 15576, 193753, 13530, 70876, 44253, 154846, 347361, 50403, 77028, 21736, 283881, 66795, 5356, 9453, 371953, 91378, 19701, 322806, 410871, 3321, 40186, 1275, 419070, 181503, 130305, 212226, 132355, 161028, 484620, 128271, 134421, 25878, 255255, 46360, 308505, 267546, 58653, 449826, 464166, 56616, 126253, 1326, 148785, 351541, 60726, 136503, 103740, 27966, 23871, 402753, 75078, 3403, 437580, 7503, 318801, 38226, 114003, 5460, 427350, 54615, 93528, 124251, 200028, 482653, 99681, 1378, 210276, 17766, 107880, 138601, 11628, 62835, 365940, 9591, 257403, 277885, 13695, 32131, 48516, 292230, 15753, 298378, 171405, 69006, 34191, 185745, 191890, 1431, 157080, 122265, 3486, 462241, 355746, 167331, 382375, 175528, 388521, 52650, 140715, 269745, 480690, 30135, 21945, 447931, 19900, 5565, 150975, 73153, 286146, 314821, 232903, 7626, 234955, 230860, 1485, 304590, 208335, 64980, 376278, 95703, 237016, 417241, 228826, 36315, 394716, 409060, 112101, 120295, 259560, 163306, 83436, 239086, 226801, 3570, 85491, 179700, 42486, 198135, 26106, 142845, 81406, 435711, 9730, 1540, 11781, 44551, 478731, 241165, 224785, 87571, 40470, 337431, 333336, 24090, 425503, 460320, 50721, 17955, 13861, 101926, 5671, 359976, 79401, 28203, 106030, 341551, 329266, 243253, 370230, 222778, 15931, 1596, 280126, 400960, 206403, 118341, 7750, 3655, 46665, 89676, 271953, 310866, 190036, 71253, 446040, 67161, 153181, 261726, 144991, 345696, 159330, 325221, 245350, 38503, 476776, 220780, 77421, 97903, 183921, 58996, 1653, 56953, 294528, 32385, 20100, 110215, 22155, 9870, 5778, 61075, 34453, 196251, 300700, 3741, 91806, 11935, 247456, 54946, 458403, 288420, 218791, 349866, 30381, 1711, 321201, 116403, 169653, 433846, 415416, 173755, 48828, 204480, 7875, 364231, 386760, 474825, 75466, 14028, 147153, 380628, 407253, 63190, 263901, 26335, 165600, 18145, 249571, 36585, 1770, 216811, 423660, 392941, 16110, 52975, 177906, 3828, 24310, 306936, 444153, 274170, 5886, 130816, 69378, 104196, 132870, 282376, 93961, 128778, 155403, 354061, 374545, 317206, 28441, 42778, 10011, 134940, 188191, 100128, 126756, 1830, 456490, 472878, 251695, 114481, 44850, 40755, 214840, 108345, 12090, 161596, 65341, 73536, 8001, 137026, 399171, 202566, 194376, 3916, 20301, 124750, 149331, 182106, 22366, 497503, 51040, 1891, 266085, 5995, 431985, 14196, 139128, 46971, 38781, 368511, 32640, 296835, 253828, 83845, 122760, 358281, 96141, 212878, 290703, 85905, 81810, 313236, 495510, 470935, 413595, 34716, 442270, 18336, 1953, 16290, 30628, 4005, 10153, 276396, 335790, 454581, 87990, 303031, 79800, 331705, 339900, 421821, 141246, 112575, 8128, 26565, 67528, 157641, 405450, 57291, 59340, 71631, 493521, 120786, 24531, 200661, 12246, 171991, 6105, 284635, 327645, 2016, 255970, 344035, 151525, 167910, 102378, 385003, 210925, 55278, 61425, 186355, 90100, 49141, 268278, 77815, 36856, 176121, 106491, 192510, 4095}
0.17399358749389648