Problem about cpu time and wall time in jupyternotebook

Question

As a physics student, I have a questions I would like to ask you about program runtime.

I'm doing a small project about algorithm optimization, my toy code is written on jupyternotebook, but when I use %%time and %%timeit in cell to test cpu time and wall time of my code, I found the following results

CPU times: total: 2min 16s Wall time: 24.2s

The huge gap between these two times makes me wonder, my confusion is

1. Which should I use as the standard to measure the speed of the algorithm, although I know that it is not reasonable to use python to measure the speed of the code
2. Why the time gap between these looks so big?

Really looking forward to any insights from kind people on this issue

Answer 1

First, what is Wall time : it's the total time needed to run the cell. Secondly, the CPU time is the time spend by the CPU, counting the different cores.

For example, if I have a cell that require a wall time of 1 second. If this cell uses during the entire time 2 cores, we will get a CPU time of 2s.

For your second question, what your results seems to indicate is that you are using along your calculus an average of 5.6 cores. And you are probably using some modules that uses numba to parallelize as it's the most commonly found in physics.

Coming back to your first question, I would advice your to run your cell with the profiler using %prun this way you will know where your program spend the more time and which parts needs to be optimize.

Answer 2

1 - CPU times can be subdivided into several times, if you don't know exactly how CPU time was subdivided you should use wall time aka Elapsed real-time.

2 - They can be many reasons for the huge gap: CPU time contains: Idle time, waiting time for systems resources to be available, etc... whereas wall time if only : finish date - starting date