I examined the four ways of computing in Python. Here’s what I found.

Photo by David Clode on Unsplash

Since Python by itself is slow, it becomes important to know the nitty-gritty of different components of our code for efficiency. In this post, we will look into the most common ways we loop in Python using a simple summing example. We will also compute the memory profile to inspect which way is the most memory-efficient for analyzing huge datasets.

The while loop

This returns while_loop: 0.727578s. We can also do the memory profiling of this function.

This returns:

In total, the while loop took 0.0743Mb of the memory usage for the above task.

The for loop

This returns for_loop: 0.490051s. Now, we do the memory profiling of this function.

In total, the for loop took 0.0351Mb of the memory usage for the above task.

The builtin python function

This returns builtinsum: 0.175238s.

In total, the "builtin function" based function took 0.0039Mb of the memory usage for the above task.

The numpy function

This returns numpysum: 0.017640s.

In total, the numpy based function took 7.6407Mb of the memory usage for the above task.

Conclusions

Please note that the above-obtained values of run time and memory usage may differ from system to system but the ratio of these values between different methods will stay very similar.

We found that the numpy is the fastest (0.017640s) and while loop sum is the slowest (0.727578s). The reason for the while loop to be slow is that we complete each step of the task in the native Python. Since numpy is written in C, it runs quite fast.

In terms of memory usage, the numpy is the worst. It took ~7Mb of the memory usage. In contrast, the usage of the “built-in” python function is the most memory-efficient as it does not store all the data into memory but does it in steps.

If we compare the while and for loop, then for loop is fast and also more memory efficient. Hence, for loop should always be our first choice (and usually is) unless we don't know the total number of runs.

References

  1. The fastest way to loop in Python - An Unfortunate truth
  2. [The right way to loop in Python(https://www.earthinversion.com/techniques/right-way-loop-in-python/)

Geophysics | Python | MATLAB | Productivity