Data Science Special Topics: Parallelization with Python

Most Python scripts use one core of a computer that has eight or more. This workshop introduces how to use the rest. It is designed for anyone who already writes Python and works with tasks that repeat — processing many files, running simulations, fitting models across parameter settings, or applying the same operation to a large dataset. We'll start with the landscape: the types of parallelization and what each is suited to, so you can recognize where your own work fits. From there we focus on the most common case, an "embarrassingly parallel" task that splits cleanly into independent pieces, and demonstrate how to spread one across cores using Python's multiprocessing module. We'll time the same task across different numbers of cores to see where the gains level off, and look at a case where running in parallel turns out slower than not. This session focuses on using multiple cores for computation; threading and asynchronous programming are not covered. This workshop incorporates hands-on work, and there will be plenty of time for questions. Some prior experience with Python, including writing your own functions and loops, is expected.

No software installation is needed; participants will be instructed how to use Google Colab to write, store, and share code in the cloud.