Run parts of a ipython notebook in a loop / with different input parameter

Question

I have written a ipython notebook, which analyses a dataset. Now I want to use this code to loop over different datasets.

The code is split into about 50 cells (including comments, markdown explanations,...). Is there a way to run parts of a notebook in a loop or running a whole notebook with different input parameters?

I don't want to merge all cells into one function or download the code as a python script, as I really like to run (and experimenting with) parts of the analysis by executing only certain cells.

Basically its refactoring parts of a script into a function and calling the function in a loop, just that the "parts of the script" are notebook cells.

Answer 1

What I usually do in these scenarios is wrap the important cells as functions (you don't have to merge any of them) and have a certain master cell that iterates over a list of parameters and calls these functions. E.g. this is what a "master cell" looks like in one of my notebooks:

import itertools
# parameters
P_peak_all = [100, 200]
idle_ratio_all = [0., 0.3, 0.6]
# iterate through these parameters and call the notebook's logic
for P_peak, idle_ratio in itertools.product(P_peak_all, idle_ratio_all):
    print(P_peak, idle_ratio, P_peak*idle_ratio)
    print('========================')
    m_synth, m_synth_ns = build_synth_measurement(P_peak, idle_ratio)
    compare_measurements(m_synth, m_synth_ns, "Peak pauser", "No scheduler", file_note="-%d-%d" % (P_peak, int(idle_ratio*100)))

You can still have some data dragging throughout the notebook (i.e. calling each function at the bottom of the cell with your data) to be able to test stuff live for individual cells. For example some cell might state:

def square(x):
    y = x**2
    return y
square(x) # where x is your data running from the prior cells 

Which lets you experiment live and still call the generic functionality from the master cell.

I know it's some additional work to refactor your notebook using functions, but I found it actually increases my notebook's readability which is useful when you come back to it after a longer period and it's easier to convert it to a "proper" script or module if necessary.

Answer 2

A cheap, but fast trick is to use "run all cells" in a kind of while loop:

Ipython/Jupyter - can we program a "run all cell above"?

change_parameters

your code

put this in your last cell:

display(Javascript('IPython.notebook.execute_all_cells()'))

Answer 3

papermill.execute_notebook was designed (by Netflix) to let us run notebooks as a function (which aids in code reuse, experiments reproducibility and memory efficiency - it runs notebooks in batch mode unlike standard python REPL):

import papermill as pm

pm.execute_notebook(
   'path/to/input.ipynb',
   'path/to/output.ipynb',
   parameters=dict(alpha=0.6, ratio=0.1)
)

More info: docs | example.

Answer 4

You can use this:

    from IPython.display import Javascript
    Javascript("Jupyter.notebook.execute_cell_range(5,51)") # 51 is not 

Note: Do not use the indices shown beside the cells because Jupyter.notebook.execute_cell_range count every single cell including heading and markdown cells. Try to count manually the range of your cells including the markdown cells to give Jupyter.notebook.execute_cell_range the correct range.