I defined a python function which run a bash script. Let's say the function is: calc(x,y,z). If I run this function in python with some variables,

>>> calc(1,2,3)

It generates a C code which simulate something using the variables (x=1, y=2, z=3), compiles the C code and executes the compiled output file.

I want to run multiple calc(x,y,z)s with different (x,y,z)s in jupyter notebook simultaneously. As you may noticed, the problem is that cells in jupyter notebook are executed sequentially. If I run three calc functions, it takes three times longer than the one function running time.

I tried two ways but they didn't work well.

1. Use multiprocessing module: By using the module, it is possible to execute multiple calcs simultaneously in "one cell". But for later analysis, I would like to execute multiple cells simultaneously which include only one calc each using different processors (or cpu cores).

2. Use ipyparallel cell magic (inspired by this answer): I tried as following after import ipyparallel

   # Cell 1
   %%px --targets 0 # use processor 0
   calc(1,1,1)
   

   .

   # Cell 2
   %%px --targets 1 # use processor 1
   calc(2,2,2)        
   

   .

   # Cell 3
   %%px --targets 2 # use processor 2
   calc(3,3,3) 
   

But the cells are executed sequentially: Cell 2 is executed after Cell 1 simulation was done, similar for Cell 3.

How to run multiple jupyter cells using different cores?

In your solution, the cells are executed by different engines as you expected. The issue is caused by default blocking behavior. You can simply add --noblock argument to execute a cell in non-blocking mode. Then the cell returns AsyncResult object that makes it possible to read the output once the execution is finished by calling method display_outputs(). Go to the documentation for details targets-and-blocking

# Cell 1
%%px --targets 0 --noblock
calc(1,1,1)

.

# Cell 2
%%px --targets 1 --noblock
calc(2,2,2)   

.

# Cell 3
%%px --targets 2 --noblock
calc(3,3,3) 

If you need to access the output you can call display_outputs() as I explained above.

# output of the first the cell 1
___.display_outputs()# output of the first the cell 2
__.display_outputs()# output of the first the cell 1
_.display_outputs()

I used underscore notation to access AsyncResult objects returned by the cells 1-3. There is a number of other ways to access those objects, for example by using Out[x] where x is the cell's execution number visible in the notebook after executing a cell.