sys.path and sys.executable is incorrect in jupyter, but no applied fix is working

2024/4/15 0:53:54

I've configured jupyter to be used from a remote computer and set a password to it while initial anaconda setup. Then after fixing this issue, I am trapped in another one. sys.path and sys.executable is incorrect in jupyter, but correct in python and ipython. Please see the details below. Anaconda3 is installed for all users in /opt/anaconda3 and I am using an environment zud for my programs.

Background

Once the above-mentioned problem is fixed, I tried to import igraph in jupyter but it gave ModuleNotFoundError as below:

import igraph
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-73ffca960e68> in <module>
----> 1 import igraphModuleNotFoundError: No module named 'igraph'

Then quickly I checked the following details using my environment zud, which were correct.

(zud) zuddler@silp252-dl360:~$ which python
/home/zuddler/.conda/envs/zud/bin/python
(zud) zuddler@silp252-dl360:~$ which ipython
/home/zuddler/.conda/envs/zud/bin/ipython
(zud) zuddler@silp252-dl360:~$ which jupyter
/home/zuddler/.conda/envs/zud/bin/jupyter
(zud) zuddler@silp252-dl360:~$ which conda
/opt/anaconda3/bin/conda(zud) zuddler@silp252-dl360:~$ conda env list
# conda environments:
#
zud                   *  /home/zuddler/.conda/envs/zud
base                     /opt/anaconda3

Then I checked sys.path and sys.executable in python, ipython, and jupyter notebook/lab:

In Python:

>>> import sys
>>> sys.path
['', '/home/zuddler/.conda/envs/zud/lib/python39.zip', '/home/zuddler/.conda/envs/zud/lib/python3.9', '/home/zuddler/.conda/envs/zud/lib/python3.9/lib-dynload', '/home/zuddler/.conda/envs/zud/lib/python3.9/site-packages']
>>> sys.executable
'/home/zuddler/.conda/envs/zud/bin/python'

In ipython:

**In [1]: import sysIn [2]: sys.path
Out[2]:
['/home/zuddler/.conda/envs/zud/bin','/home/zuddler/.conda/envs/zud/lib/python39.zip','/home/zuddler/.conda/envs/zud/lib/python3.9','/home/zuddler/.conda/envs/zud/lib/python3.9/lib-dynload','','/home/zuddler/.conda/envs/zud/lib/python3.9/site-packages','/home/zuddler/.conda/envs/zud/lib/python3.9/site-packages/IPython/extensions','/home/zuddler/.ipython']In [3]: sys.executable
Out[3]: '/home/zuddler/.conda/envs/zud/bin/python'**

In Jupyter Notebook/Lab:

[1]: import sys
[2]: sys.path
[2]:
['/home/zuddler/python scripts','/opt/anaconda3/lib/python38.zip','/opt/anaconda3/lib/python3.8','/opt/anaconda3/lib/python3.8/lib-dynload','','/opt/anaconda3/lib/python3.8/site-packages','/opt/anaconda3/lib/python3.8/site-packages/IPython/extensions','/home/zuddler/.ipython']
[3]: sys.executable
[3]: '/opt/anaconda3/bin/python'

sys.path and sys.executable were different in Jupyter Notebook/Lab but were correct in Python and ipython.

Also, to confirm if the current jupyter is running, I ran !which jupyter in the jupyter notebook and received /home/zuddler/.conda/envs/zud/bin/jupyter which confirmed that the correct jupyter notebook was running.

Tried Remedies

Remedy 1: I tried the fix mentioned here and here and added the following line in the /home/zuddler/.ipython/profile_default/ipython_config.py file

c.InteractiveShellApp.exec_lines = ['import sys; sys.path.append("/home/zuddler/.conda/envs/zud/lib/python3.9/site-packages")'
]

then restarted the jupyter notebook and tried to import igraph again and got a slightly different ModuleNotFoundError

import igraph
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-73ffca960e68> in <module>
----> 1 import igraph~/.conda/envs/zud/lib/python3.9/site-packages/igraph/__init__.py in <module>32 # pylint: disable-msg=W040133 # W0401: wildcard import
---> 34 from igraph._igraph import *35 from igraph.clustering import *36 from igraph.cut import *ModuleNotFoundError: No module named 'igraph._igraph'

I quickly checked sys.path and sys.executable:

[1]: sys.path
[1]:
['/home/zuddler/python scripts','/opt/anaconda3/lib/python38.zip','/opt/anaconda3/lib/python3.8','/opt/anaconda3/lib/python3.8/lib-dynload','','/opt/anaconda3/lib/python3.8/site-packages','/opt/anaconda3/lib/python3.8/site-packages/IPython/extensions','/home/zuddler/.ipython','/home/zuddler/.conda/envs/zud/lib/python3.9/site-packages']
[2]: sys.executable
[2]: '/opt/anaconda3/bin/python'

It seems like a dead-end, so I rolled back this fix.

Remedy 2: I tried the fix given on this post and replaced the correct executable path in kernel.json

Kernels listed are:

(zud) zuddler@silp252-dl360:~$ jupyter kernelspec list
Available kernels:python3    /opt/anaconda3/share/jupyter/kernels/python3

I made the following changes in /opt/anaconda3/share/jupyter/kernels/python3/kernel.json

{
"argv": [
"/opt/anaconda3/bin/python", "/home/zuddler/.conda/envs/zud/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3",
"language": "python"
}

after that, I restarted the jupyter notebook/lab and it worked for the environment zud, but I noticed that for the base environment it imported igraph which was not even installed in base. I checked the following values from them, which jupyter was correct for base

[1]: !which jupyter
[1]: /opt/anaconda3/bin/jupyter

but it was picking up sys.path and sys.executable from zud, and that means everone will be compelled to use zud environment.

[2]: sys.path
[2]:
['/home/zuddler/python scripts','/home/zuddler/.conda/envs/zud/lib/python39.zip','/home/zuddler/.conda/envs/zud/lib/python3.9','/home/zuddler/.conda/envs/zud/lib/python3.9/lib-dynload','','/home/zuddler/.conda/envs/zud/lib/python3.9/site-packages','/home/zuddler/.conda/envs/zud/lib/python3.9/site-packages/IPython/extensions','/home/zuddler/.ipython']
[3]: sys.executable
[3]: '/home/zuddler/.conda/envs/zud/bin/python'

this was again a dead end, so I rolled back this fix.

Remedy 3: Applying the following steps from this post also did not solve the issue and gave the wrong sys.path and sys.executable

  1. Deleting IPython and Jupyter from the system
  2. rm -rf ~/.local/share/ipython
  3. Reinstall IPython and Jupyter

Remedy 4: I tried opening ~/.conda/envs/zud/bin/jupyter lab --no-browser as suggested in this post, but it too gave wrong sys.path and sys.executable.

Other Remedies: I read this, but found no solution.

PS: I noticed that this post is also mentioning a related issue as mine, but got no solution.

I'll be doing more research on this in the hope to reach a solution, I'll be grateful if you could provide a solution to it.

Answer

I've had this issue for quite a while as well. Here's how I've solved it.

After you've created and entered the virtual environment, you probably did something like

pip install ipykernel

ipython kernel install --user --name=myenv in order to make it visible in jupyter lab/notebook.

There's this ipykernel mapping file at /home/user/.local/share/jupyter/kernels/myenv/kernel.json

It should look something like this:

{"argv": ["/usr/bin/python3","-m","ipykernel_launcher","-f","{connection_file}"],"language": "python","display_name": "myenv"
}

The problem here is that the virtual env python is mapped to system python for some reason. You want to change the python pointer to point to your virtual env python, which would be similar to /home/user/.virtualenvs/myenv/bin/python3.

TL;DR:

cat /home/user/.local/share/jupyter/kernels/myenv/kernel.json

if it has /usr/bin/python3 in it, change it to /home/user/.virtualenvs/myenv/bin/python3

You might need to restart your Jupyter Lab/Notebook for changes to take effect.

I don't fully understand why it sometimes happens and sometimes doesn't, but hopefully this fix will help you too.

https://en.xdnf.cn/q/73292.html

Related Q&A

How to multicolour text with ScrolledText widget?

from tkinter import * from tkinter.scrolledtext import ScrolledTextwindow= Tk() window.geometry(970x45) box = ScrolledText(window, width=70, height=7).pack() box.insert(END, "Ehila") #this in…

Binding Return to button is not working as expected

I bound the event <Return> to a Button, thinking that this would cause the command to be run after hitting Enter:Button(self.f, text="Print", command=self.Printer).pack(side=RIGHT, padx…

ZMQ Pub-Sub Program Failure When Losing Network Connectivity

I have a simple pub-sub setup on a mid-sized network, using ZMQ 2.1. Although some subscribers are using C# bindings, others are using Python bindings, and the issue Im having is the same for either.I…

replacing html tags with BeautifulSoup

Im currently reformatting some HTML pages with BeautifulSoup, and I ran into bit of a problem.My problem is that the original HTML has things like this:<li><p>stff</p></li>and &…

LightGBM: train() vs update() vs refit()

Im implementing LightGBM (Python) into a continuous learning pipeline. My goal is to train an initial model and update the model (e.g. every day) with newly available data. Most examples load an alread…

GTK: create a colored regular button

How do I do it? A lot of sites say I can just call .modify_bg() on the button, but that doesnt do anything. Im able to add an EventBox to the button, and add a label to that, and then change its color…

How to Normalize similarity measures from Wordnet

I am trying to calculate semantic similarity between two words. I am using Wordnet-based similarity measures i.e Resnik measure(RES), Lin measure(LIN), Jiang and Conrath measure(JNC) and Banerjee and P…

How to open chrome developer console using Selenium in Python?

I am trying to open developer console in chrome using selenium webdriver. I am doingfrom selenium import webdriverfrom selenium.webdriver.common import action_chains, keys...browser = webdriver.Chrome(…

How to enable an allow-insecure-localhost flag in Chrome from selenium?

I want to enable "allow-insecure-localhost" flag from selenium. How I can do it?selenium: 3.12.0, Python:3.6.5Chrome driver creation code:def create_driver():options = Options()if sys.plat…

Getting pandas dataframe from list of nested dictionaries

I am new to Python so this may be pretty straightforward, but I have not been able to find a good answer for my problem after looking for a while. I am trying to create a Pandas dataframe from a list o…