unable to add spark to PYTHONPATH

2024/9/8 10:26:58

I am struggling to add spark to my python path:

(myenv)me@me /home/me$ set SPARK_HOME="/home/me/spark-1.2.1-bin-hadoop2.4"
(myenv)me@me /home/me$ set PYTHONPATH=$PYTHONPATH:$SPARK_HOME:$SPARK_HOME/python:$SPARK_HOME/python/build:$SPARK_HOME/bin(myenv)me@me /home/me$ python -c 'import sys; print(sys.path)'
['', '/home/me/.virtualenvs/default/lib/python2.7', '/home/me/.virtualenvs/default/lib/python2.7/plat-x86_64-linux-gnu', '/home/me/.virtualenvs/default/lib/python2.7/lib-tk', '/home/me/.virtualenvs/default/lib/python2.7/lib-old', '/home/me/.virtualenvs/default/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/home/me/.virtualenvs/default/local/lib/python2.7/site-packages', '/home/me/.virtualenvs/default/lib/python2.7/site-packages'](myenv)me@me /home/me$ python -c 'import pyspark'
Traceback (most recent call last):File "<string>", line 1, in <module>
ImportError: No module named pyspark
Answer

I've got the same problem, but this helped.

Just add the following command in your .bashrc

export SPARK_HOME=/path/to/your/spark-1.4.1-bin-hadoop2.6
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/build:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH
https://en.xdnf.cn/q/72585.html

Related Q&A

Is there a way to shallow copy an existing file-object?

The use case for this would be creating multiple generators based on some file-object without any of them trampling each others read state. Originally I (thought I) had a working implementation using s…

Python - Import package modules before as well as after setup.py install

Assume a Python package (e.g., MyPackage) that consists of several modules (e.g., MyModule1.py and MyModule2.py) and a set of unittests (e.g., in MyPackage_test.py).. ├── MyPackage │ ├── __ini…

Where can I find the _sre.py python built-in module? [closed]

Closed. This question is seeking recommendations for books, tools, software libraries, and more. It does not meet Stack Overflow guidelines. It is not currently accepting answers.We don’t allow questi…

When numba is effective?

I know numba creates some overheads and in some situations (non-intensive computation) it become slower that pure python. But what I dont know is where to draw the line. Is it possible to use order of …

How to launch a Windows shortcut using Python

I want to launch a shortcut named blender.ink located at "D://games//blender.ink". I have tryed using:-os.startfile ("D://games//blender.ink")But it failed, it only launches exe fil…

Providing context in TriggerDagRunOperator

I have a dag that has been triggered by another dag. I have passed through to this dag some configuration variables via the DagRunOrder().payload dictionary in the same way the official example has don…

Install gstreamer support for opencv python package

I have built my own opencv python package from source. import cv2 print(cv2.__version__)prints: 3.4.5Now the issue I am facing is regarding the use of gstreamer from the VideoCapture class of opencv. I…

How to use query function with bool in python pandas?

Im trying to do something like df.query("column == a").count()but withdf.query("column == False").count()What is the right way of using query with a bool column?

Text Extraction from image after detecting text region with contours

I want to build an OCR for an image using machine learning in python. I have preprocessed image by converting it to grayscale , applied otsu thresholding . I then used the contours to find the text re…

Pyinstaller executable fails importing torchvision

This is my main.py:import torchvision input("Press key")It runs correctly in the command line: python main.pyI need an executable for windows. So I did : pyinstaller main.pyBut when I launche…