How to make setuptools install a wheel containing multiple packages?

2024/11/24 7:00:18

Suppose this wheel:

M Filemode      Length  Date         Time      File
- ----------  --------  -----------  --------  --------------------------------------------rw-rw-r--      1358  26-Sep-2018  21:08:40  azure/common/__init__.py-rw-rw-r--       327  26-Sep-2018  21:08:40  azure/common/_version.py-rw-rw-r--      8737  26-Sep-2018  21:08:40  azure/common/client_factory.py-rw-rw-r--       755  26-Sep-2018  21:08:40  azure/common/cloud.py-rw-rw-r--      2479  26-Sep-2018  21:08:40  azure/common/credentials.py-rw-rw-r--       805  26-Sep-2018  21:08:40  azure/common/exceptions.py-rw-rw-r--      6079  26-Sep-2018  21:08:40  azure/profiles/__init__.py-rw-rw-r--      3943  26-Sep-2018  21:08:40  azure/profiles/multiapiclient.py-rw-rw-r--         6  26-Sep-2018  21:21:54  azure_common-1.1.16.dist-info/top_level.txt-rw-rw-r--       110  26-Sep-2018  21:21:54  azure_common-1.1.16.dist-info/WHEEL-rw-rw-r--      3805  26-Sep-2018  21:21:54  azure_common-1.1.16.dist-info/METADATA-rw-rw-r--       997  26-Sep-2018  21:21:54  azure_common-1.1.16.dist-info/RECORD
- ----------  --------  -----------  --------  -------------------------------------------29401                         12 files

It has three different packages in it:

  • azure.common
  • azure.profiles
  • azure_common

All great names, and great layout. Also, a lot of greatness of mind that unmistakably went into engineering this miracle of modern software engineering.

This wheel is distributed by the name azure-common. So, when you depend on in in setup.py like this:

setup(...install_requires=['azure-common'],...
)

You will only get azure_common package installed. Maybe. I don't really know, it seems so, but few times that I tried it seemed to only install azure.common, or maybe I eyeballed it... It's really hard to follow all the manipulations setuptools does on a package.

Hence the question: how can I force setuptools into installing all packages found in this kind of wheel? Also, the order is important because this garbage needs to be installed some of the times with other packages which also provide azure.something packages which may overwrite the stuff in azure directory. So, Ideally, I'd also like to control in which order install_requires dependencies are processed.


This is where this started: How to specify bracket dependencies in setup.py?

Answer

It sounds like only few sub-directories like azure.common installed into your environment when you installed dependencies via setup.py with install_requires=['azure-common']. I tried to reproduce this issue, but failed that all files in this package has been installed.

Here is my steps on my local Windows machine as below, which you can refer to.

  1. Create a directory mkdir setuptmp, and create a virtual environment virtualenv setuptmp, then to cd setuptmp.
  2. Create a setup.py file with the content as below.\

    from setuptools import setup, find_packages  setup(name = "setuptmp",install_requires = ['azure-common']
    )
    
  3. Activate the virtual environment via Scripts\activate.bat.

  4. Run python setup.py install to install the dependency described in my setup.py.
  5. Run python to open the REPL interpreter to test all packages as you said,

    (setuptmp) D:\projects\setuptmp>python
    Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import azure.common
    >>> import azure.profiles
    >>> azure.common.__file__
    'D:\\projects\\setuptmp\\lib\\site-packages\\azure_common-1.1.16-py3.7.egg\\azure\\common\\__init__.py'
    >>> azure.profiles.__file__
    'D:\\projects\\setuptmp\\lib\\site-packages\\azure_common-1.1.16-py3.7.egg\\azure\\profiles\\__init__.py'
    >>> import azure_common
    Traceback (most recent call last):File "<stdin>", line 1, in <module>
    ModuleNotFoundError: No module named 'azure_common'
    

Note: azure_common is not a module, just an egg info directory.

  1. Check the packages installed in my environment via cd Lib\site-packages, dir and tree azure_common-1.1.16-py3.7.egg /F as below.

    (setuptmp) D:\projects\setuptmp\Lib\site-packages>dirVolume in drive D is DataVolume Serial Number is BA4B-64AADirectory of D:\projects\setuptmp\Lib\site-packages2018/12/26  14:48    <DIR>          .
    2018/12/26  14:48    <DIR>          ..
    2018/12/26  14:48    <DIR>          azure_common-1.1.16-py3.7.egg
    2018/12/26  14:48                61 easy-install.pth
    2018/12/26  14:46               126 easy_install.py
    2018/12/26  14:46    <DIR>          pip
    2018/12/26  14:46    <DIR>          pip-18.1.dist-info
    2018/12/26  14:46    <DIR>          pkg_resources
    2018/12/26  14:48               965 setuptmp-0.0.0-py3.7.egg
    2018/12/26  14:46    <DIR>          setuptools
    2018/12/26  14:46    <DIR>          setuptools-40.6.3.dist-info
    2018/12/26  14:46    <DIR>          wheel
    2018/12/26  14:46    <DIR>          wheel-0.32.3.dist-info
    2018/12/26  14:46    <DIR>          __pycache__3 File(s)          1,152 bytes11 Dir(s)  80,896,319,488 bytes free(setuptmp) D:\projects\setuptmp\Lib\site-packages>tree azure_common-1.1.16-py3.7.egg /F
    Folder PATH listing for volume Data
    Volume serial number is BA4B-64AA
    D:\PROJECTS\SETUPTMP\LIB\SITE-PACKAGES\AZURE_COMMON-1.1.16-PY3.7.EGG
    ├─azure
    │  ├─common
    │  │  │  client_factory.py
    │  │  │  cloud.py
    │  │  │  credentials.py
    │  │  │  exceptions.py
    │  │  │  _version.py
    │  │  │  __init__.py
    │  │  │
    │  │  └─__pycache__
    │  │          _version.cpython-37.pyc
    │  │          __init__.cpython-37.pyc
    │  │
    │  └─profiles
    │          multiapiclient.py
    │          __init__.py
    │
    └─EGG-INFOPKG-INFORECORDrequires.txttop_level.txtWHEEL
    
  2. Compare the above with the file structure of azure-common package downloaded from the link of Pypi website. I decompressed azure_common-1.1.16-py2.py3-none-any.whl file using 7-Zip into a temp directory and tree it.

    D:\tmp>tree azure_common-1.1.16-py2.py3-none-any /F
    Folder PATH listing for volume Data
    Volume serial number is BA4B-64AA
    D:\tmp\AZURE_COMMON-1.1.16-PY2.PY3-NONE-ANY
    ├─azure
    │  ├─common
    │  │      client_factory.py
    │  │      cloud.py
    │  │      credentials.py
    │  │      exceptions.py
    │  │      _version.py
    │  │      __init__.py
    │  │
    │  └─profiles
    │          multiapiclient.py
    │          __init__.py
    │
    └─azure_common-1.1.16.dist-infoMETADATARECORDtop_level.txtWHEEL
    

Then, you will find the file structure of step 6 & 7 is almost same.

Hope it helps. If you have any concern, please feel free to let me know.


I did the same above on Linux and got the same result. I saved the output of tree lib/ > lib_[before|after].txt of my Linux setuptmp before and after run python setup.py install, then to compare them using diff lib_*.txt as below.

(setuptmp) peter@peterpc:~/setuptmp$ diff lib*.txt
92a93,111
>     │   ├── azure_common-1.1.16-py3.6.egg
>     │   │   ├── EGG-INFO
>     │   │   │   ├── PKG-INFO
>     │   │   │   ├── RECORD
>     │   │   │   ├── WHEEL
>     │   │   │   ├── requires.txt
>     │   │   │   └── top_level.txt
>     │   │   └── azure
>     │   │       ├── common
>     │   │       │   ├── __init__.py
>     │   │       │   ├── _version.py
>     │   │       │   ├── client_factory.py
>     │   │       │   ├── cloud.py
>     │   │       │   ├── credentials.py
>     │   │       │   └── exceptions.py
>     │   │       └── profiles
>     │   │           ├── __init__.py
>     │   │           └── multiapiclient.py
>     │   ├── easy-install.pth
827a847
>     │   ├── setuptmp-0.0.0-py3.6.egg
1043c1063
< 118 directories, 922 files
---
> 123 directories, 937 files
https://en.xdnf.cn/q/120566.html

Related Q&A

Python - Randomly select words to display in a quiz [duplicate]

This question already has answers here:How can I randomly select (choose) an item from a list (get a random element)?(18 answers)Closed 9 years ago.Im in need of some help. Here we have my two lists:w…

How to extract only the english words from the list?

I tried to extract only the English words from the following list: l = [0, b, x14, x00, x1fP, xe0O, xd0, xea, i, x10, xa2, xd8, x08, x00, 00, x9d, x14, x00, x80, xcc, xbf, xb4, xdbLB, xb0, x7f, xe9, x9…

Javascript variable with html code regex email matching

This python script is not working to output the email address [email protected] for this case.This was my previous post.How can I use BeautifulSoup or Slimit on a site to output the email address from …

how to send a large array over tcp socket in python? is it possible to send? [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.Want to improve this question? Add details and clarify the problem by editing this post.Closed 9 years ago.Improve…

Expand Python regex to list of all possible strings [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.Want to improve this question? Update the question so it focuses on one problem only by editing this post.Closed 5…

How can I group and sum a pandas dataframe? [duplicate]

This question already has answers here:How do I Pandas group-by to get sum?(11 answers)Closed 1 year ago.Ive had a good hunt for some time and cant find a solution so asking here. I have data like so:…

python add value to a list when iterate the list

values = [2,3,4] for v in values:values.append([v,255,255])Why do the statements above never end? I make a mistake in my code. However, I find it will never stop when I execute the code above.

resizing images from 64x64 to 224x224 for the VGG model

Can we resize an image from 64x64 to 256x256 without affecting the resolution is that a way to add zero on new row and column in the new resized output I m working on vgg and I get an error while addin…

Python slicing explained [duplicate]

This question already has answers here:How slicing in Python works(38 answers)Closed 6 years ago.OK I understand the basics, but can someone explain code copied from Gregs answer here:a[1::-1] # the …