What does conda env do under the hood?

2024/11/15 23:47:17

After searching and not finding, I must ask here:

How does conda env work under the hood, meaning, how does anaconda handle environments?

To clarify, I would like an answer or a reference to questions like:

  • What is kept in the envs/myenv folder?

  • What happens upon activate myenv?

  • What happens upon conda install ...?

  • Where can i find such information?

Answer

Conda envs

Basically, conda environments replicate the structure of your system, meaning it will store /bin, /lib, /etc, /var, among other directories. This is more obvious for unix systems, but the same concept is true under windows (DLLs, libs, Scripts, ...). More details in the official documentation.

Conda install

The idea is that conda install PACKAGE will fetch a precompiled package from a channel (a conda packages repository), and install it under this system-like structure. Instead of relying on system dependencies, conda will install all dependencies of this package under the environment structure, using only conda packages. Thus installing the same package at a given time point under different systems should result in reliably identical installs.

This is a way to standardize binaries, and it is only achieved by precompiling every package against given versions of libraries, which are shipped as dependencies of the conda environment. For instance, conda-forge and bioconda channels rely on cloud-based CI/CD pipelines to compile all packages on identical and completely clean system images.

Conda also stores metadata about these packages (version, build number, dependencies, license,...) so it is able to solve pretty complex dependency trees and avoid packages/libraries incompatibilities. It is the Solving... step each time you execute conda install.

Conda activate

Then when you conda activate ENV, conda prepends the environment root $CONDA_PREFIX/bin to PATH, so that all executables installed in the environment will be found by the system (and will overload system-wide install of the same executable). You can imagine it like temporarily replacing the system executables with those from the environment.

More

This a very basic explanation, not 100% accurate, and certainly not complete. If you want to learn more, go read the documentation, experiment with conda, and maybe have an in-depth look to how Conda-forge and Bioconda do build packages, as everything is hosted on github.

https://en.xdnf.cn/q/71408.html

Related Q&A

Numpy array larger than RAM: write to disk or out-of-core solution?

I have the following workflow, whereby I append data to an empty pandas Series object. (This empty array could also be a NumPy array, or even a basic list.)in_memory_array = pd.Series([])for df in list…

Pandas DataFrame styler - How to style pandas dataframe as excel table?

How to style the pandas dataframe as an excel table (alternate row colour)? Sample style:Sample data: import pandas as pd import seaborn as snsdf = sns.load_dataset("tips")

Remove namespace with xmltodict in Python

xmltodict converts XML to a Python dictionary. It supports namespaces. I can follow the example on the homepage and successfully remove a namespace. However, I cannot remove the namespace from my XM…

Groupby count only when a certain value is present in one of the column in pandas

I have a dataframe similar to the below mentioned database:+------------+-----+--------+| time | id | status |+------------+-----+--------+| 1451606400 | id1 | Yes || 1451606400 | id1 | Yes …

how to save tensorflow model to pickle file

I want to save a Tensorflow model and then later use it for deployment purposes. I dont want to use model.save() to save it because my purpose is to somehow pickle it and use it in a different system w…

PySide2 Qt3D mesh does not show up

Im diving into Qt3D framework and have decided to replicate a simplified version of this c++ exampleUnfortunately, I dont see a torus mesh on application start. Ive created all required entities and e…

Unable to import module lambda_function: No module named psycopg2._psycopg aws lambda function

I have installed the psycopg2 with this command in my package folder : pip install --target ./package psycopg2 # Or pip install -t ./package psycopg2now psycopg2 module is in my package and I have crea…

RestrictedPython: Call other functions within user-specified code?

Using Yuri Nudelmans code with the custom _import definition to specify modules to restrict serves as a good base but when calling functions within said user_code naturally due to having to whitelist e…

TypeError: object of type numpy.int64 has no len()

I am making a DataLoader from DataSet in PyTorch. Start from loading the DataFrame with all dtype as an np.float64result = pd.read_csv(dummy.csv, header=0, dtype=DTYPE_CLEANED_DF)Here is my dataset cla…

VS Code Pylance not highlighting variables and modules

Im using VS Code with the Python and Pylance extensions. Im having a problem with the Pylance extension not doing syntax highlight for things like modules and my dataframe. I would expect the modules…