Why couldnt Julia superset python?

2024/10/18 15:43:25

The Julia Language syntax looks very similar to python, while the concept of a class (if one should address it as such a thing) is more what you use in C. There were many reasons why the creators decided on the difference with respect to the OOP. Still would it have been so hard (in comparison to create Julia in first place which is impressive) to find some canonical way to interpret python to Julia and thus get a hold of all the python libraries?

Answer

Yes. The design of Python makes it fundamentally difficult to optimize at compile-time (i.e. before you run the code). It is simply false that Julia is fast BECAUSE of its JIT. Rather, Julia is designed with its type system and multiple dispatch in mind so that way the compiler can know all of the necessary details to compile "the same code you would have written in C". That's what makes it fast: the type system. It makes a few trade-offs that allow it to, in "type-stable" functions, fully deduce what the types of every variable is, know what the memory layout of the type should be (including parametric types, so Vector{Float64} has a memory layout which is determined by the type and its parameter which inlines Float64 values like a NumPy array, except this is generalized in a way that your own struct types get the same efficiency), and compile a version of the code specifically for the types which are seen.

There are many ways where this is at odds with Python. For example, if the number of fields in a struct could change, then the memory layout could not be determined and thus these optimizations cannot occur at "compile-time". Julia was painstakingly designed to make sure that it would have type inferrability, and it uses that to generate code which is fully typed and remove all runtime checks (in type-stable functions. When a function is not type-stable, the types of the variables become dynamic rather than static and it slows down to Python-like speeds). In this sense, Julia actually isn't even optimized yet: all of its performance comes "for free" given the design of its type system. Python/MATLAB/R has to try really hard to optimize at runtime because it doesn't have the capability to do these deductions. In fact, those languages are "better optimized" right now in terms of runtime optimizations, but no one has really worked on runtime optimizations in Julia yet because in most performance sensitive cases you can get it all at compile time.

So then, what about Numba? Numba tries to take the route that Julia takes but with Python code by limiting what can be done so that way it can get type-stable code and compile that efficiently. However, this means a few things. First of all, it's not compatible with all Python codes or libraries. But more importantly, since Python is not a language built around its type system, the tools for controlling the code at the level of types is much reduced. So Numba doesn't have parametric vectors and generic codes which auto-specialize via multiple dispatch because these aren't features of the language. But that also means that it cannot make full use of the design, which limits how much it can do. It can handle the "use only floating point array" stuff just fine, but you can see it gets limited if you want one code to produce efficient code for "any number type, even ones I don't know about". However, by design, Julia does this automatically.

So at the core, Julia and Python are extremely different languages. It can be hard to see because Julia's syntax is close to Python's syntax, but they do not work the same at all.

This is a short summary of what I have described in a few blog posts. These go into more detail and show you how Julia is actually generating efficient code, how it gives you a generic "Python looking style" but doing so with full inferrability all the way down, and what the tradeoffs are.

How type-stability plus multiple dispatch gives performance:

http://ucidatascienceinitiative.github.io/IntroToJulia/Html/WhyJulia http://www.stochasticlifestyle.com/7-julia-gotchas-handle/

How the type system allows for highly performant generic designs

http://www.stochasticlifestyle.com/type-dispatch-design-post-object-oriented-programming-julia/

https://en.xdnf.cn/q/72589.html

Related Q&A

How to log into Google Cloud Storage from a python function?

I am new to google cloud storage and I try to set up a function that downloads a blob once a day. At the moment I am working in my Jupyter Notebook but finally, the code will run in an Azure Function. …

OpenCV Python, reading video from named pipe

I am trying to achieve results as shown on the video (Method 3 using netcat)https://www.youtube.com/watch?v=sYGdge3T30oThe point is to stream video from raspberry pi to ubuntu PC and process it using …

Load model with ML.NET saved with keras

I have a Neural Network implemented in Python with Keras. Once I have trained it I have exported the model and I have got two files: model.js and model.h5. Now I want to classify in real time inside a …

unable to add spark to PYTHONPATH

I am struggling to add spark to my python path:(myenv)me@me /home/me$ set SPARK_HOME="/home/me/spark-1.2.1-bin-hadoop2.4" (myenv)me@me /home/me$ set PYTHONPATH=$PYTHONPATH:$SPARK_HOME:$SPARK_…

Is there a way to shallow copy an existing file-object?

The use case for this would be creating multiple generators based on some file-object without any of them trampling each others read state. Originally I (thought I) had a working implementation using s…

Python - Import package modules before as well as after setup.py install

Assume a Python package (e.g., MyPackage) that consists of several modules (e.g., MyModule1.py and MyModule2.py) and a set of unittests (e.g., in MyPackage_test.py).. ├── MyPackage │ ├── __ini…

Where can I find the _sre.py python built-in module? [closed]

Closed. This question is seeking recommendations for books, tools, software libraries, and more. It does not meet Stack Overflow guidelines. It is not currently accepting answers.We don’t allow questi…

When numba is effective?

I know numba creates some overheads and in some situations (non-intensive computation) it become slower that pure python. But what I dont know is where to draw the line. Is it possible to use order of …

How to launch a Windows shortcut using Python

I want to launch a shortcut named blender.ink located at "D://games//blender.ink". I have tryed using:-os.startfile ("D://games//blender.ink")But it failed, it only launches exe fil…

Providing context in TriggerDagRunOperator

I have a dag that has been triggered by another dag. I have passed through to this dag some configuration variables via the DagRunOrder().payload dictionary in the same way the official example has don…