Py4J has bigger overhead than Jython and JPype

2024/10/10 18:28:21

After searching for an option to run Java code from Django application(python), I found out that Py4J is the best option for me. I tried Jython, JPype and Python subprocess and each of them have certain limitations:

  • Jython. My app runs in python.
  • JPype is buggy. You can start JVM just once after that it fails to start again.
  • Python subprocess. Cannot pass Java object between Python and Java, because of regular console call.

On Py4J web site is written:

In terms of performance, Py4J has a bigger overhead than both of the previous solutions (Jython and JPype) because it relies on sockets, but if performance is critical to your application, accessing Java objects from Python programs might not be the best idea.

In my application performance is critical, because I'm working with Machine learning framework Mahout. My question is: Will Mahout also run slower because of Py4J gateway server or this overhead just mean that invoking Java methods from Python functions is slower (in latter case performance of Mahout will not be a problem and I can use Py4J).

Answer

JPype issue that @HIP_HOP mentioned with JVM getting detached from new threads can be overcome with the following hack (add it before the first call to Java objects in the new thread which does not have JVM yet):

# ensure that current thread is attached to JVM
# (essential to prevent JVM / entire container crashes 
# due to "JPJavaEnv::FindClass" errors)
if not jpype.isThreadAttachedToJVM():jpype.attachThreadToJVM()
https://en.xdnf.cn/q/69863.html

Related Q&A

how to uninstall opencv-python package installed by using pip in anaconda?

I have tried to install OpenCV in anaconda. but when I use it, I figure out the instead of using OpenCV, the program using OpenCV-python and that why my program crashed. I type "conda uninstall op…

flask many to many join as done by prefetch_related from django

I have following Group and Contact model in flask with Sql Alchemy ORMgroup_contact = db.Table(group_contact,db.Column(group_id, db.Integer, db.ForeignKey(group.id)),db.Column(contact_id, db.Integer, d…

Django model inheritance - only want instances of parent class in a query

Lets say I have 2 models, one being the parent of another. How can I query all Places that arent restaurants in Django? Place.objects.all() would include all restaurants right? I want to exclude the …

Perfom python unit tests via a web interface

Is it possible to perform unittest tests via a web interface...and if so how?EDIT: For now I want the results...for the tests I want them to be automated...possibly every time I make a change to the …

Limit on number of HDF5 Datasets

Using h5py to create a hdf5-file with many datasets, I encounter a massive Speed drop after ca. 2,88 mio datasets. What is the reason for this?I assume that the limit of the tree structure for the dat…

Object level cascading permission in Django

Projects such as Django-guardian and django-permissions enables you to have object level permissions. However, if two objects are related to each other by a parent-child relationship, is there any way …

How do I find out eigenvectors corresponding to a particular eigenvalue of a matrix?

How do I find out eigenvectors corresponding to a particular eigenvalue? I have a stochastic matrix(P), one of the eigenvalues of which is 1. I need to find the eigenvector corresponding to the eigen…

How to install my custom Python package with its custom dependencies?

I would like to find a way to install my own python package which depends on other custom python packages. I followed this guide to create my own python packages: https://python-packaging.readthedocs.i…

How to call a function only Once in Python [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.Want to improve this question? Add details and clarify the problem by editing this post.Closed 8 years ago.The com…

Generating Compound Pie, or Pie of Pie Charts

Below is an example of a compound pie chart, also known as a pie of pie chart drawn using Excel. Is it possible to create a figure like this using python?