Pass many pieces of data from Python to C program

2024/10/14 13:24:19

I have a Python script and a C program and I need to pass large quantities of data from Python script that call many times the C program. Right now I let the user choose between passing them with an ASCII file or a binary file, but both are quite slow and useless (I mean files are useful if you want to store the data, but I delete these files at the end of the script).

os.system doesn't work, the arguments are too much as the C program too uses files to return data to Python, but this is much less data.

I wonder what I can use to make this exchange fast. Writing the files to a RAM disk? If so, how can I do this?

I heard is possible to call functions from DLL using ctypes, but don't know how to compile my program as a DLL (I use wxdevc+ on Windows 7 64). Or wrap it, but still don't know if it can work and if it is efficient.

The data are vertices of a 3D mesh.

I'm running the Python script inside another program (blender (open source), and is called many times (usually more than 500 times) because it's inside a cycle. The script send vertices information (1 int index and 3 float coords) to the program, and the program should return many vertices (only int index, because I can find the corresponding vertices with Python).

So this is not interactive, it's more like a function (but it's wrote in C). The script + C program (that are add-ons of blender) that I'm writing should be cross-platform because it will be redistributed.

The program is actually wrote in C, and from Python I can know the address in memory of the struct that contains the vertices data. If only I know how to do this, should be better to pass to the C program only an address, and from there find all the other vertices (are stored in list).

But as far as I know, I can't access to the memory space of another program, and I don't know if calling the program with pipes or whatever initialize a new thread or is run inside the script (that is actually run under the Blender thread)

Here is the source and blender/source/blender/makesdna/DNA_meshdata_types.h should be the struct definition

Answer

Pipes are the obvious way to go; if your c program accepts input from stdin, you can use Popen. This doesn't create a "thread" as you say in your edit; it creates an entirely new process with separate memory:

from subprocess import Popen, PIPEinput = "some input"
cproc = Popen("c_prog", stdin=PIPE, stdout=PIPE)
out, err = cproc.communicate(input)

Here's a more detailed example. First, a simple c program that echoes stdin:

#include<stdio.h>
#include<stdlib.h>
#define BUFMAX 100int main() {char buffer[BUFMAX + 1];char *bp = buffer;int c;FILE *in;while (EOF != (c = fgetc(stdin)) && (bp - buffer) < BUFMAX) {*bp++ = c;}*bp = 0;    // Null-terminate the stringprintf("%s", buffer);
}

Then a python program that pipes input (from argv in this case) to the above:

from subprocess import Popen, PIPE
from sys import argvinput = ' '.join(argv[1:])
if not input: input = "no arguments given"
cproc = Popen("./c_prog", stdin=PIPE, stdout=PIPE)
out, err = cproc.communicate(input)
print "output:", out
print "errors:", err

If you don't plan to use the c program without the python frontend, though, you might be better off inlining a c function, perhaps using instant.

from instant import inline
c_code = """[ ... some c code ... ] //see the below page for a more complete example.
"""
c_func = inline(c_code)

As Joe points out, you could also write a python module in c: Extending Python with C or C++

This answer discusses other ways to combine c and python: How do I connect a Python and a C program?

EDIT: Based on your edit, it sounds like you really should create a cpython extension. If you want some example code, let me know; but a full explanation would make for a unreasonably long answer. See the link above (Extending Python...) for everything you need to know.

https://en.xdnf.cn/q/69412.html

Related Q&A

Parse JavaScript to instrument code

I need to split a JavaScript file into single instructions. For examplea = 2; foo() function bar() {b = 5;print("spam"); }has to be separated into three instructions. (assignment, function ca…

Converting all files (.jpg to .png) from a directory in Python

Im trying to convert all files from a directory from .jpg to .png. The name should remain the same, just the format would change.Ive been doing some researches and came to this:from PIL import Image im…

AssertionError: Gaps in blk ref_locs when unstack() dataframe

I am trying to unstack() data in a Pandas dataframe, but I keep getting this error, and Im not sure why. Here is my code so far with a sample of my data. My attempt to fix it was to remove all rows whe…

Python does not consider distutils.cfg

I have tried everything given and the tutorials all point in the same direction about using mingw as a compiler in python instead of visual c++.I do have visual c++ and mingw both. Problem started comi…

Is it possible to dynamically generate commands in Python Click

Im trying to generate click commands from a configuration file. Essentially, this pattern:import click@click.group() def main():passcommands = [foo, bar, baz] for c in commands:def _f():print("I a…

Different accuracy between python keras and keras in R

I build a image classification model in R by keras for R.Got about 98% accuracy, while got terrible accuracy in python.Keras version for R is 2.1.3, and 2.1.5 in pythonfollowing is the R model code:mod…

Named Entity Recognition in aspect-opinion extraction using dependency rule matching

Using Spacy, I extract aspect-opinion pairs from a text, based on the grammar rules that I defined. Rules are based on POS tags and dependency tags, which is obtained by token.pos_ and token.dep_. Belo…

Python Socket : AttributeError: __exit__

I try to run example from : https://docs.python.org/3/library/socketserver.html#socketserver-tcpserver-example in my laptop but it didnt work.Server :import socketserverclass MyTCPHandler(socketserver.…

How to save pygame Surface as an image to memory (and not to disk)

I am developing a time-critical app on a Raspberry PI, and I need to send an image over the wire. When my image is captured, I am doing like this:# pygame.camera.Camera captures images as a Surface pyg…

Plotting Precision-Recall curve when using cross-validation in scikit-learn

Im using cross-validation to evaluate the performance of a classifier with scikit-learn and I want to plot the Precision-Recall curve. I found an example on scikit-learn`s website to plot the PR curve …