python postgresql: reliably check for updates in a specific table

2024/11/18 23:45:00

Situation: I have a live trading script which computes all sorts of stuff every x minutes in my main thread (Python). the order sending is performed through such thread. the reception and execution of such orders though is a different matter as I cannot allow x minutes to pass but I need them as soon as they come in. I initialized another thread to check for such data (execution) which is in a database table (POSTGRES SQL).

Problem(s): I cannot continuosly perform query every xx ms, get data from DB, compare table length, and then get the difference for a variety of reasons (not only guy to use such DB, perforamnce issues, etc). so I looked up some solutions and came up with this thread (https://dba.stackexchange.com/questions/58214/getting-last-modification-date-of-a-postgresql-database-table) where basically the gist of it was that "There is no reliable, authorative record of the last modified time of a table".

Question: what can I do about it, that is: getting near instantenuous responses from a postgres sql table without overloading the whole thing using Python?

Answer

You can use notifications in postgresql:

import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT
import selectdef dblisten(dsn):connection = psycopg2.connect(dsn)connection.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)cur = connection.cursor()cur.execute("LISTEN new_id;")while True:select.select([connection],[],[])connection.poll()events = []while connection.notifies:notify = connection.notifies.pop().payloaddo_something(notify)

and install a trigger for each update:

CREATE OR REPLACE FUNCTION notify_id_trigger() RETURNS trigger AS $$
BEGINPERFORM pg_notify('new_id', NEW.ID);RETURN new;
END;
$$ LANGUAGE plpgsql;CREATE TRIGGER data_modified AFTER insert or update on data_table for each row execute procedure notify_id_trigger();")
https://en.xdnf.cn/q/71004.html

Related Q&A

How to push to remote repo with GitPython

I have to clone a set of projects from one repository and push it then to a remote repository automatically. Therefore im using python and the specific module GitPython. Until now i can clone the proje…

How do I do use non-integer string labels with SVM from scikit-learn? Python

Scikit-learn has fairly user-friendly python modules for machine learning.I am trying to train an SVM tagger for Natural Language Processing (NLP) where my labels and input data are words and annotatio…

Python - walk through a huge set of files but in a more efficient manner

I have huge set of files that I want to traverse through using python. I am using os.walk(source) for the same and is working but since I have a huge set of files it is taking too much and memory resou…

Python: handling a large set of data. Scipy or Rpy? And how?

In my python environment, the Rpy and Scipy packages are already installed. The problem I want to tackle is such:1) A huge set of financial data are stored in a text file. Loading into Excel is not pos…

Jupyter notebook - cant import python functions from other folders

I have a Jupyter notebook, I want to use local python functions from other folders in my computer. When I do import to these functions I get this error: "ModuleNotFoundError: No module named xxxxx…

Can pandas plot a time-series without trying to convert the index to Periods?

When plotting a time-series, I observe an unusual behavior, which eventually results in not being able to format the xticks of the plot. It seems that pandas internally tries to convert the index into …

pip install syntax for allowing insecure

I tried to run$pip install --upgrade --allow-insecure setuptoolsbut it doesnt seem to work? is my syntax wrong?this is on ubuntu 13.10 I need --allow-insecure as I havent been able to the get the co…

how do I determine the locations of the points after perspective transform, in the new image plane?

Im using OpenCV+Python+Numpy and I have three points in the image, I know the exact locations of those points.(P1, P2);N1I am going to transform the image to another view, (for example I am transformin…

How to do a simple Gaussian mixture sampling and PDF plotting with NumPy/SciPy?

I add three normal distributions to obtain a new distribution as shown below, how can I do sampling according to this distribution in python?import matplotlib.pyplot as plt import scipy.stats as ss im…

Python dict.get() or None scenario [duplicate]

This question already has answers here:Truth value of a string in python(4 answers)Closed 7 years ago.I am attempting to access a dictionarys values based on a list of keys I have. If the key is not pr…