I am building a web app with Python Flask with JavaScript. I am a beginner of Javascript.
The process I do now:
In Flask Python code,
1. I get data by scrapping the web (numeric data that updates every minute).
2. use the data and calculate something and get the final numbers.
3. make a list that contains the final numbers
4. feed the list to a page by adding the list to the page's definition of Flask
5. Now in HTML get the list by capturing it with {{ data|safe }} tag
6. use it with Javascript to make a chart.
The problem is:
In the step 1, the data I get is being updated every minute. For example, on that web page now there are 15 data points. I parse the last 10 data points from that web page and then I put them in a list in my Python and then do the following steps and make a chart on my webpage. One minute later, in the data source web page, there will be 16 data point available, and I need to get the last 10 data points. In that case, I need to run the python code again to get the latest 10 data points to use them to make a chart on my web page.
So, I need to always run the whole python code which is the whole Flask app init.py file and re-render my webpage to see the updated chart. If I do not re-run the init.py file in my server, then even after 10 minutes or 2 hours, I will only see the data that I parsed for the first time forever.
How should I run the Flask and always get the updated data without always re-run the flask init.py every time.
I thought about using time.sleep(60) so that the flask app python file is run every 1 minutes. But this is really taking alot of time when my code gets much more thinks to calculate. And do not really work.
How should I solve this problem??
Should I use time.sleep ? or is threre better way?
This is a classic "push" update problem. Your web server is updated with information that you would like to frequently update (or "push") to your web client front-end.
As PJ Santoro suggests, you can repeatedly poll your server (every 10 seconds say) along the lines of "Is there new data? No. Okay. Is there new data? No. Okay. Is there new data? Yes, here it is! Great! Is there new data? ...." It is rather inefficient, but for low data volumes you might never see that problem.
A more efficient update system would have the server send an update signal only when new data is ready. There are two major ways to do that:
Long polling, also known as reverse AJAX and Comet. Your web page opens an AJAX request on a server URL such as /update_data with a really long timeout (say 30 minutes or an hour). When the server has data, it sends it through the waiting AJAX connection. There are complications with this, including managing concurrency on the server side (not historically Python's strong suit) and the fact that even the maximum timeout (~1 hour last I used this technique) might expire without data, and some error handling is needed to manage that. If you wan to try long polling / reverse AJAX, here is an example using Flask.
WebSockets. WebSockets are a much more recent approach to interactive and "push" updates between servers and clients. They're very well-used; projects like the Jupyter Notebook depend on them extensively. They're extremely efficient and well-fit for this kind of incremental update of client state, and the code is often much less Byzantine and confusing than retrofits like reverse AJAX. But...there are complexities with WebSockets too. Managing concurrency in the Flask server side is still a significant issue, for example. If you'd like to consider WebSockets as your update mechanism, here is an example of how to use them with Flask.
Either approach you use, if your data communicated over time grows, you will also need to structure the data transmitted on each update so that it's an incremental update. No mater how good the plumbing, if you transmit thousands of data points every update, it's not going to be fast. But, long polling and WebSockets plumbing should at least get you a great distance toward a legitimate real-time update capability.