How do I list my scheduled queries via the Python google client API?

2024/11/15 23:54:38

I have set up my service account and I can run queries on bigQuery using client.query().

I could just write all my scheduled queries into this new client.query() format but I already have many scheduled queries so I was wondering if there is a way I can get/list the scheduled queries and then use that information to run those queries from a script.

I can query with big Query but I want to list queries

Answer

Yes, you can use the APIs. When you don't know which one to use, I have a tip. Use the command proposed by @Yev

bq ls --transfer_config --transfer_location=US --format=prettyjson

But log the API calls. for that use the --apilog <logfile name> parameter like that

bq --apilog ./log ls --transfer_config --transfer_location=US --format=prettyjson

And, magically, you can find the API called by the command: https://bigquerydatatransfer.googleapis.com/v1/projects/<PROJECT-ID>/locations/US/transferConfigs?alt=json

Then, a simple google search leads you to the correct documentation


In python, add that dependencies in your requirements.txt: google-cloud-bigquery-datatransfer and use that code

from google.cloud import bigquery_datatransferclient = bigquery_datatransfer.DataTransferServiceClient()
parent = client.common_project_path("<PROJECT-ID>")
resp = client.list_transfer_configs(parent=parent)
print(resp)
https://en.xdnf.cn/q/71409.html

Related Q&A

What does conda env do under the hood?

After searching and not finding, I must ask here:How does conda env work under the hood, meaning, how does anaconda handle environments?To clarify, I would like an answer or a reference to questions l…

Numpy array larger than RAM: write to disk or out-of-core solution?

I have the following workflow, whereby I append data to an empty pandas Series object. (This empty array could also be a NumPy array, or even a basic list.)in_memory_array = pd.Series([])for df in list…

Pandas DataFrame styler - How to style pandas dataframe as excel table?

How to style the pandas dataframe as an excel table (alternate row colour)? Sample style:Sample data: import pandas as pd import seaborn as snsdf = sns.load_dataset("tips")

Remove namespace with xmltodict in Python

xmltodict converts XML to a Python dictionary. It supports namespaces. I can follow the example on the homepage and successfully remove a namespace. However, I cannot remove the namespace from my XM…

Groupby count only when a certain value is present in one of the column in pandas

I have a dataframe similar to the below mentioned database:+------------+-----+--------+| time | id | status |+------------+-----+--------+| 1451606400 | id1 | Yes || 1451606400 | id1 | Yes …

how to save tensorflow model to pickle file

I want to save a Tensorflow model and then later use it for deployment purposes. I dont want to use model.save() to save it because my purpose is to somehow pickle it and use it in a different system w…

PySide2 Qt3D mesh does not show up

Im diving into Qt3D framework and have decided to replicate a simplified version of this c++ exampleUnfortunately, I dont see a torus mesh on application start. Ive created all required entities and e…

Unable to import module lambda_function: No module named psycopg2._psycopg aws lambda function

I have installed the psycopg2 with this command in my package folder : pip install --target ./package psycopg2 # Or pip install -t ./package psycopg2now psycopg2 module is in my package and I have crea…

RestrictedPython: Call other functions within user-specified code?

Using Yuri Nudelmans code with the custom _import definition to specify modules to restrict serves as a good base but when calling functions within said user_code naturally due to having to whitelist e…

TypeError: object of type numpy.int64 has no len()

I am making a DataLoader from DataSet in PyTorch. Start from loading the DataFrame with all dtype as an np.float64result = pd.read_csv(dummy.csv, header=0, dtype=DTYPE_CLEANED_DF)Here is my dataset cla…