Running parameterized queries

2024/11/19 20:17:51

Quite new to this google bigquery sql thing so please bear with me. I'm trying to build a google standardSQL parameterized query. The following sample was used and ran successfully on Google BigQuery WebUI.

#standardSQLWITH time AS (SELECT TIMESTAMP_MILLIS(timestamp) AS trans_time,inputs.input_pubkey_base58 AS input_key,outputs.output_pubkey_base58 AS output_key,outputs.output_satoshis AS satoshis,transaction_id AS trans_idFROM `bigquery-public-data.bitcoin_blockchain.transactions`JOIN UNNEST (inputs) AS inputsJOIN UNNEST (outputs) AS outputsWHERE inputs.input_pubkey_base58 = '1XPTgDRhN8RFnzniWCddobD9iKZatrvH4'OR outputs.output_pubkey_base58 = '1XPTgDRhN8RFnzniWCddobD9iKZatrvH4')SELECT input_key, output_key, satoshis, trans_id,EXTRACT(DATE FROM trans_time) AS dateFROM timeWHERE trans_time >= '2010-05-21' AND trans_time <= '2010-05-23' AND satoshis >= 1000000000000--ORDER BY date

Sample extracted from here as a side note.

This gives 131 rows:

Table sample

What I would like to be able to do, is to use the ScalarQueryParameter, so I could programatically use some vars along the way. Like this:

myquery = """
#standardSQLWITH time AS (SELECT TIMESTAMP_MILLIS(timestamp) AS trans_time,inputs.input_pubkey_base58 AS input_key,outputs.output_pubkey_base58 AS output_key,outputs.output_satoshis AS satoshis,transaction_id AS trans_idFROM `bigquery-public-data.bitcoin_blockchain.transactions`JOIN UNNEST (inputs) AS inputsJOIN UNNEST (outputs) AS outputsWHERE inputs.input_pubkey_base58 = @pubkeyOR outputs.output_pubkey_base58 = @pubkey)SELECT input_key, output_key, satoshis, trans_id,EXTRACT(DATE FROM trans_time) AS dateFROM timeWHERE trans_time >= @mdate AND trans_time <= @tdate AND satoshis >= 1000000000000--ORDER BY date
"""varInitDate = '2010-05-21'
varEndDate = '2010-05-23'
pubkey = '1XPTgDRhN8RFnzniWCddobD9iKZatrvH4'query_params = [bigquery.ScalarQueryParameter('mdate', 'STRING', varInitDate),bigquery.ScalarQueryParameter('tdate', 'STRING', varEndDate),bigquery.ScalarQueryParameter('pubkey', 'STRING', pubkey)
]job_config = bigquery.QueryJobConfig()
job_config.query_parameters = query_params
query_job = client.query(myquery,job_config=job_config)

Nevertheless, i'm facing the following error:

<google.cloud.bigquery.table.RowIterator object at 0x7fa098be85f8>
Traceback...
TypeError: 'RowIterator' object is not callable

Can someone pls enlighten me on how can i achieve the mentioned purpose ?

P.S - '1XPTgDRhN8RFnzniWCddobD9iKZatrvH4' is the Laszlo’s Pizza 10.000 bitcoin exchange (1000000000000 satoshis).

Answer

So ... the problem was with this line of code that didn't work as expected. Not sure why though, as it worked with queries that didn't have parameterized vars.

results = query_job.result()
df = results().to_dataframe()

And the actual code... Remember to replace with your own login credentials for this to work.

import datetime, time
from google.cloud import bigquery
from google.oauth2 import service_account
import pandas as pd#login
credentials = service_account.Credentials.from_service_account_file('your.json')
project_id = 'your-named-project'
client = bigquery.Client(credentials= credentials,project=project_id)#The query
q_input = """
#standardSQLWITH time AS (SELECT TIMESTAMP_MILLIS(timestamp) AS trans_time,inputs.input_pubkey_base58 AS input_key,outputs.output_pubkey_base58 AS output_key,outputs.output_satoshis AS satoshis,transaction_id AS trans_idFROM `bigquery-public-data.bitcoin_blockchain.transactions`JOIN UNNEST (inputs) AS inputsJOIN UNNEST (outputs) AS outputsWHERE inputs.input_pubkey_base58 = @pubkeyOR outputs.output_pubkey_base58 = @pubkey)SELECT input_key, output_key, satoshis, trans_id,EXTRACT(DATE FROM trans_time) AS dateFROM timeWHERE trans_time >= @mdate AND trans_time <= @tdate AND satoshis >= @satoshis--ORDER BY date
"""#The desired purpose
def runQueryTransaction(varInitDate,varEndDate,pubkey,satoshis):global dfquery_params = [bigquery.ScalarQueryParameter('mdate', 'STRING', varInitDate),bigquery.ScalarQueryParameter('tdate', 'STRING', varEndDate),bigquery.ScalarQueryParameter('pubkey', 'STRING', pubkey),bigquery.ScalarQueryParameter('satoshis', 'INT64', satoshis),]job_config = bigquery.QueryJobConfig()job_config.query_parameters = query_paramsquery_job = client.query(q_input,job_config=job_config)  # API request - starts the queryresults = query_job.result()  # Waits for job to complete.df=pd.DataFrame(columns=['input_key', 'output_key', 'satoshis', 'trans_id', 'date'])for row in results:df.loc[len(df)] = [row.input_key, row.output_key, row.satoshis, row.trans_id, row.date]#print("{} : {} : {} : {} : {}".format(row.input_key, row.output_key, row.satoshis, row.trans_id, row.date))return df#runQueryTransaction(InitialDate,EndDate,WalletPublicKey,Satoshis)
runQueryTransaction('2010-05-21','2010-05-23','1XPTgDRhN8RFnzniWCddobD9iKZatrvH4',1000000000000)

Cheers

https://en.xdnf.cn/q/118498.html

Related Q&A

how to do this disparity map in OpenCV

paper "Fast Obstacle Detection Using U-Disparity Maps with Stereo Vision"reference paper: "Fast Obstacle Detection Using U-Disparity Maps with Stereo Vision"I want to ask can opencv…

How do I give out a role when I click on a reaction? It doesnt work for me?

Im trying to make sure that when you enter a command, a message is sent and a reaction is set, and those who click on the reaction get a role, but when you enter a command, nothing happens. Here is the…

Python Pandas DataFrame Rounding of big fraction values [closed]

Closed. This question needs debugging details. It is not currently accepting answers.Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to repro…

Splitting Data in Python?

it does not work. I want to split data as in code in lines attribute. class movie_analyzer:def __init__(self,s):for c in punctuation:import removiefile = open(s, encoding = "latin-1")movielis…

Use local function variable inside loop [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.Want to improve this question? Add details and clarify the problem by editing this post.Closed 7 years ago.Improve…

Groupby Pandas , calculate multiple columns based on date difference

I have a pandas dataframe shown below: CID RefID Date Group MID 100 1 1/01/2021 A 100 2 3/01/2021 A 100 3 4/01/20…

EC2 .bashrc and .bash_profile re-setting

Reason Im asking: pycurl requires both libcurl-devel and openssl-devel. To install these, I have these two lines the my .bash_profile: sudo yum install libcurl-devel sudo yum install -y openssl-develPr…

How to load mutiple PPM files present in a folder as single Numpy ndarray?

The following Python code creates list of numpy array. I want to load by data sets as a numpy array that has dimension K x M x N x 3 , where K is the index of the image and M x N x 3 is the dimension …

Python: Understanding Threading Module

While learning Pythons threading module Ive run a simple test. Interesting that the threads are running sequentially and not parallel. Is it possible to modify this test code so a program executes the …

How the program has control with the break statement [closed]

Its difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying thi…