Could someone please share syntax to read/write bigquery table in a pipeline written in python for GCP Dataflow
Could someone please share syntax to read/write bigquery table in a pipeline written in python for GCP Dataflow
Run on Dataflow
First, construct a Pipeline
with the following options for it to run on GCP DataFlow:
import apache_beam as beamoptions = {'project': <project>,'runner': 'DataflowRunner','region': <region>,'setup_file': <setup.py file>}
pipeline_options = beam.pipeline.PipelineOptions(flags=[], **options)
pipeline = beam.Pipeline(options = pipeline_options)
Read from BigQuery
Define a BigQuerySource
with your query and use beam.io.Read
to read data from BQ:
BQ_source = beam.io.BigQuerySource(query = <query>)
BQ_data = pipeline | beam.io.Read(BQ_source)
Write to BigQuery
There are two options to write to bigquery:
use a BigQuerySink
and beam.io.Write
:
BQ_sink = beam.io.BigQuerySink(<table>, dataset=<dataset>, project=<project>)
BQ_data | beam.io.Write(BQ_sink)
use beam.io.WriteToBigQuery
:
BQ_data | beam.io.WriteToBigQuery(<table>, dataset=<dataset>, project=<project>)