'How to write to the _default stream of a BigQuery table using the Write Storage API
I would like to use the BigQuery Write Storage API to stream single rows of data into a table. However, I do not want to create new write streams for each row.
According to the docs, each table has a default stream that can be written to. Google's explanation:
Additionally, every table has a special stream named ‘_default’ to which data can be written. This stream doesn’t need to be created using CreateWriteStream. It is a stream that can be used simultaneously by any number of clients. Data written to this stream is considered committed as soon as an acknowledgement is received.
I am wondering how to access this "_default" stream and write to it using the Python client?
I tried the following:
from google.cloud import bigquery_storage_v1
write_client = bigquery_storage_v1.BigQueryWriteClient()
stream_name = "projects/{projectId}/datasets/{datasetId}/tables/{tableName}/streams/_default"
stream = write_client.get_write_stream(name=stream_name)
But was met with the error: NotFound: 404 Requested entity was not found
EDIT 2022-02-24:
It turns out you cannot GET the _default stream using the client code in my example above. However, you can write to the _default stream using the name format projects/{projectId}/datasets/{datasetId}/tables/{tableName}/_default
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|