Category "google-bigquery"

BigQuery - extract the most used tables

I'm using the Google BigQuery and looking on the default audit dataset. I know that this dataset contains various data about the queries the users are running.

Big Query - Table 'Copy' from Console Not working, But the same works fine from bq CLI

Across project level when trying to Copying the tables in big query, it works fine using bq CLI but not from Console. Big Query --> Project:Dataset.tableXYZ

Number of rows affected by BigQuery query

I run daily commands to insert new records into a BigQuery table, and would like to log how many records get inserted each day. I create a QueryJob object that

Is there a way to upload the byte type created with zlib to Google bigquery?

I want to input string data into bigquery by implied by pyhton's zlib library. Here is an example code that uses zlib to generate data: import zlib import p

How do I produce a new column with values based on a Partition of one column while also based on the values of two additional columns (BigQuery)?

I have a table that records all the different statuses for a list of Jobs with timestamps. So the ID column has many Ids that appear several times as their stat

GBQ Transpose Dynamically without Execute Immediate

Hi All trying to figure out a way to transpose data in columns dynamically. In the data the distinct number of Traits will increase/decrease. I know I can hardc

BigQuery Results to Panda DataFrame in Chunks

I am trying to save the results of a BigQuery query to a Panda DataFrame using bigquery.Client.query.to_dataframe() This query can return millions of rows. Gi

GBQ Execute Immediate into a CTE

I am building an application/script for users that do not have write access to the database. Normally I would use Execute Immediate and save that result into a

TensorFlow SavedModel output output had no dimensions when loading a model with BigQuery ML

I'm trying to use BigQuery ML to load a saved tensorflow model to make predictions. However when I run the query that read the saved model in GCS I got the foll

BigQuery standard SQL: how to group by an ARRAY field

My table has two columns, id and a. Column id contains a number, column a contains an array of strings. I want to count the number of unique id for a given arra

Discrepancy in Daily user engagement between Firebase Analytics dashboard and BigQuery

I'm trying to develop a query against Firebase Analytics data linked to BigQuery to reproduce the "Daily user engagement" graph from the Firebase Analytics dash

How to avoid double counting while using SQL

I have a transaction table which contains date, transaction_id and category (it is a sales table of clothes). It looks like this: ndate transaction_id category

BigQuery - loading CSV with newlines fail with Node.js, but works with gsutil

I'm trying to load CSV from Cloud Storage to Big Query with a Cloud Function. The file has newlines in some of the strings. When I load from Cloud Shell it load

How do I find yesterday's date

I'm stuck on a query regarding dates in Big Query using SQL. I have a table that consists of customer_id (int), date_purchase (date), sales (int). The query is

Is there any way we can load BigTable data into BigQuery?

I want to load BigTable data into BigQuery with direct way. Till now I am loading BigTable data into CSV file using Python and then loading csv file into BigQue

UsageError: Line magic function `%%bigquery` not found

I am very new to Big query. I am trying to load data from a Big query table to pandas dataframe. I followed the syntax given in the documentation here. Unfortun

Table expiration in GCS to BQ Airflow task

I am copying a CSV into a new BQ table using the GCSToBigQueryOperator task in Airflow. Is there a way to add a table expiration to this table within this task?

Access Denied: BigQuery BigQuery: Permission denied while opening file

When trying to perform a simple query in BigQuery I am getting this error: Access Denied: BigQuery BigQuery: Permission denied while opening file. I am using a

Can we check if table in bigquery is in locked or DML operation is being performed

There is a BQ table which has multiple data load/update/delete jobs scheduled in. Since this is automated jobs many of it are failing due to concurrent update i

BigQuery Stored Procedure VS BigQuery Scripts

I am able to perform a task of join and aggregation using both Big Query Script and BigQuery Stored Procedure, which is better , which one should be my first ch