ePrivacy and GPDR Cookie Consent by Cookie Consent Skip to main content

Loader Google BigQuery

The Google BigQuery Loader job entry enables you to load data into Google BigQuery from a Google Cloud Storage account.

Prerequisite

OAuth
  • Create OAuth client IDs or obtain an existing client id (OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET)
  • Use these client credentials in (https://developers.google.com/oauthplayground)[Google OAuth playground] to create new authorization, use https://www.googleapis.com/auth/bigquery scope. 
Service Account

This app requires 2 sets of service account credentials - one for managing the tests (fixtures, etc), one for running the tests. They have different scopes.

Manage

  • Create a new service account (IAM > Service accounts), e.g. BigQuery Writer Manage
  • Add BigQuery Data Owner and BigQuery Job User roles to the user
  • Create JSON key
  • Remove newlines from the JSON and assign it to the SERVICE_ACCOUNT_MANAGE variable in the .env file

User

  • Create a new service account (IAM > Service accounts), e.g. BigQuery Writer User
  • Add BigQuery User and BigQuery Job User roles to the user
  • Create JSON key
  • Remove newlines from the JSON and assign it to the SERVICE_ACCOUNT_USER variable in the .env file

Data In

Upload all files in /data/in/files to the specified bucket and prefix. Existing files are overwritten.

Data Out

N/A

Read more: about the folder structure here.

Parameters

All parameters are set in Google.

01.png

Project (required) 
Project name

02.png

 

 

 

 

 

 

Back to top