Loader Google BigQuery

The Google BigQuery Loader job entry enables you to load data into Google BigQuery from a Google Cloud Storage account.

Prerequisite

OAuth

• Create OAuth client IDs or obtain an existing client id (OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET)
• Use these client credentials in (https://developers.google.com/oauthplayground)[Google OAuth playground] to create new authorization, use https://www.googleapis.com/auth/bigquery scope. 

Service Account

This app requires 2 sets of service account credentials - one for managing the tests (fixtures, etc), one for running the tests. They have different scopes.

Manage

• Create a new service account (IAM > Service accounts), e.g. BigQuery Writer Manage
• Add BigQuery Data Owner and BigQuery Job User roles to the user
• Create JSON key
• Remove newlines from the JSON and assign it to the SERVICE_ACCOUNT_MANAGE variable in the .env file

User

• Create a new service account (IAM > Service accounts), e.g. BigQuery Writer User
• Add BigQuery User and BigQuery Job User roles to the user
• Create JSON key
• Remove newlines from the JSON and assign it to the SERVICE_ACCOUNT_USER variable in the .env file

Data In

Upload all files in /data/in/files to the specified bucket and prefix. Existing files are overwritten.

Data Out

N/A

Read more: about the folder structure here.

Parameters

All parameters are set in Google.