Skip to content

Add Impersonate Service Account argument #2015

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wintermi wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add Impersonate Service Account argument #2015

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c3094eb
Add the '--impersonate-service-account' argument to the 'run' and 'te…
wintermi Sep 11, 2025
f3feefd
Merge upstream/main to sync with dataform-co/dataform
wintermi Jan 5, 2026
c65768d
Make changes as requested
wintermi Jan 5, 2026
85a0440
Make changes as requested
wintermi Jan 5, 2026
a5b4288
Make changes as requested
wintermi Jan 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cli/api/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ ts_library(
"@npm//deepmerge",
"@npm//fs-extra",
"@npm//glob",
"@npm//google-auth-library",
"@npm//google-sql-syntax-ts",
"@npm//js-beautify",
"@npm//js-yaml",
Expand Down
54 changes: 37 additions & 17 deletions cli/api/dbadapters/bigquery.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { GoogleAuth, Impersonated } from "google-auth-library";
import Long from "long";
import { PromisePoolExecutor } from "promise-pool-executor";

Expand Down Expand Up @@ -102,8 +103,8 @@ export class BigQueryDbAdapter implements IDbAdapter {
try {
await this.pool
.addSingleTask({
generator: () =>
this.getClient().query({
generator: async () =>
(await this.getClient()).query({
useLegacySql: false,
query,
dryRun: true
Expand All @@ -128,7 +129,8 @@ export class BigQueryDbAdapter implements IDbAdapter {
}

public async tables(): Promise<dataform.ITarget[]> {
const datasets = await this.getClient().getDatasets({ autoPaginate: true, maxResults: 1000 });
const client = await this.getClient();
const datasets = await client.getDatasets({ autoPaginate: true, maxResults: 1000 });
const tables = await Promise.all(
datasets[0].map(dataset => dataset.getTables({ autoPaginate: true, maxResults: 1000 }))
);
Expand Down Expand Up @@ -218,7 +220,7 @@ export class BigQueryDbAdapter implements IDbAdapter {
}

public async schemas(database: string): Promise<string[]> {
const data = await this.getClient(database).getDatasets();
const data = await (await this.getClient(database)).getDatasets();
return data[0].map(dataset => dataset.id);
}

Expand All @@ -238,7 +240,7 @@ export class BigQueryDbAdapter implements IDbAdapter {
metadata.schema.fields
);

await this.getClient(target.database)
await (await this.getClient(target.database))
.dataset(target.schema)
.table(target.name)
.setMetadata({
Expand All @@ -250,7 +252,7 @@ export class BigQueryDbAdapter implements IDbAdapter {

private async getMetadata(target: dataform.ITarget): Promise<TableMetadata> {
try {
const table = await this.getClient(target.database)
const table = await (await this.getClient(target.database))
.dataset(target.schema)
.table(target.name)
.getMetadata();
Expand All @@ -265,19 +267,36 @@ export class BigQueryDbAdapter implements IDbAdapter {
}
}

private getClient(projectId?: string) {
private async getClient(projectId?: string) {
Copy link
Contributor

@kolina kolina Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #2001 @ashish10alex added support for impersonating a service account through ADC. Will it be enough for your use case or you need an explicit option as well?

If the latter, can you please validate manually that it works?

projectId = projectId || this.bigQueryCredentials.projectId;
if (!this.clients.has(projectId)) {
this.clients.set(
const clientConfig: any = {
projectId,
new BigQuery({
scopes: EXTRA_GOOGLE_SCOPES,
location: this.bigQueryCredentials.location
};

if (this.bigQueryCredentials.impersonateServiceAccount) {
// For impersonation, create an Impersonated credential directly
const sourceAuth = new GoogleAuth({
projectId,
scopes: EXTRA_GOOGLE_SCOPES,
location: this.bigQueryCredentials.location,
credentials:
this.bigQueryCredentials.credentials && JSON.parse(this.bigQueryCredentials.credentials)
scopes: ['https://www.googleapis.com/auth/cloud-platform'],
credentials: this.bigQueryCredentials.credentials && JSON.parse(this.bigQueryCredentials.credentials),
});

const authClient = await sourceAuth.getClient();

clientConfig.authClient = new Impersonated({
sourceClient: authClient,
targetPrincipal: this.bigQueryCredentials.impersonateServiceAccount,
targetScopes: ['https://www.googleapis.com/auth/cloud-platform']
Copy link
Contributor

@kolina kolina Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EXTRA_GOOGLE_SCOPES?

Copy link
Author

@wintermi wintermi Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what you would like done here?

Copy link
Contributor

@kolina kolina Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean using EXTRA_GOOGLE_SCOPES here instead of hard-coding

})
);
} else {
clientConfig.credentials =
this.bigQueryCredentials.credentials && JSON.parse(this.bigQueryCredentials.credentials);
}

this.clients.set(projectId, new BigQuery(clientConfig));
}
return this.clients.get(projectId);
}
Expand All @@ -289,12 +308,12 @@ export class BigQueryDbAdapter implements IDbAdapter {
byteLimit?: number,
location?: string
) {
const results = await new Promise<any[]>((resolve, reject) => {
const results = await new Promise<any[]>(async (resolve, reject) => {
const allRows = new LimitedResultSet({
rowLimit,
byteLimit
});
const stream = this.getClient().createQueryStream({
const stream = (await this.getClient()).createQueryStream({
query,
params,
location
Expand Down Expand Up @@ -330,7 +349,8 @@ export class BigQueryDbAdapter implements IDbAdapter {
return retry(
async () => {
try {
const job = await this.getClient().createQueryJob({
const client = await this.getClient();
const job = await client.createQueryJob({
useLegacySql: false,
jobPrefix: "dataform-" + (jobPrefix ? `${jobPrefix}-` : ""),
query,
Expand Down
20 changes: 19 additions & 1 deletion cli/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,14 @@ const jobPrefixOption: INamedOption<yargs.Options> = {
}
};

const impersonateServiceAccountOption: INamedOption<yargs.Options> = {
name: "impersonate-service-account",
option: {
describe: "Service account email to impersonate during authentication.",
type: "string"
}
};

const bigqueryJobLabelsOption: INamedOption<yargs.Options> = {
name: "job-labels",
option: {
Expand All @@ -191,6 +199,7 @@ const bigqueryJobLabelsOption: INamedOption<yargs.Options> = {
}
};


const quietCompileOption: INamedOption<yargs.Options> = {
name: "quiet",
option: {
Expand Down Expand Up @@ -463,7 +472,7 @@ export function runCli() {
format: `test [${projectDirMustExistOption.name}]`,
description: "Run the dataform project's unit tests.",
positionalOptions: [projectDirMustExistOption],
options: [credentialsOption, timeoutOption, ...ProjectConfigOptions.allYargsOptions],
options: [credentialsOption, impersonateServiceAccountOption, timeoutOption, ...ProjectConfigOptions.allYargsOptions],
processFn: async argv => {
print("Compiling...\n");
const compiledGraph = await compile({
Expand All @@ -479,6 +488,10 @@ export function runCli() {
const readCredentials = credentials.read(
getCredentialsPath(argv[projectDirOption.name], argv[credentialsOption.name])
);
if (argv[impersonateServiceAccountOption.name]) {
(readCredentials as any).impersonateServiceAccount =
argv[impersonateServiceAccountOption.name];
}

if (!compiledGraph.tests.length) {
printError("No unit tests found.");
Expand Down Expand Up @@ -523,6 +536,7 @@ export function runCli() {
},
actionsOption,
credentialsOption,
impersonateServiceAccountOption,
fullRefreshOption,
includeDepsOption,
includeDependentsOption,
Expand Down Expand Up @@ -559,6 +573,10 @@ export function runCli() {
const readCredentials = credentials.read(
getCredentialsPath(argv[projectDirOption.name], argv[credentialsOption.name])
);
if (argv[impersonateServiceAccountOption.name]) {
(readCredentials as any).impersonateServiceAccount =
argv[impersonateServiceAccountOption.name];
}

const dbadapter = new BigQueryDbAdapter(readCredentials);
const executionGraph = await build(
Expand Down
1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
"estraverse": "^5.1.0",
"fs-extra": "^9.0.0",
"glob": "^10.5.0",
"google-auth-library": "~8.9.0",
"google-sql-syntax-ts": "^1.0.3",
"js-beautify": "^1.10.2",
"js-yaml": "^4.1.1",
Expand Down
1 change: 1 addition & 0 deletions packages/@dataform/cli/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ externals = [
"deepmerge",
"fs-extra",
"glob",
"google-auth-library",
"google-sql-syntax-ts",
"js-beautify",
"js-yaml",
Expand Down
2 changes: 2 additions & 0 deletions protos/profiles.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ message BigQuery {
string credentials = 3;
// Options are listed here: https://cloud.google.com/bigquery/docs/locations
string location = 4;
// Service account email to impersonate during authentication
string impersonate_service_account = 5;

reserved 2;
}
12 changes: 9 additions & 3 deletions yarn.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2076,7 +2076,7 @@ google-auth-library@^7.0.0, google-auth-library@^7.0.2:
jws "^4.0.0"
lru-cache "^6.0.0"

google-auth-library@^8.0.2:
google-auth-library@^8.0.2, google-auth-library@~8.9.0:
version "8.9.0"
resolved "https://registry.yarnpkg.com/google-auth-library/-/google-auth-library-8.9.0.tgz#15a271eb2ec35d43b81deb72211bd61b1ef14dd0"
integrity sha512-f7aQCJODJFmYWN6PeNKzgvy9LI2tYmXnzpNDHEjG5sDNPgGb2FXQyTBnXeSH+PAtpKESFD+LmHw3Ox3mN7e1Fg==
Expand Down Expand Up @@ -3961,8 +3961,7 @@ string_decoder@~1.1.1:
dependencies:
safe-buffer "~5.1.0"

"strip-ansi-cjs@npm:strip-ansi@^6.0.1", strip-ansi@^6.0.1:
name strip-ansi-cjs
"strip-ansi-cjs@npm:strip-ansi@^6.0.1":
version "6.0.1"
resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9"
integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==
Expand All @@ -3983,6 +3982,13 @@ strip-ansi@^6.0.0:
dependencies:
ansi-regex "^5.0.0"

strip-ansi@^6.0.1:
version "6.0.1"
resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9"
integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==
dependencies:
ansi-regex "^5.0.1"

strip-ansi@^7.0.1:
version "7.1.0"
resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-7.1.0.tgz#d5b6568ca689d8561370b0707685d22434faff45"
Expand Down