Wrong default project ID is chosen when active configuration is not the default or service keys are used

Wrong gcloud command configuration is loaded

Problem

My ~/.config/gcloud/configurations/config_default contents:

[core]
account = ***REDACTED***@gmail.com
project = gcpnext2016-swast
[compute]
zone = us-central1-a
region = us-central1

My active configuration:

$ gcloud config list
Your active configuration is: [swast-test-flexible]
[app]
suppress_change_warning = true
[compute]
region = us-central1
zone = us-central1-b
[core]
account = swast@google.com
disable_usage_reporting = False
project = swast-test-flexible
[metrics]
command_name = gcloud.config.list

When I run the Bookshelf application, which uses Application Default Credentials locally with mvn -Plocal clean jetty:run-exploded and I connect to http://localhost:8080, I see data from my gcpnext2016-swast project, not the swast-test-flexible project like I expect.

Steps to reproduce

Run the Bookshelf application, which uses Application Default Credentials locally with mvn -Plocal clean jetty:run-exploded and I connect to http://localhost:8080
Add a few books.
Run gcloud init.
Create a new configuration instead of modifying the default, and select a different project this time.
Run the Bookshelf application again.
Notice that you still see the same data from the project selected in the default configuration, not the new configuration that you just activated.

Suspected root cause

gcloud-java uses default configuration rather than the currently selected one to figure out which project to send RPCs to. https://github.com/GoogleCloudPlatform/gcloud-java/blob/d03e5a7d80dc17d9012476557015e4f44be64687/gcloud-java-core/src/main/java/com/google/gcloud/ServiceOptions.java#L394

Project ID is not extracted from service keys

Problem

When I use a service key, with the Bookshelf app command

GOOGLE_APPLICATION_CREDENTIALS=$HOME/src/service-keys/swast-test-flexible.json mvn -Plocal clean jetty:run-exploded

I get an exception when my default gcloud command configuration has a different project ID than the service key.

Caused by: com.google.gcloud.datastore.DatastoreException: Unauthorized.

Steps to reproduce

Create a JSON service key for a project (a different project than what the default gcloud command configuration is set to).
Run the Bookshelf application using this service key.
GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json mvn -Plocal clean jetty:run-exploded
Observe an unexpected com.google.gcloud.datastore.DatastoreException.

Suspected root cause

The defaultProjectId is not extracting the project information from the service key, as I would expect it to. Note that a key is of the format:

{
  "type": "service_account",
  "project_id": "swast-test-flexible",  <-- gcloud-java should be using this, but it's not.
  "private_key_id": "abcdedfghijklmnop",
  "private_key": "-----BEGIN PRIVATE KEY-----\nBLAHblahBLAH\n-----END PRIVATE KEY-----\n",
  "client_email": "getting-started-java-laptop@swast-test-flexible.iam.gserviceaccount.com",
  "client_id": "1234567890",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/getting-started-java-laptop%40swast-test-flexible.iam.gserviceaccount.com"
}

Just a note here --

The Datastore project you want to talk to is not necessarily the project that your service account is attached to. So gcloud-java shouldn't grab your service account from the key.

However, it would be great if it could pull from the active project and not the default one from the cloud SDK.

@pcostell

@pcostell If I'm using a service account key, I would most definitely expect to be using that project by default. This is the way that Application Default Credentials are defined to work.

I would expect the DATASTORE_ENV_NAME or PROJECT_ENV_NAME to be able to override this, but using the active gcloud configuration over the service account key is the wrong order for Application Default Credentials.

CC @jerjou @lesv

I'm not sure that's true. There is nothing in the ADC site which says it also sets your project ID that you want to connect to, only that it sets up which one you want to connect as.

@tswast

Thanks @tswast for the thorough report (I wish all issues where this detailed). I will look into this. Let me start with a couple of thoughts:

Wrong gcloud command configuration is loaded

You are right this must be fixed by loading the project ID from the active config.

Project ID is not extracted from service keys

I believe that the project_id field is a recent addition to the JSON credentials (I have ~4 months old JSON credentials without that field). If this is there to last I think we should use it (this should be notified to other gcloud-* maintainers). /cc @jgeewax @aozarov