Google Storage using cURL

I got around to troubleshooting a Python process running in Docker that had some permission problems accessing Google Storage Reproing inside Python with google-cloud-storage was a bit bothersome, and installing the gcloud cli on an Ubuntu-based Docker image takes too much effort on an ephemeral container (if I change stuff, I need to create a new container and I’ll have to reinstall it all over again). I was looking for something that is both easy to run, and doesn’t require me to modify the base container image (the build pipeline was annoying to mess with).

cURL to the rescue!

Getting an access token

This fetches a token from GKE’s metadata endpoint.

TOKEN="$(curl -H "Metadata-Flavor: Google" 'http://metadata/computeMetadata/v1/instance/service-accounts/default/token' -s | jq .access_token -r)"

Listing files in a bucket

curl -X GET -H "Authorization: Bearer $TOKEN" "https://storage.googleapis.com/storage/v1/b/$BUCKET_NAME/o/"

Getting a file

Note that FILENAME needs to be url-encoded, even the slashes that are used for “directories”.
Without ?alt=media, you’ll get metadata for the file

curl -X GET -H "Authorization: Bearer $TOKEN" "https://storage.googleapis.com/storage/v1/b/$BUCKET_NAME/o/$FILENAME?alt=media"

Putting a file

Again, FILENAME needs to be properly encoded

curl -X POST --data-binary @ACTUAL_FILE_PATH \
    -H "Authorization: Bearer $TOKEN" \
    "https://storage.googleapis.com/upload/storage/v1/b/$BUCKET_NAME/o?uploadType=media&name=$FILENAME"

Further reading

Copied most of these from https://cloud.google.com/storage/docs/downloading-objects et al