The past few weeks I developed and deployed a cloud function that is supposed to get called only by authorized users/service accounts and the truth is that the documentation I found wasn’t really helpful. — First I created a service account, gave it roles/cloudfunctions.invoker permission. If you’re dealing with an extracted service account the code is pretty simple from google.oauth2 import service_account from google.auth.transport.requests import AuthorizedSession

In my previous post, I explained how to stream data from Salesforce to PubSub in real-time. The next logical step would be to store the data somewhere, right? One option could be, for example, to batch the data, and write them to files in GCS. That’s a good start and…

TL/DR The working implementation is on Third Attempt Intro A couple of months ago when we started building our data lake, one of the requirements was to try and get real-time data in. Most of our external and internal sources supported that, I had a bit more knowledge on Salesforce so I…

Ok, so, we’ve written our Dataflow Template with Python, now what? We want to schedule it to run daily and we’re going to use Airflow for that. The first thing we want, for security reasons, is to keep service accounts separate. In the previous post, we’ve created a service account…