This article from Google provides a good overview of BigQuery for a data warehouse practitioner. However, organizing data is not covered in much detail. This blog is focusing solely on this part — how to organize data in BigQuery for effective and compliant management across multiple teams in your organization. Each organization is different and Convey’s law is definitely applicable in data modeling. Still, we think this could be a starting point for anyone to extend on.
When we started with BigQuery, we were focusing mostly on “how to get work done” and not much on effective data management. But pretty soon, we ran into the following problems.
Now let’s talk about how we tackled these issues. To understand it better, first I will present the general data org structure we follow. Then we will dive into BigQuery specific practices.
Click here to read.