#422 – Twin Databases, Half the Hassle

We’ve recently upgraded the architecture of our data pipeline, Data Sanctum (DS). This upgrade simplifies our data flow, reduces costs, and provides near real-time data access to downstream services.

Previously, production Postgres from the app side generated daily backups, which were restored into a dedicated CloudSQL instance in the DS side. From there, data was dumped to Cloud Storage and loaded into BigQuery. This process introduced delays of 24-48 hours, added infrastructure costs, and consumed significant compute resources during backup restoration.

With the new architecture, DS/Airbyte reads directly from a read replica of the production CloudSQL instance from the app side. This new process eliminates the backup-and-restore step, streamlining data flow and enabling near real-time updates for consumers of DS like Looker Studio and HubSpot.

We intend to implement a cost-saving measure by downsizing the newly introduced read replica. Subsequently, we will focus on migrating to a real-time data processing pipeline next year.

This week we continued strong strength on number of weekly meetings run, reaching to the highest number in the last few weeks.

…met with DoD Platform One in San Antonio. We received praise for being ahead of schedule and meeting our milestones!

…conducted analysis on our integration adoption. ~35% of the paid teams have at least 1 integrations enabled and our top 3 integrated services are Jira, Google Calendar and Slack.

…laid out the roadmap for SOC-2 certification.

…enabled the team insights feature flag for some of our customers. We’d like to hear what you think about this new exciting feature we are about to launch!

…take a couple days off for the US Thanksgiving holidays. We here at Parabol are definitely thankful for all of our customers and we wish you a Happy Thanksgiving!