Bootstrapping a Real-Time Analytics Pipeline Without Breaking the Bank

Why?

After running our LMS for over a year, we had collected a significant volume of user engagement data covering course progress, test submissions, module completion patterns, and more. As we prepared to raise our next round of funding, this data became crucial not just for internal understanding but for telling a compelling story to VCs. Investors needed to see momentum, traction, and retention. We needed a way to surface those insights through dashboards.

But we hit a critical blocker: getting clean, up-to-date data from our Supabase PostgreSQL database into visualization tools like Google Looker Studio was either unreliable, too slow, or too expensive. We couldn't afford to pay for enterprise-grade ETL tools or BigQuery connectors, many of which charged hundreds per month. Yet we needed to present a picture of growth, fast, confidently, and cost-efficiently.

Challenges

Direct connections between Supabase and tools like Looker Studio were unstable. We often had to manually trigger refreshes or re-authenticate data sources. Even when connected, refresh rates were slow, sometimes taking several minutes or failing entirely, making our dashboards feel outdated or frozen. Most third-party sync tools that offered “real-time” updates came with steep SaaS pricing that didn't make sense for our cash-constrained, pre-funding startup. We needed something that gave us near real-time data with full control and ideally, no additional monthly spend.

Solution

To eliminate our dependency on costly ETL tools and regain control over our analytics pipeline, I designed and built a custom Express.js server that acted as a lightweight connector between our Supabase database and Google BigQuery. This server was responsible for querying the latest data from our LMS including user engagement, test activity, progress tracking and transforming it into clean, structured formats that could be uploaded into BigQuery tables.

Instead of aiming for full real-time sync (which would have been expensive and overkill for our use case), I scheduled the Express server to run on a 15-30 minute cron job. This allowed our dashboards to stay fresh enough for business decisions, investor updates, and product health tracking without ever feeling stale. The BigQuery tables then acted as a single source of truth for our BI stack, powering dynamic visualizations inside Looker Studio and similar tools.

To avoid cloud hosting costs and retain full control, I containerized the Express server using Docker and deployed it on a spare company laptop. This "bare-metal server" ran Ubuntu LTS and was hardened with production-ready security:

1. A UFW firewall to control network access

2. Root login and password-based login disabled

3. SSH key-based access only

4. A dedicated non-root user for managing Docker and deployment scripts

The entire system was built for resilience and cost-efficiency, while staying flexible enough to adapt as we scaled. And importantly, it worked silently in the background giving our leadership team the data they needed, when they needed it, without worrying about refresh failures or connector limitations.

Impact

By removing the dependency on external connectors and expensive ETL pipelines, we saved approximately $500-$700 per month in recurring SaaS costs. More importantly, we enabled fast, near-real-time dashboards that helped our leadership team understand user activity, product adoption, and retention patterns with clarity and confidence. The data became a powerful narrative asset, it supported investor conversations, shaped product roadmaps, and helped drive internal decisions.

The dashboards were no longer just pretty charts, they were living proof of our product's traction. Time-to-insight was reduced by more than 70%, and we no longer had to rely on slow manual exports or flaky refreshes. In the end, this solution played a crucial role in showing that we were a data-literate, execution-focused team capable of building smart, scalable systems without bloating our costs.