Hacker News new | ask | show | jobs
by spookylettuce 1846 days ago
I tend to utilize bigquery's external connections and scheduled queries to periodically clone my "hot" PG data to an BQ table for long term storage and analysis.

It seems so much easier to go PG=>BQ than the other way around.

4 comments

Is the idea something like a scheduled:

   INSERT INTO bq_table SELECT * FROM EXTERNAL_QUERY('');
I'm guessing you're on the hook for keeping the schema up to date with the Postgres schema.
I also use the external data connector, the problem is when you have two projects in different locations. Then I go for psycopg2 and uploat it to bigquery using bigquery magic lines
Same here. We previously used Stitch until they dropped the free tier.
I second federated table connector (and scheduled queries).