|
|
|
|
|
|
by jacAtSea
95 days ago
|
|
|
More context:
I was a 60% Data Engineer and 40% back eng engineer. The job ads now ask for cloud computing, I did on-perm with similar components but built in house. Step Functions
Glue
Athena
S3
RDS Databrick GCP
Snowflake I am looking at ways to 'modernize' 'cloudifly' myself This is chatgpt suggestions:
Step 3: Your FIRST project (this is the breakthrough)
Project: “Cloud Data Pipeline” (must-do)
What you build:
Pull financial data (API)
Store in S3
Process with Python
Schedule with Airflow
Load into warehouse (Redshift or PostgreSQL)
Architecture (simple)
API → Python → S3 → Airflow → DB → (optional dashboard) |
|
|