|
|
|
|
|
by itronitron
2811 days ago
|
|
Sounds like fun, you should probably look at Jupyter or Spark as a system to manage the data transformations and that allows team members to create and share scripts and workbooks. Develop the ETL process so that it just pulls data and writes it to your team's ideal form as flat files, then write another process that pushes that data where and how you want it (because that can change in six months), also develop automated processes for measuring/ensuring the quality of data being added to your system. |
|
Jupyter have used but was thinking to have the team settle on mainly using SAS code to built the ETL process since that is the language most of them are familiar with using. (even though I personally HATE writing SAS)