Team Rannoch's Data Projectpresent
We know what you sold last Summer.
Create a data platform that extracts data from an operational database (and potentially other sources), archives it in a data lake, and makes it available in a remodelled OLAP data warehouse, using an ETL format.
The Team
Luna Birtles
Ali Anvari
Stephen Molano-James
James Ault
Yaroslav Davydchuk
Technologies
![Technologies section image](https://images.prismic.io/northcoders/65e5ad51676dc480aae090ea_TeamRannoch.png?auto=compress%2Cformat&fit=max&q=60&w=800)
We used: AWS (Eventbridge, Lambda, S3, SNS, Cloudwatch), Terraform, GitHub Actions, Python (pandas, pytest, boto3, moto), PostgreSQL
They were the most appropriate technologies available for the project.
Challenges Faced
We faced challenges figuring out approprate properties for our lambda functions (e.g. layers, memory and timeout), ensuring NaN representation from pandas was handled in the PostgreSQL database, ensuring that updates to the lambda code were deployed properly with Terraform, and sequencing our functions correctly to overcome issues with foreigns keys when loading data in to the OLAP database. All of which we overcame.