Tote-ally Awesome Final Projectpresent
Toteally Awesome Demo Video
Database Solutions for the Modern Tote Lover
Our project extracts data from a tote bag store's database, transforms it into a structured format, and stores it in a data warehouse. A dashboard accesses the data warehouse to allow insights to be gleamed from sales data.
The project infrastructure is built on AWS and provisioned using Terraform an IoC language. Lambda functions are used in AWS running python code. We have used pg8000 to integrate postgreSQL into our code.
Our project pipeline takes data from a postgreSQL database and puts it into an 'ingestion' S3 bucket. From here, pandas transforms the data into a star schema format and uploads it to a 'processed' S3 bucket. A third lambda function uploads it to our data warehouse hosted in RDS.
Power BI is then used to create a dashboard for the business to analyse their sales.
The Team
Joe Christian
Mark Bradley
Tasneem Aiar
Erin Monaghan
Ben Morgan
Technologies
We used: Python (3.10) AWS (S3, Lambda, Step Functions, CloudWatch, Secrets Manager, RDS) Terraform (Infrastructure as Code) PostgreSQL GitHub Actions (CI/CD)
Python was used because it is a universal and well supported language with lots of modules and plugins available.
AWS was chosen as it is a major player in the cloud computing industry.
Terraform is open source and well integrated with AWS, so was an ideal candidate for our Infrastructure as Code provisioning.
PostgreSQL is another open source language well suited to handling large databases.
GitHub was used for our repository as it is a very popular and well supported option.
Challenges Faced
We found it challenging to merge lots of languages and systems together to create one coherent piece of software for our end user. Deciding what information each function should pass to the next required many iterations of our design.