Rossolimo Project

present

Team Rossolimo successfully designed and deployed an ETL pipeline, which extracted data from Totesys' remote database and loaded it into an AWS S3 bucket using a Lambda function with Python code. The data underwent transformation and formatting via the pandas library into dataframes storing as parquet files in our S3 bucket by a subsequent Lambda function. In the final stage, the processed data was loaded into an external data warehouse through an additional Lambda function, utilising Postgres SQL with the Star Schema Warehouse Design.

Automated deployment of Infrastructure using Infrastructure as Code (IaC) tools such as Terraform allows scalability and consistent resource management. Workflow efficiency was also improved with Continuous Integration/Continuous Deployment (CI/CD) using Github Actions.

Each Lambda function was meticulously logged using AWS CloudWatch. The resulting data warehouse can be seamlessly queried using Tableau for in-depth data analysis and visualisation.

With test-driven development at the forefront of our operating practices, we designed a comprehensive testing suite for every aspect of our project. For example, making significant use of mocking and patching for cloud based technologies such as moto for boto3, achieving over 90% coverage of the codebase.

Developing strong teamwork and collaboration skills, we rotated the role of Scrum Master with our daily stand-ups and end of sprint retro. Through the Agile framework, we communicated openly and shared ownership of tasks visualised our GitHub Projects and Jira Kanban board. This transparent progress tracking ensured alignment with project objectives and deadlines