Fenor Final Projectpresent
Fenor Demo Video
We wrangle data so you don’t have to.
This project is an ETL (Extract, Transform, Load) pipeline that successfully automates data movement from a live PostgreSQL relational database (ToteSys) into a data warehouse optimised for analytics. The pipeline is designed for scalability, reliability and automation using AWS services and Infrastructure as Code.
This project has been an opportunity for our team to showcase our knowledge of Python, SQL, database modelling, AWS, good DevOps practices and Agile working, amongst other technical and non-technical skills learnt over our time on the Northcoders Data Engineering Bootcamp.
The project uses AWS cloud services for storage (S3), computing (lambda), and monitoring (Cloudwatch), allowing for security and scalability. Terraform is used to store and deploy the project as Infrastructure as Code, allowing for reliable and repeatable deployment.
A GitHub Actions CI/CD pipeline is used alongside rigourus automated testing and compliance checks to ensure the quality and consistency of our codebase.
We adhered to the principles of TDD (Test driven development) throughout our time working on the project using Pytest, and practiced pair programming to ensure that each piece of work met the highest possible standards. Our team had a wide range of expertise and knowledge, and we ensured that each team member worked on various elements of the project to encourage knowledge sharing and collaboration.
Our working practices included daily morning standup meetings, task-ticketing through a Trello Kanban board, and regular code reviews and group reflections.
Our team members are:
- Brendan Corbett | https://github.com/BrendanC8450
- Connor Creed | https://github.com/creedy293 | https://www.linkedin.com/in/c-creed-942132231/
- Fabio Greenwood | https://github.com/FabioGreenwood | https://www.linkedin.com/in/fabio-greenwood-b0832676/
- Tor Satherley | https://github.com/TorSatherley | https://www.linkedin.com/in/tor-satherley-95a789265/
- Vincent Toor-Azorin | https://github.com/Armatoor147 | https://www.linkedin.com/in/vincent-toor-azorin/
The Team
Connor Creed
Tor Satherley
Fabio Greenwood
Vincent Toor-Azorin
Brendan Corbett
Technologies
We used: Python, AWS Lambda, SQL, cloud watch, Terraform, github actions.
Effective tools for creating a cloud ETL solution
Challenges Faced
Learning to work cooperatively, Learning new technologies on the job and communicating complex concepts during meetings effectively.