So far on my Northcoders Data Engineering bootcamp, all our coding has been done locally.  This means I was coding and developing software within my own computer, accessible only on that device. The only exception to this was when my coursemates and I collaborated and shared code with Git. 

On week 7 and 8, we took a deep dive into the Cloud, using AWS (Amazon Web Services).

Using cloud servers allows your work to be accessible from anywhere. Working locally is similar to saving a photo on your phone: you can only view the photo from this specific device. With cloud servers, it’s more like saving a photo to your cloud storage account, like iCloud or Google Cloud: you can see this photo from your laptop, tablet, or any other device connected to the account. 

It’s fair to say that the content of previous weeks, although challenging, was at least often familiar, as it was in Python and related to the Precourse that we had all completed to prepare for the bootcamp. AWS was a very different beast to try and tame. Almost every day, a new system was brought into play. We started with the web based dashboard, then moved onto a CLI (Command Line Interface), and into Terraform using code as infrastructure to get Amazon doing what we want consistently. 

We started off setting up an EC2, which is a virtual computer in the cloud, that we can set up to do almost anything. We ended up hosting a server running FastAPI and PG8000, similar to the local tasks earlier in the course. You start to become very accustomed to the login screen given the system loves to log you out numerous times during the day.

A vital part of the cloud is security, so we were taught and utilised AWS IAM to authorise or restrict access to various systems and files we store in S3 buckets. We then got working with AWS RDS, to set up remote databases to service our data.

Thankfully it wasn’t too long before we got back to VS Code. Using a system called Terraform ,we could program exactly what we want set up in AWS and push it to go live. I don’t think I’ve read as much documentation during this course as I have for Terraform!

During the weekend, I also experimented with setting up a VPC (Virtual Private Cloud) via Terraform that contained a publicly accessible EC2 and locally (within the VPC) accessible databases, to get a bit more practice.

We also learned AWS Lambda, which is a way to run serverless code functions. Using a system called Cloudwatch, we triggered actions, such as reading files periodically or alerting us to errors (that we usually caused…).

We then had a review day to look back on what we had learned over the past few weeks. The main sections covered were SQL, the cloud, and servers.

SQL is one of my weaker topics (hence plenty of revision), but thankfully it was only a few questions to establish that you know how to form search functions and get data from various tables.

For the cloud, the review mirrored a lot of what I had experimented on the weekend before using Terraform. There were some new parts to research during the day, but getting it integrated was not too difficult. I had a few messages asking about it, and thankfully had links to the correct documents at hand, saving a few seconds of googling.

Servers was another FastAPI system using a remote database. Oddly there was an issue with the latest version of Python, but a few easy workarounds were presented on Slack to keep everyone going. One error I had was simply solved with the usual ‘turn it off and on’ method (well, specifically, uninstalling and reinstalling FastAPI).

Thankfully everyone I know on the course made it through and we now get to put the cloud on hold as we turn to more advanced uses of SQL for our 9th week on the bootcamp!