The above diagram shows the various AWS services which were used. One of the constraints was to not have any extra servers which the infrastructure team needs to maintain, and also from a cost standpoint, the team decided to go with a serverless architecture.

AWS Step Functions were a natural choice to implement a serverless workflow as they offer built in error handling, execution event history, automatic scaling, high availability, pay per use and administrative security, thus allowing the developers to focus on the actual business logic and not on writing system code. There are some recently introduced features like “Map State” which allow for high levels of concurrent execution which fit in perfectly with what the team wanted.

Individual tasks were implemented using either AWS Lambdas or ECS Fargate tasks or a combination of both. This helped optimize the cost, as the ECS container shuts down after it is done processing for the day. Selenium was used for extracting information from third-party portal.

MySQL is used as the reporting store to store all the statistics and metrics related to runs and processing of positions.

The team utilized other features of AWS like Secrets Manager and AppConfig for storing application secrets and configuration.

AWS Event Bridge and Lambda were used to schedule the daily runs,

Google Drive Integration was also involved as the summary sheet required by HR was created on it. This was so as not to change the existing paradigms which HR were used to.

Google APIs were used to extract information from resumes in the form of a pdf and then rate them using proprietary algorithms.

The processing summary is then posted on the slack channel of the recruitment team that gives the required visibility on the status of the workflow and highlights if there have been any failures/exceptions in any stages of workflow execution.

Any failures during the course of execution are notified on email using the SNS notification. These notifications are directed to the technical support team that are responsible for checking and actioning these kinds of failures. For Observability of the system, Cloudwatch was used for tracing, logs and diagnostics.