1-888-317-7920 info@2ndwatch.com

Migrating scheduled workloads into AWS Lambda

A very common operational requirement is to run workloads at perpetually scheduled points in time.  This has historically been done by using cron in the UNIX world or using Task Scheduler in the Windows world.  I have worked for more than one company in the past where we had a specific piece of infrastructure built exclusively to handle this functionality.  Because of the criticality of the workloads, this usually involved some form of a cluster and redundant hardware.  If you were to analyze the utilization of those systems, you would see that there were only small spikes at various times throughout the day, but by and large those systems were sitting idle.  What a waste!

AWS introduced their Lambda service at Re:Invent in 2014.  Lambda is described as a “Zero administration compute service that runs developers’ code in response to events and automatically manages the compute resources for them, making it easy to build applications that respond quickly to new information”.

They key to driving Lambda functions is the word “events” as it is an event-driven service.  When Lambda was launched there was a small handful of event types (updates to S3 objects, Kinesis streams, or DynamoDB tables) that could initiate the running of a Lambda function.  However, time-based scheduling was not one of them.  Also, at Lambda’s launch functions had to be written using node.js (though you could access the functionality of the underlying Amazon Linux operating system).

Since AWS announced the release of Lambda I have been anxiously awaiting the day when they would add Python support and an event source that would allow you to trigger Lambda functions using scheduled points in time.  A few months back they gave me an early Christmas present and did just that by adding python support AND the “Scheduled Event” event source type!  In addition to having the ability to write functions in python, this means you can now write Lambda functions to perform all of the work you used to do in cron or Task Scheduler and save yourself the money wasted in running idle infrastructure and the support costs associated with that.

Here is what creating a Lambda event source using the new Scheduled Event type looks like:

Migrating Workloads to AWS Lambda

Now all you have to do is start migrating your functions from their current form into either node.js or python Lambda functions.  If you are like me and already writing most of those tasks in python using modular programming techniques, this migration will be relatively simple.

Contact 2nd Watch for questions.

-Ryan Kennedy, Sr Cloud Architect

Facebooktwittergoogle_pluslinkedinmailrss