Please explain Amazon SQS (and queueing in general)

Question

For my web app I will need a separate instance of EC2 to process CPU-intensive things, and things that can be queued so they don't encumber the web serving instance, like image resizing, sending email....

When you create an AWS Elastic Beanstalk instance it asks you to choose between a "web" or "worker" environment. From my understanding it's in the worker environment that I will process those kind of tasks.

What's the role of SQS in this context? I read it is only about sending "messages" and this kind of stuff, but how will I get my image resized with a "message"?

Should I create specific, distinct code for the worker instance, to handle image resizing, and then use SQS to order it to process the image? Then can SQS pass image from web instance, to worker instance? I completely miss the main concept.

Answer 1

A queuing service (such as Amazon SQS) is used to store messages for later retrieval. Think of it like a TODO list -- you add items to the queue, and later you retrieve an item from the queue and take action on the item.

For example, let's say users upload images to a website and you wish to generate thumbnails from those images. Your website will store the image in Amazon S3, then push a message into an SQS queue. The message would include a reference to the image in S3 and details about the user.

Then, your Elastic Beanstalk worker will request a message from the queue and process the image. It would retrieve the image from S3, resize it, store it in an S3 bucket, then perhaps email the user to say that the job is finished. The worker then exits, and Elastic Beanstalk will trigger a new worker to read the next message from the queue and do it all again.

So, yes -- you will create the worker code. Elastic Beanstalk will trigger the worker with the SQS message. SQS itself doesn't trigger anything -- it is actually Elastic Beanstalk that retrieves the message and runs the worker.

See: Elastic Beanstalk worker environments