Refer to the architecture diagram above of a batch processing solution using Simple Queue Service (SQS) to
set up a message queue between EC2 instances which are used as batch processors Cloud Watch monitors
the number of Job requests (queued messages) and an Auto Scaling group adds or deletes batch servers
automatically based on parameters set in Cloud Watch alarms.
You can use this architecture to implement which of the following features in a cost effective and efficient
manner?
A.
Reduce the overall lime for executing jobs through parallel processing by allowing a busy EC2 instance that
receives a message to pass it to the next instance in a daisy-chain setup.
B.
Implement fault tolerance against EC2 instance failure since messages would remain in SQS and worn can
continue with recovery of EC2 instances implement fault tolerance against SQS failure by backing up
messages to S3.
C.
Implement message passing between EC2 instances within a batch by exchanging messages through
SQS.
D.
Coordinate number of EC2 instances with number of job requests automatically thus Improving cost
effectiveness.
E.
Handle high priority jobs before lower priority jobs by assigning a priority metadata field to SQS messages.
Explanation:
There are cases where a large number of batch jobs may need processing, and where the jobs may need to be
re-prioritized.
For example, one such case is one where there are differences between different levels of services for unpaid
users versus subscriber users (such as the time until publication) in services enabling, for example,
presentation files to be uploaded for publication from a web browser. When the user uploads a presentation file,
the conversion processes, for example, for publication are performed as batch processes on the system side,
and the file is published after the conversion. Is it then necessary to be able to assign the level of priority to the
batch processes for each type of subscriber?
Explanation of the Cloud Solution/Pattern
A queue is used in controlling batch jobs. The queue need only be provided with priority numbers. Job requests
are controlled by the queue, and the job requests in the queue are processed by a batch server. In Cloud
computing, a highly reliable queue is provided as a service, which you can use to structure a highly reliable
batch system with ease. You may prepare multiple queues depending on priority levels, with job requests put
into the queues depending on their priority levels, to apply prioritization to batch processes. The performance
(number) of batch servers corresponding to a queue must be in accordance with the priority level thereof.
Implementation
In AWS, the queue service is the Simple Queue Service (SQS). Multiple SQS queues may be prepared to
prepare queues for individual priority levels (with a priority queue and a secondary queue). Moreover, you may
also use the message Delayed Send function to delay process execution.
Use SQS to prepare multiple queues for the individual priority levels.
Place those processes to be executed immediately (job requests) in the high priority queue.
Prepare numbers of batch servers, for processing the job requests of the queues, depending on the priority
levels.
Queues have a message “Delayed Send” function. You can use this to delay the time for starting a process.
Configuration
Benefits
You can increase or decrease the number of servers for processing jobs to change automatically the
processing speeds of the priority queues and secondary queues.
You can handle performance and service requirements through merely increasing or decreasing the number of
EC2 instances used in job processing.
Even if an EC2 were to fail, the messages (jobs) would remain in the queue service, enabling processing to be
continued immediately upon recovery of the EC2 instance, producing a system that is robust to failure.Cautions
Depending on the balance between the number of EC2 instances for performing the processes and the number
of messages that are queued, there may be cases where processing in the secondary queue may be
completed first, so you need to monitor the processing speeds in the primary queue and the secondary queue.
D