Integrate Mule ESB with Amazon S3

Overview

MuleSoft provides the most widely used integration platform to connect any application, data service or API, across the cloud and on-premise continuum. Amazon S3 is cloud-based data-storage infrastructure that is accessible to the user which is programmable via a MuleSoft Amazon S3 connector. The Amazon S3 Mule connector allows developers to access any amount of data, at any time, from anywhere on the web. With connectivity to the Amazon S3 API, users can interface Amazon S3 to store objects, download and use data with other AWS services, and build applications that call for internet storage - all from within MuleSoft Anypoint Platform.

Instant access to the Amazon S3 API enables businesses to create seamless integrations between Amazon S3 and other databases, CMS applications such as Drupal, and CRM applications such as Salesforce.

Prerequisites

1. Amazon Account
2. Amazon Access and Secret keys
3. Mule Anypoint Amazon Connector

Amazon Account

Creating Amazon S3 Bucket

Before integrating S3, a S3 bucket has to be created. Every object in Amazon S3 is stored in this bucket. Following are the steps to create a bucket:

• Enter a bucket name in the ‘Bucket Name’ field
• Select the region
• If  we need to set up-logging, then click on ‘Set Up Logging’, Else, go to ‘Create‘ and create the bucket
• When Amazon S3 successfully creates the bucket, console will display the properties of the empty bucket
  
  

Amazon Access and Secret Key

Access key and secret key can be retrieved  from the Amazon account page

Mule Anypoint Amazon S3 Connector

Configure Mule Anypoint Amazon S3 connector with the following settings -

Configuration: S3 connector configured using the config element) This element must be placed out of flow and at the root of the Mule application. We can create as many configurations deemed necessary as long as each carries its own name.

Connection Pool: S3 connector offers automatic connection management via the use of a connection pool. This pool will act a storage mechanism for all the connections that are in-use by the user of this connector.

Prior to execution of a processor, the connector will attempt to lookup an already established connection and if one doesn't exist it will create one. That lookup mechanism is done in the connection pool via the use of connection variables declared as keys.

Reconnection Strategy: Reconnection Strategies specify on how a connector behaves when the connection fails. We can control the attempts by Mule to reconnect using several criteria

• Type of exception
• Number and frequency of reconnection attempts
• Notifications generated

With a reconnection strategy, the behaviour of a failed connection can be controlled in a much better way by configuring it, for example, to re-attempt the connection only once every 15 minutes, and to give up after 30 attempts. An automatic notification can be sent to the IT administrator whenever this reconnection strategy goes into effect. A strategy can also be defined which attempts to reconnect only during business hours. Such a setting can prove useful if the server is frequently shut down for maintenance.

Mule ESB – The best way to Integrate Amazon S3

The below Mule application is used to read the object (.jpeg image) from HTTP outbound component using the absolute URL and upload the same object under Amazon S3 bucket using Mule Anypoint Amazon S3 connector.

Steps to be followed for integrating 'Amazon S3' with MuleSoft Applications:

1. Install the Amazon S3 Connector in Anypoint Studio (3.5 and above) http://repository.mulesoft.org/connectors/releases/3.5
2. Create a new Anypoint Studio Project and Flow
3. Before using the Amazon S3 Connector in the Mule Flows, create a global element for Amazon S3 configuration which can be reused in all other flows across the Mule project - wherever the objects are created & deleted to & from Amazon S3 cloud server.
4. Configure the Amazon S3 connector by providing the following information for the global element
   • Access Key
   • Secret Key
5. Use HTTP outbound endpoint to pass the object location to get the object and upload to the Amazon S3 storage server
6. Configure the Amazon S3 endpoint by providing the following to create the object under Amazon S3 bucket
   • Link to the ‘global Connector Configuration’
   • Operation
   • Bucket Name
   • ACL (Access level)

7. Run the Mule Application and trigger the URL (using browser)

http://localhost:8081/createObject

8. Check the Amazon S3 bucket, the object (.jpeg object) will be created under the bucket.

Architecture to Integrate Amazon S3 using Mule Enterprises ESB

Benefits

• No point-to-point integration required
• Seamless integration running in the background
• Quick data synchronisation between Amazon S3 with an on-premise and cloud based application
• Facilitate applications with Amazon S3 capacities for large volume data storage
• Stores application data and can be rolled back during disaster recovery
• Bi-directional data communication between applications and Amazon S3
• Highly scalable, it secures a solution for backing up and archiving the critical data
• Send event notifications when objects are uploaded to Amazon S3
• Access any amount of data, any time and from anywhere on the web.

Author: Mr. Sanjeet Pandey is a Masters of Computer Application (MCA) graduate having 4+ years of experience as a Technical Specialist with Coforge. He has varied experience in ESB’s like Mule, JCaps.  Sanjeet loves dreaming up new ideas, movies & music, photography & travelling and developing web projects. To know more about Sanjeet log on to www.sanjeetpandey.com

If you would like to find out more about how Systems Integration could help you make the most out of your current infrastructure while enabling you to open your digital horizons, do give us a call at +44 (0)203 475 7980 or email us at Salesforce@coforge.com

Other useful links:

Anypoint Microsoft Azure Connector Developed by Coforge

Mule 2 to Mule 3 Migration Case Study

Case Studies