Amazon S3 - Get Objects
Declaration
<AMAWSS3 OVERWRITE="yes/no" ONLYIFEXIST="yes/no" MATCHCASE="yes/no" CHECKSUM="yes/no" EXCLUDE="text" RE="yes/no" ACCESSKEY="text" SECRETKEY="text (encrypted)" PROTOCOL="text (options)" MAXERRORRETRY="number" SERVICEURL="text" PROXYHOST="text" PROXYPORT="number" PROXYUSER="text" PROXYPWD="text (options)" BUCKETNAME="text" KEYNAME="text" RESULTVARIABLE="text" VERSION="number" RESULTDATASET="text"><HEADER NAME="text" VALUE="text" /></AMAWSS3>
Description: Retrieves a single object or multiple objects contained in an S3 bucket. You can store an object into a local file or store an object's contents into a variable. To specify more than one object, use wildcard characters (* or ?). To specify multiple objects or wildcard masks, separate them with a pipe symbol (|). Example: *.txt|*.bak
Practical Usage
Used to download one or more S3 objects. Note that in order for this activity function properly, you must have READ access to the object(s) you wish to retrieve.
Connection Parameters
Property |
Type |
Required |
Default |
Markup |
Description |
---|---|---|---|---|---|
Connection |
|
|
|
|
Indicates where AWS user credentials and preferences should originate from. This is a design mode parameter used only during task construction and configuration, thus, comprises no markup. The available options are:
|
Session |
Text |
Yes if connection is session-based |
EC2Session1 |
SESSION="S3Session1" |
The name of an existing session to attach this activity to. This parameter is active only if the Connection parameter is set to Session. The default session name is 'S3Session1'. |
Access key |
Text |
Yes if connection is host-based |
(Empty) |
ACCESSKEY= "022QF06E7MXBSH9DHM02" |
A 20-character alphanumeric string that uniquely identifies the owner of the AWS service account, similar to a username. This key along with a corresponding secret access key forms a secure information set that AWS uses to confirm a valid user's identity. This parameter is active only if the Connection parameter is set to Host. |
Secret Access key |
Text |
Yes if connection is host-based |
(Empty) |
SECRETKEY= "kWcrlUX5JEDGM/LtmEENI/ aVmYvHNif5zB+d9+ct" |
A 40-character string that serves the role as password to access the AWS service account. This along with an associated access key forms a secure information set that EC2 uses to confirm a valid user's identity. This parameter is active only if the Connection parameter is set to Host. |
Protocol |
Text (options) |
No |
HTTP |
PROTOCOL="HTTPS" |
The protocol required. The available options are:
|
User agent |
Text |
No |
AutoMate |
USERAGENT="AutoMate" |
The name of the client or application initiating requests to AWS, which in this case, is AutoMate. This parameter's default value is 'AutoMate'. |
Service URL |
Text |
No |
(Empty) |
SERVICEURL= "https://s3.eu-west-1.amazonaws.com" |
The URL that provides the service endpoint. To make the service call to a different region, you can pass the region-specific endpoint URL. For example, entering https://s3.us-west-1.amazonaws.com points to US West (Northern California) region. A complete list of S3 regions, along with associated endpoints and valid protocols can be found below under S3 Endpoints and Regions. |
Maximum retry on error |
Number |
No |
(Empty) |
MAXERRORRETRY="4" |
The total amount of times this activity should retry its request to the server before returning an error. Network components can generate errors anytime in the life of a request, thus, implementing retries can increase reliability. |
Proxy host |
Text |
No |
(Empty) |
PROXYHOST="proxy.host.com" |
The host name (e.g., server.domain.com) or IP address (e.g., xxx.xxx.xxx.xxx) of the proxy server to use when connecting to AWS. |
Proxy port |
Number |
No |
(Empty) |
PROXYPORT="1028" |
The port that should be used to connect to the proxy server. |
Proxy username |
Text |
No |
(Empty) |
PROXYUSER="username" |
The username that should be used to authenticate connection with the proxy server (if required). |
Proxy password |
Text |
No |
(Empty) |
PROXYPWD="encrypted" |
The password that should be used to authenticate connection with the proxy server (if required). |
Object Parameters
Property |
Type |
Required |
Default |
Markup |
Description |
---|---|---|---|---|---|
Bucket Name |
Text |
Yes |
(Empty) |
BUCKETNAME="MyBucket" |
Indicates the name of the bucket in which to get object(s) from. |
Key Name |
Text |
Yes |
(Empty) |
|
Specifies the key name of the object (s) to get. A key is the unique identifier for an object within a bucket. To specify more than one object, use wildcard characters (* or ?). To specify multiple objects or wildcard masks, separate them with a pipe symbol (|). Example: *.txt|*.bak |
Store object into local file |
Text |
No |
(Empty) |
|
Specifies the destination folder and (optionally) the file name(s) in which to store the retrieved S3 object(s). This can be a fully qualified path or a file name. Folders that do not exist will be automatically created at runtime. |
Store object data into variable |
Text |
No |
(Empty) |
RESULTVARIABLE="myVar" |
The name of an existing variable in which to populate with the contents of the retrieved object. This property will only work with single object retrieval. |
Create and populate dataset with S3 object information |
Text |
No |
(Empty) |
RESULTDATASET="PublicReadWrite" |
Specifies the name of the dataset to create and populate with information regarding the retrieved object(s). This activity creates and populates a dataset containing a specific set of fields in addition to the standard dataset fields. More information can be gathered below under Datasets. |
Description tab - A custom description can be provided on the Description tab to convey additional information or share special notes about a task step.
Error Causes tab - Specify how this step should behave upon the occurrence of an error. (Refer to Task Builder > Error Causes Tab for details.)
On Error tab - Specify what AWE should do if this step encounters an error as defined on the Error Causes tab. (Refer to Task Builder > On Error Tab for details.)
S3 Endpoints and Regions
This table contains a complete list of Amazon Simple Storage Service endpoints, along with their corresponding regions, supported protocols and location constraints.
Endpoint |
Region |
Protocol |
Location Constraints |
---|---|---|---|
s3.amazonaws.com |
US Standard * |
HTTP and HTTPS |
(none required) |
s3.us-west-2.amazonaws.com |
US West (Oregon) Region |
HTTP and HTTPS |
us-west-2 |
s3.us-west-1.amazonaws.com |
US West (Northern California) Region |
HTTP and HTTPS |
us-west-1 |
s3.eu-west-1.amazonaws.com |
EU (Ireland) Region |
HTTP and HTTPS |
EU |
s3.ap-southeast-1.amazonaws.com |
Asia Pacific (Singapore) Region |
HTTP and HTTPS |
ap-southeast-1 |
s3.ap-southeast-2.amazonaws.com |
Asia Pacific (Sydney) Region |
HTTP and HTTPS |
ap-southeast-2 |
s3.ap-northeast-1.amazonaws.com |
Asia Pacific (Tokyo) Region |
HTTP and HTTPS |
ap-northeast-1 |
s3.sa-east-1.amazonaws.com |
South America (Sao Paulo) Region |
HTTP and HTTPS |
sa-east-1 |
* The US Standard region automatically routes requests to facilities in Northern Virginia or the Pacific Northwest using network maps.
Datasets
A dataset is a multiple column, multiple row container object. This activity creates and populates a dataset containing a specific set of fields in addition to the standard dataset fields.
Name |
Type |
Return Value |
---|---|---|
theDataset.KeyName |
Text |
The Key Name of the retrieved object. |
theDataset.AmazonID2 |
Text |
A special token associated to the retrieved object that helps Amazon Web Service (AWS) troubleshoot problems. |
theDataset.ETag |
Text |
The object's entity tag. This is a hash of the object that can be used to do conditional operations. The ETag only reflects changes to the contents of an object, not its metadata. |
theDataset.Size |
Number |
The size of the object, in bytes. |
theDataset.ContentType |
Text |
A standard MIME type describing the format of the object (e.g., text/plain). |
theDataset.VersionID |
Text |
The version ID of the retrieved object if it has a unique version ID. NOTE: To get the version ID, Versioning must be enabled using the Set Bucket Versioning activity. |
Example
Description: Get object "testFile.txt" from bucket "myBucket". Store it into file "C:\Temp\testFile.txt". Create and populate dataset "theDataset" with S3 object information. Use "mySession" S3 session.
<AMAWSS3 BUCKETNAME="myBucket" KEYNAME="testFile.txt" FILE="C:\Temp\testFile.txt" RESULTDATASET="theDataset" SESSION="mySession" />