Amazon S3 - List versions
Declaration
<AMAWSS3 ACTIVITY="list_versions" ACCESSKEY="text" SECRETKEY="text (encrypted)" PROTOCOL="text (options)" USERAGENT="text" MAXERRORRETRY="number" SERVICEURL="text" PROXYHOST="text" PROXYPORT="number" PROXYUSER="text" PROXYPWD="text (encrypted)" BUCKETNAME="text" RESULTDATASET="text" PREFIX="text" MARKER="text" MAXKEYS="number" DELIMITER="text" VERSIONMARKER="text" NEXTMARKER="text" NEXTVERSIONID="text"><HEADER NAME="text" VALUE="text" /></AMAWSS3>
Description
Retrieves the metadata about all of the versions of objects in a bucket and populates a dataset and/or variable with results.
IMPORTANT: The AWS S3 activities
are performed using Amazon's Simple Storage Service engine, therefore,
launching and operating Amazon S3 through Automate requires a
valid Access Key ID and Secret Access Key.
Practical usage
Used to get version information about a list of objects contained in a bucket.
Parameters
Connection
| Property | Type | Required | Default | Markup | Description |
|---|---|---|---|---|---|
| Connection | --- | --- | --- | --- | Indicates
where AWS user credentials and preferences should originate from.
This is a design mode parameter used only during task construction
and configuration, thus, comprises no markup. The available options
are:
|
Connection - Session
| Property | Type | Required | Default | Markup | Description |
|---|---|---|---|---|---|
| Session | Text | Yes, if Connection is set to Session | EC2Session1 | SESSION="S3Session1" | The name of an existing session to attach this activity to. This parameter is active only if the Connection parameter is set to Session. The default session name is S3Session1. |
Connection - Host > Credentials
| Property | Type | Required | Default | Markup | Description |
|---|---|---|---|---|---|
| Access key | Text | Yes, if Connection is set to Host | (Empty) | ACCESSKEY="022QF06E7MXBSH9DHM02" | A 20-character alphanumeric string that uniquely identifies the owner of the AWS service account, similar to a username. This key along with a corresponding secret access key forms a secure information set that AWS uses to confirm a valid user's identity. This parameter is active only if the Connection parameter is set to Host. |
| Secret access key | Text | Yes, if Connection is set to Host | (Empty) | SECRETKEY="text (encrypted)" | A 40-character string that serves the role as password to access the AWS service account. This along with an associated access key forms a secure information set that EC2 uses to confirm a valid user's identity. This parameter is active only if the Connection parameter is set to Host. |
Connection - Host > Advanced
| Property | Type | Required | Default | Markup | Description |
|---|---|---|---|---|---|
| Protocol | Text (options) | No | HTTP | PROTOCOL="HTTPS" | The
protocol required. The available options are:
|
| User agent | Text | No | Automate | USERAGENT="Automate" | The name of the client or application initiating requests to AWS, which in this case, is Automate. This parameter's default value is Automate. |
| Service URL | Text | No | (Empty) | SERVICEURL="https://s3.eu-west-1.amazonaws.com" | The URL that provides the service endpoint. To make the service call to a different region, you can pass the region-specific endpoint URL. For example, entering https://s3.us-west-1.amazonaws.com points to US West (Northern California) region. A complete list of S3 regions, along with associated endpoints and valid protocols can be found below under Amazon S3 regions and endpoints . |
| Maximum number of retries on error | Number | No | (Empty) | MAXERRORRETRY="4" | The total amount of times this activity should retry its request to the server before returning an error. Network components can generate errors anytime in the life of a request, thus, implementing retries can increase reliability. |
| Proxy host | Text | No | (Empty) | PROXYHOST="proxy.host.com" | The hostname (for example, server.domain.com) or IP address (for example, 192.168.0.100) of the proxy server to use when connecting to AWS. |
| Proxy port | Number | No | (Empty) | PROXYPORT="1028" | The port that should be used to connect to the proxy server. |
| Proxy username | Text | No | (Empty) | PROXYUSER="username" | The username that should be used to authenticate connection with the proxy server (if required). |
| Proxy password | Text | No | (Empty) | PROXYPWD="encrypted" | The password that should be used to authenticate connection with the proxy server (if required). |
Object
| Property | Type | Required | Default | Markup | Description |
|---|---|---|---|---|---|
| Bucket name | Text | Yes | (Empty) | BUCKETNAME="myBucket" | The name of the bucket in which to retrieve |
| Create and populate dataset with S3 object information | Text | Yes | (Empty) | RESULTDATASET="myDataset" | Indicates the name of the dataset in which to create and populate with the S3 object information. See Datasets for more information. |
| Populate variable with next marker | Text | No | (Empty) | NEXTMARKER="Var1" | The name of an existing variable in which to populate with the next marker. Ideally used within a loop block to retrieve markers one after another during each iteration. (ideally used within a loop block). |
| Populate variable with next version ID marker | Text | Yes | (Empty) | NEXTVERSIONID="Var2" | Specifies an existing variable in which to populate with the next version ID marker. Ideally used within a loop block to retrieve version ID markers one after another during each iteration. |
Advanced
Each Amazon S3 object has a set of key-value pairs with which it is associated called Headers or Metadata. Metadata provides important details about an object, such as file name, type, date of creation/modification etc. There are two kinds of metadata in S3; system metadata, and user metadata. System metadata is used and processed by Amazon S3. User metadata (also known as custom header) is specified by you, the user. Amazon S3 simply stores it and passes it back to you upon request.
Automate lets you to store your personal information as custom headers or user metadata like name, company name, and phone numbers etc, so that you can distinguish specific files. Using this option, you can add new custom header/user metadata to existing S3 objects, edit default S3 metadata on a bucket or store/upload new objects with custom header or metadata.
| Property | Type | Required | Default | Markup | Description |
|---|---|---|---|---|---|
| Prefix | Text | No | (Empty) | PREFIX='photos/2006/" | Restricts the response to only contain results that begin with the specified prefix. You can use prefixes to separate a bucket into different groupings of keys. |
| Marker | Text | No | (Empty) | MARKER="6" | Specifies the key in the bucket that you want to start listing from. This enables pagination of large result sets by restricting the response to only contain results that occur alphabetically after the value of the marker. This parameter is optional. |
| Maximum Keys | Text | No | (Empty) | MAXKEYS="10" | Limits the number of keys returned in response to your query. The response might contain fewer keys, but will never contain more. |
| Delimiter | Text | No | (Empty) | DELIMITER="text" | A character that you specify to group keys. All keys that contain the same string between the Prefix and the first occurrence of the delimiter are grouped under a single result element. These groups are counted as one result against Maximum keys limitation. |
| Version ID marker | Text | No | (Empty) | VERSIONMARKER="text" | Specifies the object version you want to start listing from. |
| Name | Text | No | (Empty) | HEADER NAME="myHeader" | Specifies
the "key" in a key-value pair. This is the handle that
you assign to an object. In Amazon S3, details about each file
and folder are stored in key value pairs called metadata or headers.
System metadata is used and processed by Amazon S3, however, user
metadata or custom headers can be specified by you. This adds
more flexibility and enables you to better distinguish specific
files by adding or editing custom headers on existing S3 objects
or assigning custom headers to new objects. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair. |
| Value | Text | No | (Empty) | VALUE="theValue" | Specifies
the "value" in a key-value pair. This is the content
that you are storing for an object. In Amazon S3, details about
each file and folder are stored in key value pairs called metadata
or headers. System metadata is used and processed by Amazon S3,
however, user metadata or custom headers can be specified by you.
This adds more flexibility and enables you to better distinguish
specific files by adding or editing custom headers on existing
S3 objects or assigning custom headers to new objects. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair. |
Additional notes
Amazon S3 regions and endpoints
This table contains a complete list of Amazon Simple Storage Service endpoints, along with their corresponding regions, supported protocols and location constraints.
| Endpoint | Region | Protocol | Location Constraints |
|---|---|---|---|
| s3.amazonaws.com | US Standard * | HTTP and HTTPS | (none required) |
| s3.us-west-2.amazonaws.com | US West (Oregon) Region | HTTP and HTTPS | us-west-2 |
| s3.us-west-1.amazonaws.com | US West (Northern California) Region | HTTP and HTTPS | us-west-1 |
| s3.eu-west-1.amazonaws.com | EU (Ireland) Region | HTTP and HTTPS | EU |
| s3.ap-southeast-1.amazonaws.com | Asia Pacific (Singapore) Region | HTTP and HTTPS | ap-southeast-1 |
| s3.ap-southeast-2.amazonaws.com | Asia Pacific (Sydney) Region | HTTP and HTTPS | ap-southeast-2 |
| s3.ap-northeast-1.amazonaws.com | Asia Pacific (Tokyo) Region | HTTP and HTTPS | ap-northeast-1 |
| s3.sa-east-1.amazonaws.com | South America (Sao Paulo) Region | HTTP and HTTPS | sa-east-1 |
* The US Standard region automatically routes requests to facilities in Northern Virginia or the Pacific Northwest using network maps.
Datasets
A dataset is a multiple column, multiple row container object. This activity creates and populates a dataset containing a specific set of fields in addition to the standard dataset fields. The table below describes these fields (assuming the dataset name assigned was theDataset).
| Name | Type | Return Value |
|---|---|---|
| theDataset.KeyName | Text | The Key Name of the object. |
| theDataset.LastModified | Date | The last modified date/time of the object. |
| theDataset.ETag | Text | The object's entity tag. This is a hash of the object that can be used to do conditional operations. The tag only reflects changes to the contents of an object, not its metadata. |
| theDataset.Size | Number | The size of the object, in bytes. |
| theDataset.Owner | Text | The owner assigned to the object. |
| theDataset.StorageClass | Text | The storage class of the object (for example, STANDARD) |
Example
NOTE:
- Copy and paste the sample AML code below directly into the Task Builder Steps Panel.
- To successfully run the sample code, update parameters containing user credentials, files, file paths, or other information specific to the task to match your environment.
Description
This task populates a dataset with a list of objects contained in a specific bucket. The dataset is then looped and within each iteration, information about the current object is displayed in a message box.
Copy
<AMAWSS3 ACTIVITY="create_session" SESSION="mySession" ACCESSKEY="AKIAJU2LHLMG6BA6BZTA" SECRETKEY="AM1B2xUP4i/Z1KM5ySNpiipKpe+rOm+wdDFaME" />
<AMAWSS3 ACTIVITY="get_objects_dataset" BUCKETNAME="myBucket" RESULTDATASET="test" SESSION="mySession" />
<AMLOOP TYPE="DATASET" DATASET="test">
<AMMESSAGEBOX>Owner - %test.ETag%Key Name - %test.KeyName%Last Modified Date - %test.LastModified%Owner - %test.Owner%Size - %test.Size%Storage Class - %test.StorageClass%</AMMESSAGEBOX>
</AMLOOP>
<AMAWSS3 ACTIVITY="end_session" SESSION="mySession" />