|
Amazon S3 - Copy objects |
Declaration
<AMAWSS3 ACTIVITY="copy_object" PROVIDER="text (options)" SESSION="text" ACCESSKEY="text" SECRETKEY="text (encrypted)" PROTOCOL="text (options)" USERAGENT="text" MAXERRORRETRY="number" SERVICEURL="text" PROXYHOST="text" PROXYPORT="number" PROXYUSER="text" PROXYPWD="text (encrypted)" BUCKETNAME="text" KEYNAME="text" DESTBUCKET="text" DESTKEY="text" TIMEOUT="number" ACL="text (options)" DIRECTIVE="text (options)" SOURCEVERSION="number" RESULTDATASET="text" MATCHCASE="YES/NO" EXCLUDE="text" RE="YES/NO" ISNEWERTHAN="%date/time expression%" ISOLDERTHAN="%date/time expression%"><HEADER NAME="text" VALUE="text" /><METADATA NAME="text" VALUE="text" /></AMAWSS3>
Description
Copies an S3 object from one bucket to another. This operation is identical to performing a Amazon S3 - Get object(s) activity followed by a Amazon S3 - Put object(s) activity.
Practical Usage
Used to copy objects within Amazon S3. With this activity, you can create additional copies of objects, rename objects by copying them and deleting the original ones, move objects across Amazon S3 locations and update object metadata by copying original objects to new ones that contain new metadata.
Parameters
Connection
Property | Type | Required | Default | Markup | Description |
---|---|---|---|---|---|
Connection | --- | --- | --- | --- | Indicates
where AWS user credentials and preferences should originate from.
This is a design mode parameter used only during task construction
and configuration, thus, comprises no markup. The available options
are:
|
Session | Text | Yes, if the Connection parameter is set to Session | EC2Session1 | SESSION="S3Session1" | The name of an existing session to attach this activity to. This parameter is active only if the Connection parameter is set to Session. The default session name is S3Session1. |
Access key | Text | Yes, if the Connection parameter is set to Host | (Empty) | ACCESSKEY="022QF06E7MXBSH9DHM02" | A 20-character alphanumeric string that uniquely identifies the owner of the AWS service account, similar to a username. This key along with a corresponding secret access key forms a secure information set that AWS uses to confirm a valid user's identity. This parameter is active only if the Connection parameter is set to Host. |
Secret access key | Text | Yes, if the Connection parameter is set to Host | (Empty) | SECRETKEY="text (encrypted)" | A 40-character string that serves the role as password to access the AWS service account. This along with an associated access key forms a secure information set that EC2 uses to confirm a valid user's identity. This parameter is active only if the Connection parameter is set to Host. |
Protocol | Text (options) | No | HTTP | PROTOCOL="HTTPS" | The
protocol required. The available options are:
|
User agent | Text | No | Automate | USERAGENT="Automate" | The name of the client or application initiating requests to AWS, which in this case, is Automate. This parameter's default value is Automate. |
Service URL | Text | No | (Empty) | SERVICEURL="https://s3.eu-west-1.amazonaws.com" | The URL that provides the service endpoint. To make the service call to a different region, you can pass the region-specific endpoint URL. For example, entering https://s3.us-west-1.amazonaws.com points to US West (Northern California) region. A complete list of S3 regions, along with associated endpoints and valid protocols can be found below under Amazon S3 Endpoints and Regions . |
Maximum number of retries on error | Number | No | (Empty) | MAXERRORRETRY="4" | The total amount of times this activity should retry its request to the server before returning an error. Network components can generate errors anytime in the life of a request, thus, implementing retries can increase reliability. |
Proxy host | Text | No | (Empty) | PROXYHOST="proxy.host.com" | The host name (for example, server.domain.com) or IP address (for example, 192.168.0.100) of the proxy server to use when connecting to AWS. |
Proxy port | Number | No | (Empty) | PROXYPORT="1028" | The port that should be used to connect to the proxy server. |
Proxy username | Text | No | (Empty) | PROXYUSER="username" | The username that should be used to authenticate connection with the proxy server (if required). |
Proxy password | Text | No | (Empty) | PROXYPWD="encrypted" | The password that should be used to authenticate connection with the proxy server (if required). |
Object
Property | Type | Required | Default | Markup | Description |
---|---|---|---|---|---|
Bucket name | Text | Yes | (Empty) | BUCKETNAME="MyBucket" | Indicates the name of the bucket in which to copy objects from. To specify more than one object, use wildcard characters (for example, * or ?). To specify multiple objects or wildcard masks, separate them with a pipe symbol (|) (for example, *.txt|*.bak). |
Key name | Text | Yes | (Empty) |
|
Specifies the key name of the objects in which to copy. A key is the unique identifier for an object within a bucket. To specify more than one object, use wildcard characters (for example, * or ?). To specify multiple objects or wildcard masks, separate them with a pipe symbol (|) (for example, *.txt|*.bak). |
Destination bucket | Text | Yes | (Empty) | DESTBUCKET="myBucket" | Indicates the bucket to copy the objects to. |
Destination key (optional) | Text | No | (Empty) | DESTKEY="keyName" | Indicates the destination key name to copy the objects to. If a matching key name already exists in the destination, it will be overwritten. |
Canned ACL | Text (options) | Yes | Private |
|
Because
of restrictions in what can be sent by way of http headers, Amazon S3
supports the concept of canned access policies. The available
canned ACL options are:
|
Metadata directive | Text (options) | Yes | Copy |
|
Specifies
whether the metadata is copied from the source object or replaced
with metadata provided in the request. The available Options are:
NOTE: If copied, the metadata remains unchanged. Otherwise, all original
metadata is replaced by the metadata you specify. You cannot copy
an object to itself unless this parameter is specified and its
value set to Replace. |
Source version ID (optional) | Number | No | (Empty) | SOURCEVERSION="333333" | The version of the Amazon S3 object in which to copy. If versioning is enabled on your S3 bucket, (using the Amazon S3 - Set bucket versioning activity) this parameter is useful in identifying objects in the same bucket that have the same key name but different version IDs. |
Timeout (minutes) | Number | No | 20 |
TIMEOUT="25" |
The timeout value (in minutes) that should be set for this activity. The value is assigned to the timeout properties of the requested object used for S3 copy requests. The default value is 20 minutes. |
Create and populate dataset with S3 object information | Text | No | (Empty) | RESULTDATASET="myData" | The name of the dataset to create and populate with information in regards to the objects to copy. More details regarding the individual fields that this dataset creates can be found below under Datasets. |
File Options
Property | Type | Required | Default | Markup | Description |
---|---|---|---|---|---|
Match case | Yes/No | No | No | MATCHCASE="YES" | If selected, this activity becomes case sensitive. This parameter is disabled by default. |
Exclude Mask | Text | No | (Empty) | EXCLUDE="*.txt" | Causes this activity to omit matching masks specified. Key names or wildcard masks may be used. To specify more than one object, use wildcard characters (for example, * or ?). To specify multiple key names or wildcard masks, separate them with a pipe symbol (|) (for example, *.txt|*.bak). |
Regular expression | Yes/No | No | No | RE="YES" | If selected, the value entered in the Exclude mask parameter will be interpreted as regular expression. If disabled, the value will be interpreted as normal readable text. |
Only if newer than | Date | No | (Empty) | ISNEWERTHAN="%DateSerial(2001,10,12) + TimeSerial(00,00,00)%" | If
enabled, causes this activity to only act on objects that are
newer than the date/time specified. If this parameter is left
blank or disabled (default), object dates are ignored. Click the Custom button to select from a list of pre-defined date parameters. Enable the Expression option to allow entry of a date/time expression. |
Only if older than | Date | No | (Empty) | ISOLDERTHAN="%DateSerial(2001,10,12) + TimeSerial(00,00,00)%" | If
enabled, causes this activity to only act on objects that are
older than the date/time specified. If this parameter is left
blank or disabled (default), object dates are ignored. Click the Custom button to select from a list of pre-defined date parameters. Enable the Expression option to allow entry of a date/time expression. |
Advanced
Each Amazon S3 object has a set of key-value pairs with which it is associated called Headers or Metadata. Metadata can provide important details about an object, such as file name, type, date of creation/modification, etc. There are two kinds of metadata in S3; system metadata, and user metadata. System metadata is used and processed by Amazon S3. User metadata (also known as custom header) is specified by you, the user. Amazon S3 simply stores it and passes it back to you upon request. Automate lets you to store your personal information as custom headers or user metadata such as First Name, Last Name, Company Name, Phone Numbers, etc, so that you can distinguish specific files. Using this parameter, you can add new custom header/user metadata to existing S3 objects, edit default S3 metadata on a bucket or store/upload new objects with custom header or metadata.
Property | Type | Required | Default | Markup | Description |
---|---|---|---|---|---|
Headers Name | Text | No | (Empty) |
|
Specifies the "key" in a header key-value pair. In Amazon S3, details about each file and folder are stored in key value pairs called metadata or headers. System metadata is used and processed by Amazon S3, however, user metadata or custom headers can be specified by you. This adds more flexibility and enables you to better distinguish specific files by adding or editing custom headers on existing S3 objects or assigning custom headers to new objects. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair. |
Headers Value | Text | No | (Empty) |
|
Specifies the "value" in a header key-value pair. In Amazon S3, details about each file and folder are stored in key value pairs called metadata or headers. System metadata is used and processed by Amazon S3, however, user metadata or custom headers can be specified by you. This adds more flexibility and enables you to better distinguish specific files by adding or editing custom headers on existing S3 objects or assigning custom headers to new objects. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair. |
Metadata Name | Text | No | (Empty) | METADATA NAME="mymetadata" | Specifies the "key" in a metadata key-value pair. This is the handle that you assign to an object. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair. |
Metadata Value | Text | No | (Empty) | VALUE="metadataValue" | Specifies the "value" in a metadata key-value pair. This is the content that you are storing for an object. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair. |
Additional Notes
Amazon S3 Endpoints and Regions
This table contains a complete list of Amazon Simple Storage Service endpoints, along with their corresponding regions, supported protocols and location constraints.
Endpoint | Region | Protocol | Location Constraints |
---|---|---|---|
s3.amazonaws.com | US Standard * | HTTP and HTTPS | (none required) |
s3.us-west-2.amazonaws.com | US West (Oregon) Region | HTTP and HTTPS | us-west-2 |
s3.us-west-1.amazonaws.com | US West (Northern California) Region | HTTP and HTTPS | us-west-1 |
s3.eu-west-1.amazonaws.com | EU (Ireland) Region | HTTP and HTTPS | EU |
s3.ap-southeast-1.amazonaws.com | Asia Pacific (Singapore) Region | HTTP and HTTPS | ap-southeast-1 |
s3.ap-southeast-2.amazonaws.com | Asia Pacific (Sydney) Region | HTTP and HTTPS | ap-southeast-2 |
s3.ap-northeast-1.amazonaws.com | Asia Pacific (Tokyo) Region | HTTP and HTTPS | ap-northeast-1 |
s3.sa-east-1.amazonaws.com | South America (Sao Paulo) Region | HTTP and HTTPS | sa-east-1 |
* The US Standard region automatically routes requests to facilities in Northern Virginia or the Pacific Northwest using network maps.
Datasets
A dataset is a multiple column, multiple row container object. This activity creates and populates a dataset containing a specific set of fields in addition to the standard dataset fields.
Name | Type | Return Value |
---|---|---|
theDataset.KeyName | Text | The Key Name of the retrieved object. |
theDataset.AmazonID2 | Text | A special token associated to the retrieved object that helps Amazon Web Service (AWS) troubleshoot problems. |
theDataset.ETag | Text | The object's entity tag. This is a hash of the object that can be used to do conditional operations. The ETag only reflects changes to the contents of an object, not its metadata. |
theDataset.Size | Number | The size of the object, in bytes. |
theDataset.ContentType | Text | A standard MIME type describing the format of the object (for example, text/plain). |
theDataset.VersionID | Text | The version ID of the retrieved object if it
has a unique version ID. NOTE:
To get the version ID, Versioning must be enabled using the Amazon S3 - Set bucket versioning
activity. |
Example
- The sample AML code below can be copied and pasted directly into the Steps Panel of the Task Builder.
- Parameters containing user credentials, files, file paths, and/or other information specific to the task must be customized before the sample code can run successfully.
Description
This sample task copies an object from a source bucket to a destination bucket, overwriting an existing object.
<AMAWSS3 ACTIVITY="create_session" SESSION="mySession" ACCESSKEY="AKIAJU2LHLMG6BA6BZTA" SECRETKEY="AM1B2xUP4i/Z1KM5ySNpiipKpe+rOm+wdDFaME" />
<AMAWSS3 ACTIVITY="list_objects" PROVIDER="session_based" SESSION="S3Session1" BUCKETNAME="myBucket" RESULTDATASET="test" />
<AMLOOP ACTIVITY="dataset" DATASET="test" />
<AMSHOWDIALOG>Owner - %test.ETag%Key Name - %test.KeyName%Last Modified Date - %test.LastModified%Owner - %test.Owner%Size - %test.Size%Storage Class - %test.StorageClass%</AMSHOWDIALOG><AMLOOP ACTIVITY="end" />
<AMAWSS3 ACTIVITY="end_session" SESSION="mySession" />