Amazon S3 - Copy Objects

Declaration

<AMAWSS3 ACTIVITY="copy_object" MATCHCASE="yes" EXCLUDE="text" 
RE="yes" ACCESSKEY="text" SECRETKEY="AM3bajY9WlJ018=aME" 
PROTOCOL="https" MAXERRORRETRY="text" SERVICEURL="text" 
PROXYHOST="text" PROXYPORT="text" PROXYUSER="text" 
PROXYPWD="AM3bajY9WlJ018=aME" BUCKETNAME="text" 
KEYNAME="text" DESTBUCKET="text" DESTKEY="text" 
TIMEOUT="number" ACL="Private" DIRECTIVE="Copy" 
SOURCEVERSION="number" RESULTDATASET="text"><HEADER 
NAME="text" VALUE="text" /><METADATA 
NAME="text" VALUE="text" /></AMAWSS3>

Description: Copies an S3 object from one bucket to another. This operation is identical to performing a Get object activity followed by a Put object activity.

IMPORTANT: The AWS S3 activities are performed using Amazon's Simple Storage Service engine, therefore, launching and operating Amazon S3 a valid Access Key ID and Secret Access Key.

Practical Usage

Used to copy objects within Amazon S3. With this activity, you can create additional copies of objects, rename objects by copying them and deleting the original ones, move objects across Amazon S3 locations and update object metadata by copying original objects to new ones that contain new metadata.

Connection Parameters

Property

Type

Required

Default

Markup

Description

Connection

 

 

 

 

Indicates where AWS user credentials and preferences should originate from. This is a design mode parameter used only during task construction and configuration, thus, comprises no markup. The available options are:

  • Host (default) - Specifies that user credentials and/or advanced preferences are configured individually for this activity. This option is normally chosen if only a single activity is required to complete an operation.

  • Session - Specifies that user credentials and/or advanced preferences are obtained from a pre-configured session created in an earlier step with the use of the S3 - Create session activity. This option is normally chosen if a combination of related activities are required to complete an operation. Linking several activities to a single session eliminates redundancy. Additionally, a single task supports construction and simultaneous execution of multiple sessions, improving efficiency.

Session

Text

Yes if connection is session-based

EC2Session1

SESSION="S3Session1"

The name of an existing session to attach this activity to. This parameter is active only if the Connection parameter is set to Session. The default session name is 'S3Session1'.

Access key

Text

Yes if connection is host-based

(Empty)

ACCESSKEY=

"022QF06E7MXBSH9DHM02"

A 20-character alphanumeric string that uniquely identifies the owner of the AWS service account, similar to a username. This key along with a corresponding secret access key forms a secure information set that AWS uses to confirm a valid user's identity. This parameter is active only if the Connection parameter is set to Host.

Secret Access key

Text

Yes if connection is host-based

(Empty)

SECRETKEY=

"kWcrlUX5JEDGM/LtmEENI/

aVmYvHNif5zB+d9+ct"

A 40-character string that serves the role as password to access the AWS service account. This along with an associated access key forms a secure information set that EC2 uses to confirm a valid user's identity. This parameter is active only if the Connection parameter is set to Host.

Protocol

Text (options)

No

HTTP

PROTOCOL="HTTPS"

The protocol required. The available options are:

  • HTTP (default)

  • HTTPS

User agent

Text

No

AutoMate

USERAGENT="AutoMate"

The name of the client or application initiating requests to AWS, which in this case, is AutoMate. This parameter's default value is 'AutoMate'. 

Service URL

Text

No

(Empty)

SERVICEURL=

"https://s3.eu-west-1.amazonaws.com"

The URL that provides the service endpoint. To make the service call to a different region, you can pass the region-specific endpoint URL. For example, entering  https://s3.us-west-1.amazonaws.com points to US West (Northern California) region. A complete list of S3 regions, along with associated endpoints and valid protocols can be found below under S3 Endpoints and Regions.

Maximum retry on error

Number

No

(Empty)

MAXERRORRETRY="4"

The total amount of times this activity should retry its request to the server before returning an error. Network components can generate errors anytime in the life of a request, thus, implementing retries can increase reliability. 

Proxy host

Text

No

(Empty)

PROXYHOST="proxy.host.com"

The host name (e.g., server.domain.com) or IP address (e.g., xxx.xxx.xxx.xxx) of the proxy server to use when connecting to AWS.  

Proxy port

Number

No

(Empty)

PROXYPORT="1028"

The port that should be used to connect to the proxy server.

Proxy username

Text

No

(Empty)

PROXYUSER="username"

The username that should be used to authenticate connection with the proxy server (if required).

Proxy password

Text

No

(Empty)

PROXYPWD="encrypted"

The password that should be used to authenticate connection with the proxy server (if required).

Object Parameters

Property

Type

Required

Default

Markup

Description

Bucket name

Text

Yes

(Empty)

BUCKETNAME="MyBucket"

Indicates the name of the bucket in which to copy object(s) from. To specify more than one object, use wildcard characters (* or ?). To specify multiple objects or wildcard masks, separate them with a pipe symbol (|). Example: *.txt|*.bak

Key name

Text

Yes

(Empty)

  1. KEYNAME="keyname.txt"

  2. KEYNAME="*.txt"

  3. KEYNAME=*.txt|*.doc"

  4. KEYNAME="*.*"

Specifies the key name of the object(s) in which to copy. A key is the unique identifier for an object within a bucket. To specify more than one object, use wildcard characters (* or ?). To specify multiple objects or wildcard masks, separate them with a pipe symbol (|). Example: *.txt|*.bak

Destination bucket

Text

Yes

(Empty)

DESTBUCKET="myBucket"

Indicates the bucket to copy the object(s) to.

Destination key (optional)

Text

No

(Empty)

DESTKEY="keyName"

Indicates the destination key name to copy the object(s) to. If a matching key name already exists in the destination, it will be overwritten.

Canned ACL

Text (options)

Yes

Private

  1. ACL="noacl"

  2. ACL="private"

  3. ACL="publicread"

  4. ACL="publicreadwrite"

  5. ACL="authenticateread"

  6. ACL="bucketownerread"

  7. ACL="bucketownerfull"

Because of restrictions in what can be sent via http headers, Amazon S3 supports the concept of canned access policies. The available canned ACL options are:

  • NoACL - No access policies.

  • Private (default) - Owner gets full control. No one else has access rights.

  • PublicRead - Owner gets full control and the anonymous principal is granted read access.

  • PublicReadWrite - Owner gets full control, the anonymous principal is granted read/write access. Useful policy to apply to a bucket, but is generally not recommended.

  • AuthenticatedRead - Owner gets full control, and any principal authenticated as a registered Amazon S3 user is granted read access.

  • BucketOwnerRead - Object owner gets full control. Bucket owner gets read access. This ACL applies only to objects and is equivalent to Private when used with Create Bucket activity. Use this ACL to let someone other than the bucket owner write content (get full control) in the bucket but still grant the bucket owner read access to the objects.

  • BucketOwnerFullControl - Object owner gets full control. Bucket owner gets full control. Applies only to objects and is equivalent to Private when used with Create Bucket activity. Use this ACL to let someone other than the bucket owner write content (get full control) in the bucket but still grant the bucket owner full rights over the objects.

Metadata directive

Text (options)

Yes

Copy

  1. DIRECTIVE="copy"

  2. DIRECTIVE="replace"

Specifies whether the metadata is copied from the source object or replaced with metadata provided in the request. The available Options are:

  • Copy (Default) - Metadata is copied from the source.

  • Replace - Metadata is replaced with metadata provided in the request.

NOTE: If copied, the metadata remains unchanged. Otherwise, all original metadata is replaced by the metadata you specify. You cannot copy an object to itself unless this parameter is specified and its value set to Replace.

Source version ID (optional)

Number

No

(Empty)

SOURCEVERSION="333333"

The version of the Amazon S3 object in which to copy. If versioning is enabled on your S3 bucket, (using the Set Bucket Versioning activity) this parameter is useful in identifying objects in the same bucket that have the same key name but different version IDs.

Timeout (in minutes)

Number

No

20

TIMEOUT="25"

The timeout value (in minutes) that should be set for this activity. The value is assigned to the timeout properties of the requested object used for S3 copy requests. The default value is 20 minutes.

Create and populate dataset with S3 object information

Text

No

(Empty)

RESULTDATASET="myData"

The name of the dataset to create and populate with information in regards to the object(s) to copy. More details regarding the individual fields that this dataset creates can be found below under Datasets.

Filter Options

Property

Type

Required

Default

Markup

Description

Match case

Yes/No

No

No

MATCHCASE="YES"

If set to YES, this activity becomes case sensitive. This parameter is set to NO by default.

Exclude Mask

Text

No

(Empty)

EXCLUDE="*.txt"

Causes this activity to omit matching mask(s) specified. Key names or wildcard masks may be used. Multiple entries may be specified by separating them with a pipe symbol (|), Example: *.txt|*.bak

Regular expression

Yes/No

No

No

RE="YES"

If set to YES, the value entered in the font-size: 8pt; font-weight: bold;">Exclude mask parameter will be interpreted as regular expression. If set to NO, the value will be interpreted as normal readable text.

Only if newer than

Date

No

(Empty)

ISNEWERTHAN=

"%DateSerial(2001,10,12) + TimeSerial(00,00,00)%"

If enabled, causes this activity to only act on objects that are newer than the date/time specified. If this parameter is left blank or disabled (default), object dates are ignored.

Click the Custom button to select from a list of pre-defined date parameters. Enable the Expression option to allow entry of a date/time expression.

Only if older than

Date

No

(Empty)

ISOLDERTHAN=

"%DateSerial(2001,10,12) + TimeSerial(00,00,00)%"

If enabled, causes this activity to only act on objects that are older than the date/time specified. If this parameter is left blank or disabled (default), object dates are ignored.

Click the Custom button to select from a list of pre-defined date parameters. Enable the Expression option to allow entry of a date/time expression.

Advanced Parameters

Each Amazon S3 object has a set of key-value pairs with which it is associated called Headers or Metadata. Metadata can provide important details about an object, such as file name, type, date of creation/modification etc. There are two kinds of metadata in S3; system metadata, and user metadata. System metadata is used and processed by Amazon S3. User metadata (also known as custom header) is specified by you, the user. Amazon S3 simply stores it and passes it back to you upon request. S3 lets you to store your personal information as custom headers or user metadata such as First Name, Last Name, Company Name, Phone Numbers, etc, so that you can distinguish specific files. Using this parameter, you can add new custom header/user metadata to existing S3 objects, edit default S3 metadata on a bucket or store/upload new objects with custom header or metadata.

Property

Type

Required

Default

Markup

Description

Name

Text

No

(Empty)

HEADER NAME="myHeader"

Specifies the "key" in a key-value pair. This is the handle that you assign to an object. In Amazon S3, details about each file and folder are stored in key value pairs called metadata or headers. System metadata is used and processed by Amazon S3, however, user metadata or custom headers can be specified by you. This adds more flexibility and enables you to better distinguish specific files by adding or editing custom headers on existing S3 objects or assigning custom headers to new objects. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair.

Value

Text

No

(Empty)

VALUE="theValue"

Specifies the "value" in a key-value pair. This is the content that you are storing for an object. In Amazon S3, details about each file and folder are stored in key value pairs called metadata or headers. System metadata is used and processed by Amazon S3, however, user metadata or custom headers can be specified by you. This adds more flexibility and enables you to better distinguish specific files by adding or editing custom headers on existing S3 objects or assigning custom headers to new objects. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair.

Property

Type

Required

Default

Markup

Description

Header(s) Name

Text

No

(Empty)

  1. HEADER NAME="City"

  2. HEADER NAME="State"

Specifies the "key" in a header key-value pair. In Amazon S3, details about each file and folder are stored in key value pairs called metadata or headers. System metadata is used and processed by Amazon S3, however, user metadata or custom headers can be specified by you. This adds more flexibility and enables you to better distinguish specific files by adding or editing custom headers on existing S3 objects or assigning custom headers to new objects. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair.

Header(s) Value

Text

No

(Empty)

  1. VALUE="Los Angeles"

  2. VALUE="California"

Specifies the "value" in a header key-value pair. In Amazon S3, details about each file and folder are stored in key value pairs called metadata or headers. System metadata is used and processed by Amazon S3, however, user metadata or custom headers can be specified by you. This adds more flexibility and enables you to better distinguish specific files by adding or editing custom headers on existing S3 objects or assigning custom headers to new objects. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair.

Metadata Name

Text

No

(Empty)

METADATA NAME="mymetadata"

Specifies the "key" in a metadata key-value pair. This is the handle that you assign to an object. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair.

Metadata Value

Text

No

(Empty)

VALUE="metadataValue"

Specifies the "value" in a metadata key-value pair. This is the content that you are storing for an object. Press Click here to add new row... to add a key-value pair. Press the red X to remove an existing key-value pair.

Description tab - A custom description can be provided on the Description tab to convey additional information or share special notes about a task step.

Error Causes tab - Specify how this step should behave upon the occurrence of an error. (Refer to Task Builder > Error Causes Tab for details.)

On Error tab - Specify what AWE should do if this step encounters an error as defined on the Error Causes tab. (Refer to Task Builder > On Error Tab for details.)

S3 Endpoints and Regions

This table contains a complete list of Amazon Simple Storage Service endpoints, along with their corresponding regions, supported protocols and location constraints.

Endpoint

Region

Protocol

Location Constraints

s3.amazonaws.com

US Standard *

HTTP and HTTPS

(none required)

s3.us-west-2.amazonaws.com

US West (Oregon) Region

HTTP and HTTPS

us-west-2

s3.us-west-1.amazonaws.com

US West (Northern California) Region

HTTP and HTTPS

us-west-1

s3.eu-west-1.amazonaws.com

EU (Ireland) Region

HTTP and HTTPS

EU

s3.ap-southeast-1.amazonaws.com

Asia Pacific (Singapore) Region

HTTP and HTTPS

ap-southeast-1

s3.ap-southeast-2.amazonaws.com

Asia Pacific (Sydney) Region

HTTP and HTTPS

ap-southeast-2

s3.ap-northeast-1.amazonaws.com

Asia Pacific (Tokyo) Region

HTTP and HTTPS

ap-northeast-1

s3.sa-east-1.amazonaws.com

South America (Sao Paulo) Region

HTTP and HTTPS

sa-east-1

* The US Standard region automatically routes requests to facilities in Northern Virginia or the Pacific Northwest using network maps.

Datasets

A dataset is a multiple column, multiple row container object. This activity creates and populates a dataset containing a specific set of fields in addition to the standard dataset fields.

Name

Type

Return Value

theDataset.KeyName

Text

The Key Name of the retrieved object.

theDataset.AmazonID2

Text

A special token associated to the retrieved object that helps Amazon Web Service (AWS) troubleshoot problems.

theDataset.ETag

Text

The object's entity tag. This is a hash of the object that can be used to do conditional operations. The ETag only reflects changes to the contents of an object, not its metadata.

theDataset.Size

Number

The size of the object, in bytes.

theDataset.ContentType

Text

A standard MIME type describing the format of the object (e.g., text/plain).

theDataset.VersionID

Text

The version ID of the retrieved object if it has a unique version ID.

NOTE: To get the version ID, Versioning must be enabled using the Set bucket versioning activity.

Example

The sample AML code below can be copied and pasted directly into the Steps panel of the Task Builder.

Description: Copy object "myFile.txt" from source bucket "myBucket" to destination bucket "Bucket2". Overwrite exiting object. Use "Session2" S3 session.

<AMAWSS3 ACTIVITY="copy_object" BUCKETNAME="myBucket" 
KEYNAME="myFile.txt" DESTBUCKET="Bucket2" 
DIRECTIVE="Replace" RESULTDATASET="myDataset" 
SESSION="Session2" />