FTP - Synchronize Folder(s)

Declaration

<AMFTP ACTIVITY="synchronize_folder" REMOTEFOLDER="text" 
LOCALFOLDER="text" RESULTDATASET="text" 
SYNCHRONIZATION="remote_to_local" 
SYNCOVERWRITE="text (options)" SUBFOLDERS="YES/NO" 
MATCHCASE="YES/NO" CHECKSUM="YES/NO" 
VALIDATECHECKSUM="YES/NO" 
EXCLUDE="text" RE="YES/NO" INCLUDE="text" 
SESSION="text" />

Description: Synchronizes folders located on the FTP server with folders located on the local machine. Folders can be synchronized either uni-directionally (one-way) or bi-directionally (both ways). Other options include the ability to ignore or overwrite existing files, include/exclude masks and checksum validation.

Practical Usage

Usually used to replicate folders between an FTP server and a local machine or to backup specific folders.

File Parameters

Property

Type

Required

Default

Markup

Description

Remote Folder

 

Text

 

Yes

 

(Empty)

 

EMOTEFOLDER="/Home/FTPfolder"

Specifies the FTP server folder that should be synchronized.

If synchronization is set to Remote --> Local (Unidirectional) then files from the FTP server that do not exist locally will be transferred to the folder specified in the Local Folder parameter.

If synchronization is set to Local --> Remote (Unidirectional) then local files that do not exist in the FTP server will be transferred to the folder specified in the Remote Folder parameter.

If synchronization is set to Remote <-->Local (Bidirectional) then files from the FTP server that do not exist locally will be transferred to the folder specified in the Local Folder parameter and local files that do not exist in the FTP server will be transferred to the folder specified in the Remote Folder parameter.

Local Folder

Text

Yes

(Empty)

LOCALFOLDER="c:\foldername\" 

Specifies the destination folder that resides locally.

Create and populate dataset

Text

No

(Empty)

RESULTDATASET="theDataset"

The name of a dataset to be populated with information regarding the folder/files to be synchronized.

Synchronization

Options

Yes

Bidirectional

SYNCHRONIZATION=

"remote_to_local"

Specifies the method by which the files should be synchronized. The available options are:

  • Remote <-->Local (Bidirectional) - The files will be synchronized both ways, from the remote folder to the local folder and from the local folder to the remote folder. files from the FTP server that do not exist locally will be transferred to the folder specified in the Local Folder parameter and local files that do not exist in the FTP server will be transferred to the folder specified in the Remote Folder parameter.

  • Remote --> Local (Unidirectional) - The files will be synchronized one way, from remote folder to the local folder. Files from the FTP server that do not exist locally will be transferred to the folder specified in the Local Folder parameter.

  • Local --> Remote (Unidirectional) - The files will be synchronized one way, from local folder to the remote folder. Local files that do not exist in the FTP server will be transferred to the folder specified in the Remote Folder parameter.

Exact copy in remote/local folder

Yes/No

No

No

EXACTCOPY="YES"

When set to YES, an exact copy of files in the source folder will be created in the destination folder. Files and folders that exist in the destination folder but do not exist in the source folder are removed. If a newer version of a file or folder exists in the source, it will be copied to the destination. The default value is NO.

This parameter is available only if Remote --> Local (Unidirectional) or Local --> Remote (Unidirectional) is selected via the Synchronization parameter.

If file exists at both locations

 

Options

 

No

 

Newer Overwrites Older

 

SYNCHOVERWRITE="ignore"

 

Specifies what should be done if a file with the same name exists in both source and destination folders. The available options are:

  • Newer Overwrites Older (Default) - The file with the newer modified date will overwrite the matching file with an older modified date.

  • Ignore - The file with the same name that exists in the source folder will be ignored. No actions will be performed in regards to this file whether it is newer or older than its counterpart.

  • Overwrite - The file with the same name that exists in the source folder will overwrite its counterpart that resides in the destination folder, whether it is newer or older.

FTP Server Timezone

Options

No

(GMT) Coordinated Universal Time [UTC]

FTPSERVERTIMEZONE="Fiji [Fiji Standard Time]"

Indicates the timezone to set for the FTP server. This is useful if file date/time specifications are set for this activity and the FTP server is in a different timezone than that of the local machine. This property converts the local time into the server timezone selected.   

File Options Parameters

Property

Type

Required

Default

Markup

Description

Include Subfolders

Yes/No

No

No

SUBFOLDERS="YES"

When set to YES, indicate that, if present, subfolders should be searched for files matching the mask specified in the Remote File(s) parameter. The default value is NO.

Match Case

Yes/No

No

No

MATCHCASE="YES"

Indicates whether the properties set within this activity should be case sensitive in relation to the FTP server.  The default value is NO.

Transfer Type

Options

Yes

Binary

TRANSFERTYPE="ascii"

Specifies whether the transfer type should be set to binary or ASCII mode. The default is binary.

Binary mode refers to transferring files as a binary stream of data. Where ASCII mode may use special control characters to format data, binary mode transmits the raw bytes of the file being transferred. In this way, the file is transferred in its exact original form.

When files are transferred in ASCII mode, the transferred data is considered to contain only ASCII formatted text. The party that is receiving the transferred data is responsible for translating the format of the received text to one that is compatible with their operating system.

Validate Checksum

Yes/No

No

No

VALIDATECHECKSUM="YES"

Indicates whether to validate the integrity of data being transferred by calculating a checksum using the selected algorithm. The default value is NO.

Checksum Type

Options

No

CRC

CHECKSUMTYPE="sha1"

Specifies the checksum algorithm to use. The available options are:

  • CRC (Default)

  • MD5

  • SHA1

This parameter is valid only if the Validate Checksum parameter is set to YES.

Include Mask

Text

No

*.*

INCLUDE="*.jpg"

Specifies a wildcard mask (such as * or ?) representing the files that should be synchronized. The defaultvalue is *.* which specifies all files.

Exclude Mask

Text

No

(Empty)

EXCLUDE="*.txt"

Causes this activity to omit any files matching the mask(s) specified. Filenames or wildcard (e.g., * or ?) masks may be used, multiple entries may be specified by separating them with the | symbol (e.g., *.txt|*.bak).

Regular Expression

Yes/No

No

No

RE="YES"

Specifies whether the value in the Exclude Mask parameter is a regular expression. Regular Expression, also referred to as regex or regexp, provide a concise and flexible means for matching strings of text.

Credentials Parameters

The Credentials parameters allow you to set or link specific recommendations for this particular FTP activity such as Host, Username, Password, etc.

           

Property

Type

Required

Default

Markup

Description

Provide Credentials

 

 

 

 

Indicates where this activity's credentials should originate from. Different properties apply depending on the option selected. The available options are:

  • Custom (Default) - Specifies that a custom set of credentials will be entered for this FTP activity. Select this option if performing a single FTP activity.

  • Session Based - Specifies that credentials should derive from a session created in a previous step with the use of the FTP Log On activity. This allows several FTP activities to be linked to a specific session.

Host

Text

Yes

(Empty)

a)SERVER="ftp.netauto.com"

b)SERVER="123.456.789.012"

Specifies the FTP server to connect to. This can be an IP address (e.g., xxx.xxx.xxx.xxx) or a server and domain name (e.g., server.domain.com).

This property is active only if Custom is selected under the Provide Credentials property.

Username

Text

Yes

(Empty)

USERNAME="ClarkKent"

Specifies the username that should be used when logging on to the FTP Server. The Username should be pre-configured at the server level.

This property is active only if Custom is selected under the Provide Credentials property.

Password

Text

Yes

(Empty)

PASSWORD="password"

Specifies the password that should be used when logging on to the FTP Server. When the step is created using the Task Builder it is written to the task encrypted.

This property is active only if Custom is selected under the Provide Credentials property.

Use Anonymous Logon

 

Yes/No

 

No

 

No

 

ANONYMOUSLOGIN="YES"

 

Specifies that the FTP server should be logged onto as an "Anonymous" user. The server must be configured to accept Anonymous connections. If set to YES, the Username/Password parameters are ignored. The default value is NO.

This property is active only if Custom is selected under the Provide Credentials property.

Session

Text

Yes

FTPSession

SESSION="FTPSession2"

Indicates the session name that this activity should originate from. This allows several FTP activities to be linked to a specific session. Numerous sessions can be used within a single task.

This property is active only if Session Based is selected under the Provide Credentials property.

Advanced Parameters

These parameters are active only if Custom is selected under the Provide Credentials property.

Property

Type

Required

Default

Markup

Description

Connection Type

Options

Yes

FTP (Standard FTP)

TYPE="ftps_implicit"

Specifies the type of FTP connection that should be used. The available options are:

  • FTP (Standard FTP): Connect using standard FTP .

  • FTP with SSL (Implicit): Connect using FTPS (also known as FTP Secure or FTP-SSL). This connection type adds support for the Secure Sockets Layer (SSL) and Transport Layer Security (TLS) cryptographic protocols. In Implicit mode, a client immediately sends the FTPS server a TLS/SSL "ClientHello" message. If such a message is not received by the FTPS server, the connection is dropped.

  • FTP with SSL (Explicit): Connect using FTPS (also known as FTP Secure or FTP-SSL). This connection type adds support for the Secure Sockets Layer (SSL) and Transport Layer Security (TLS) cryptographic protocols. In Explicit mode, a client must "explicitly request" security from a FTPS server and then use a mutually agreed encryption method.

  • SFTP (Password): Connect using SFTP (also known as Secure FTP). SFTP uses SSH (Secure Shell) protocol to transfer files. Unlike standard FTP, it encrypts both commands and data, preventing sensitive information from being transmitted in the clear over the network. In Password mode, a password is used to authenticate the SFTP connection.

  • SFTP (Public Key File): Connect using SFTP (also known as Secure FTP). SFTP uses SSH (Secure Shell) protocol to transfer files. Unlike standard FTP, it encrypts both commands and data, preventing sensitive information from being transmitted in the clear over the network. In Public Key File mode, connection is validated via a specified public key.

Port

Text

Yes

210

PORT="1000"

Specifies the port that should be used to connect to the FTP server. Most standard FTP servers operate on port 21 (the default port specified) however, this parameter can be customized in case the FTP server operates on other ports.

NOTE: Other default ports may be assigned depending on the server connection type selected.

FTP Log File

Text

No

(Empty)

LOGFILE="c:\temp\error.log"

Specifies the location of a detailed FTP log file. This log file contains the exact text of the FTP session which can be useful in diagnosing particular errors or elusive problems.

Overwrite Existing Log File

Yes/No

No

Yes

OVERWRITELOG="NO"

If YES, specifies that the log file should be overwritten if the same file already exists. The default value is YES.

Timeout

Number

Yes

30 Seconds

TIMEOUT="60"

Indicates a connection timeout to customize how long will wait before aborting a connection attempt. If connection is not established within the timeout value specified, it is automatically aborted. The default value is 30 seconds.

Upload Buffer Size

Number

Yes

65536 Bytes

UPLOADBUFFERSIZE="54443"

Specifies the upload buffer size value for files being uploaded. In some cases changing the upload buffer size can make a difference; particularly in transfers over high loss, high speed connections where latency plays an important role. The default value is 65536 bytes, which is the maximum buffer size.

NOTE: Setting the Upload Buffer Size value too high for slow connections might cause time-outs and the transfer speed calculation becomes inaccurate.

Passive Mode (For Firewalls)

 

Yes/No

 

No

No

PASSIVEMODE="YES"

 

Determines how an FTP data connection is made. Ifset to YES, issues the PASV command and the server tells where to establish the data connection. initiates both connections to the server, solving the problem of firewalls filtering the incoming data port connection to the client from the server. If set to NO, the PORT method is used. listens for a data connection which is established by the server and the PORT command tells the server where to connect to. then connects to the server where the server indicated. This method is sometimes used with some proxy configurations. However, some proxy configurations require PORT transfers and some FTP servers do not support PASV transfers.

This option is available only if the FTP (Standard FTP), FTP with SSL (Implicit) or FTP with SSL (Explicit) option is selected from the Server Connection Type parameter.

Encrypt data channel

Yes/No

No

No

ENCRYPTDATACHANNEL="YES"

If YES, specifies that all data channel communication between the FTPS client and server are to be encrypted. The default value is NO.

It may NOT be advantageous to use data channel encryption when performing transfers under the following scenarios:

  • Files being transferred are of a non-sensitive nature, making encryption unnecessary.

  • Files being transferred are already encrypted at the file level, making encryption redundant.

This option is available only if the FTP with SSL (Implicit) or FTP with SSL (Explicit) option is selected from the Server Connection Type parameter.

Ignore invalid server certificates

Yes/No

No

No

IGNOREINVALIDCERTIFICATE="YES"

If YES, specifies that this action will ignore invalid certificates when connecting to an FTP server using SSL. The default value is NO.

This option is available only if the FTP with SSL (Implicit) or FTP with SSL (Explicit) option is selected from the Server Connection Type parameter.

Clear Command Channel

Yes/No

No

No

CLEARCOMMANDCHANNEL="YES"

If YES, this option enables support for the Clear Command Channel (CCC) functionality. The CCC command can be issued by a remote FTPS client and will cause FTP client to fall out of secure mode and back in to unsecure mode. This option is useful for clients who only need to secure the authentication portion of the session. Once the USER/PASS has completed, some clients will use CCC to return to unsecure mode, which is faster. The default value is NO.

This option is available only if the FTP with SSL (Implicit) or FTP with SSL (Explicit) option is selected from the Server Connection Type parameter.

Enable Tumbleweed compatibility mode

Yes/No

No

No

TUMBLEWEEDSERVER="YES"

If set to YES, allows compatibility with Tumbleweed Secure Content Delivery.

Client Certificate Source

Options

No

No Certificate

CERTIFICATESOURCE="file"

FTP over SSL allows sessions to be encrypted between an FTP client and server. This property is used to select the source of the certificate. A certificate is a digitally-signed statement that binds the value of a public key to the identity of the person, device, or service that holds the corresponding private key. One of the main benefits of certificates is that hosts no longer have to maintain a set of passwords for individual subjects who need to be authenticated as a prerequisite to access. Instead, the host merely establishes trust in a certificate issuer. The available options are:

  • No Certificate (Default)

  • Certificate Store - Allows selection of a certificate using the Window's trusted certificate store for certificate approval.

  • File(s) - Allows selection of a certificate file.

  • Automatically select certificate - will select the best suitable certificate from the user store that matches the FTP server request.

This option is available only if the FTP with SSL (Implicit) or FTP with SSL (Explicit) option is selected from the Server Connection Type parameter.

Certificate Issuer

Text

No

(Empty)

CERTIFICATEISSUERID="Name"

Specifies information regarding the certification authority that issued the certificate.

This parameter is available only when Certificate Store option is selected from the Client Certificate Source parameter.

Certificate Serial Number

Number

No

(Empty)

CERTIFICATESERIAL=

"c7 f5 fa f8 6d ab 77 87 43 4a 11 43 f1 cd 3c 0f"

Specifies the unique serial number that the issuing certification authority assigns to the certificate. The serial number is unique for all certificates issued by a given certification authority.

This parameter is available only when Certificate Store option is selected from the Client Certificate Source parameter.

Certificate File

Text

No

(Empty)

CERTIFICATE=

"C:\Temp\Certificate_Location"

Specifies the path and filename of the certificate file.

This parameter is available only when the File(s) option is selected from the Client Certificate Source parameter.

Passphrase

Text

No

(Empty)

PASSPHRASE="passphrase"

Specifies the passphrase used to authenticate connection. A passphrase is a password that comprises a whole phrase.

This parameter is available only when the File(s) option is selected from the Client Certificate Source parameter.

Certificate private key

Text

No

(Empty)

CERTIFICATEPRIVATEKEY=

"C:\Temp\Private_Key"

Specifies the path and filename of the Private key file.

This parameter is available only when the File(s) option is selected from the Client Certificate Source parameter.

In public/private key encryption, different keys are used to encrypt and decrypt information. The first key is a private key (a key that is known only to its owner), while the second key (called the public key) can be made known and available to other entities on the network.

The two keys are different but complementary in function. For example, a user’s public key can be published in a certificate in a directory so that it is accessible to other people in the organization. The sender of a message can retrieve the user’s certificate from Active Directory, obtain the public key from the certificate, and then encrypt the message by using the recipient's public key. Information that is encrypted with the public key can be decrypted only by using the corresponding private key of the set, which remains with its owner, the recipient of the message.

Proxy Properties

These properties are active only if Custom is selected under the Provide Credentials property.

Property

Type

Required

Default

Markup

Description

Proxy Type

Options

No

None

PROXYTYPE="Socks5"

Specifies proxy protocol that should be used.  If you are unsure of the value to use in this parameter, contact your network administrator.

The available options are:

  • System Default - The settings specified in the Task Administrator preferences are used.

  • None (Default) - Instructs the activity not to use a proxy server regardless of the system default.

  • Socks 4: Instructs the activity to go through a Socks 4 compliant server.

  • Socks 4a: Instructs the activity to go through a Socks 4a compliant server.

  • Socks 5: Instructs the activity to go through a Socks 5 compliant server.

  • HTTP - Instructs the activity to go through a HTTP compliant server.

  • FTPSite - Instructs the activity to go through a FTPSite compliant server.

  • FTPUser - Instructs the activity to go through a FTPUser compliant server.

  • FTPOpen - Instructs the activity to go through a FTPOpen compliant server.

Proxy Host

Text

No

(Empty)

PROXYSERVER="proxy.host.com"

pecifies the hostname (e.g., server.domain.com) or IP address e.g., (xxx.xxx.xxx.xxx) of the proxy server.

Proxy Port

Text

Yes

21

PORT="1000"

Specifies the port that should be used to connect to the FTP server. Most standard FTP servers operate on port 21 (the default port specified) however, this parameter can be customized in case the FTP server operates on other ports.

NOTE: Other default ports may be assigned depending on the server connection type selected.

Proxy Username

Text

No

(Empty)

ROXYUSERNAME="username"

The username that should be used to authenticate when connecting through the proxy server. This option is only valid when Socks 5 is specified in the Proxy type as it is the only version that supports authentication.

Proxy Password

text

No

(Empty)

PROXYPASSWORD="encrypted"

The password that should be used to authenticate when connecting through the proxy server. When the step is created using the Task Builder it is written to the task encrypted. This option is only valid when Socks 5 is specified in the Proxy type as it is the only version that supports authentication.

Description tab - A custom description can be provided on the Description tab to convey additional information or share special notes about a task step.

Error Causes tab - Specify how this step should behave upon the occurrence of an error. (Refer to Task Builder > Error Causes Tab for details.)

On Error tab - Specify what AWE should do if this step encounters an error as defined on the Error Causes tab. (Refer to Task Builder > On Error Tab for details.)

Using the Create and populate dataset parameter creates a fixed field dataset. A dataset  is a multiple column, multiple row container object.

Datasets

The table below describes the set of columns that a dataset creates exclusive to this activity (assuming the dataset name assigned was theDataset).

Name

Data Type

Return Value

theDataset.DownloadResult

Boolean

The result of the download. If successful, 1 is returned. Otherwise, 0 is returned.

theDataset.FTPFileDate

Date

The modified date and time of the file to download.

theDataset.FTPFileName

Text

The name of the file to download.

theDataset.FTPFileSize

Number

The size of the file to download (in kb).

theDataset.LocalFileName

Text

The local file name of the file.

Variables and Expressions

All text fields allow the use of expressions, which can be entered by surrounding the expression in percentage signs (example: %MYVARIABLE%, %Left('Text',2)%). To help construct these expressions, you can open Expression Builder from these fields by pressingF2.

More on variables
More on expressions
More on the expression builder

Example

The sample AML code below can be copied and pasted directly into the Steps panel of the Task Builder.

Description: This task logs onto an FTP site, synchronizes the remote folder "/home/myFolders/FTPFolder1" with the local folder "C:\myFolders\Folder1" and logs off.

For this task to work in your environment, please make the appropriate modifications in the properties of each activity.

<AMFTP ACTIVITY="logon" SESSION="YourFTPSession" 
SERVER="YourFTPHost" USERNAME="YourUsername" 
PASSWORD="AM2Ykm5dOYX3LA=aME" />
<AMFTP ACTIVITY="synchronize_folder" 
REMOTEFOLDER="/home/myFolders/FTPFolder1" 
LOCALFOLDER="C:\myFolders\Folder1" 
RESULTDATASET="theDataset" FTPSERVERTIMEZONE="(UTC-11:00) 
Coordinated Universal Time-11 [UTC-11]" 
SUBFOLDERS="YES" SESSION="YourFTPSession" />
<AMFTP ACTIVITY="logoff" SESSION="YourFTPSession" />