Folder Monitor Event

EFT’s Folder Monitor Event Rule trigger is used to detect the creation, deletion, and renaming of files in a monitored folder and to perform Actions based on these triggers. You can use a Folder Monitor Rule to trigger when files are added to a folder using the network file system. When monitoring folders for files added to EFT via the FTP/S, SFTP, and HTTP/S protocols, use File Uploaded, File Downloaded, and other File System Events. Folder Monitor Rules are not fired for Events happening to folders, such as the addition, renaming, or removal of a folder; it only applies to file changes within the folder or subfolders.

The Folder Monitor Rule can pass Unicode filenames to the Event Rule system, including the Advanced Workflow Engine, Custom Commands, text-based log files, and ARM. The Unicode filename will be saved in the auditing database, but the reporting tool cannot display Unicode filenames.

Folder Sweep

Occasionally, file system notification will fail (e.g., due to network errors), so files added to the monitored folder are missed and not processed (e.g., not moved to another location) if the Rule is using only notifications to detect files. After the Folder Monitor Rule is created, the Event Rule system can periodically poll the monitored folder (and subfolders, if specified) to ensure that all files have been processed. This "Folder Sweep" feature is allowed only for "file added" Actions. The Folder Sweep polling occurs at a user-specified frequency. Immediately upon Site or Event Rule start, the initial polling occurs and will trigger any Actions added to the Rule. Folder Sweep is enabled by selecting the Scan for files every check box in the Monitor Folder dialog box. If the check box is not selected, the associated frequency controls are disabled. Refer to the procedure below for instructions for enabling Folder Sweep.

A new Event type named "Folder Monitor – sweep" is defined and used to populate the eventType field in the auditing database when reporting Folder Monitor Rules that were triggered because of Folder Sweep. Also, the Folder Sweep archiving of files will be recorded using the EVENT_ACTIONS value of EVENT_ACTION_FS_ARCHIVED.

The following table describes the Folder Sweep information entered in the log:

Log Level

Event

Debug

  • When a Folder Monitor Rule starts execution, log which triggering mechanism(s) are being employed and whether subfolders are being monitored. Also log:

  • If folder sweep is on, show frequency, time units, and archive subfolder name.

  • If RDCW* is on, show whether health check is on and its frequency.

  • When a monitored folder is polled for its contents with special indication for the first poll.

  • Log which mechanism, RDCW notification or folder polling, triggers the processing of a file.

  • Log when file has been archived.

  • Log when file is still in folder after Event Rule Actions have completed and user chose not to archive.

  • Record trigger collisions by logging if Event is being ignored because file is already in process.

  • For folder sweep, log when folder contents have been received and are about to be processed.

*RDCW = ReadDirectoryChangesW function; Retrieves information that describes changes within a specified directory.

Error

  • Log reason for archive folder creation failure.

  • Log reason for file archive Action failure.

Risks associated with Folder Sweep include:

  • If you do not use the archive feature and the file is not removed from the Monitored Folder due to an Action failure, the file will be reprocessed in the next Folder Sweep cycle.

  • If the Health Check fails, it is possible to see duplicate Folder Monitor errors in the log.

  • If the Event Rule has been placing files in the Archive subfolder specified in the Folder Monitor and then you change the name of the Archive subfolder, files that were previously archived by Folder Sweep will be reprocessed.

  • If multiple Folder Monitor Rules point to same folder, a "race condition" can occur when the two Rules attempt to concurrently process the same file.

  • Folder Monitor does not trigger when Unicode filenames are added to the monitored folder; however, Folder Sweep archives them. Refer to Unicode Support in EFT for more information.

Archiving

After all Folder Monitor Rule Actions have been executed and if the archive option is enabled, the Folder Monitor Rule will determine whether a file is still in the monitored folder. For this reason, Rule Actions are forced to Stop processing so that execution returns to the Rule only after all Actions have finished. If the file is still in the folder, the Folder Monitor Rule creates the Archive subfolder (if not there already) in the folder containing the file to be archived. If an error occurs while creating the Archive subfolder, a message containing the failure reason will be logged; otherwise, the file is moved from the monitored folder into the Archive subfolder. If an error occurs during archival, a message containing the failure reason is logged. Whatever the reason, if a file’s archival fails, the file is left alone. If the archive feature is not enabled, files are left in the monitored folder, if Event Rule Actions have not otherwise disposed of them. Archive folders will have the same permissions as their parent folders and will not be given special attributes for connecting clients.

Creating a Folder Monitor Rule

EFT keeps track of the number of active threads over time and periodically calculates the average number of concurrent active threads during that time. The sample rate is once every 5 seconds, and the sample period is 10 samples. After sampling 10 times and finding the average concurrent active threads over that period, the system can grow the pool of the concurrent active threads, up to a set maximum number of threads. This means that if EFT is currently running close to or above the prior average of concurrent threads, it will grow the thread pool to allow for room for more Events. By default, EFT starts with 3 threads in the pool per Site, and can grow to a maximum of 32 threads.

EFT will only reset affected (modified) folders when applying configuration changes to an Event Rule, rather than resetting all folders.

When monitoring a folder, EFT watches for any file being added to, removed from, or renamed in the monitored folder. Moving a file, performing OpenPGP operations, and other Actions can trigger the Rule again, resulting in failures. This can be avoided by selecting the Stop processing this rule check box after if action failed then.

The Require Active Directory domain trust relationship check box is cleared by default for new installs and selected by default when upgrading from a version prior to EFT v6.4, if the FolderMonitorUseNonInteractiveLogon registry entry is present during the upgrade. The Scan for files every check box is not selected and associated controls are disabled. All other control settings are carried over from existing Rules during upgrade (health check yes/no and rate, subfolders yes/no, login credentials).

To configure a Folder Monitor Rule

  1. Open the Create New Event Rule dialog box.

  2. In the Create New Event Rule dialog box, click Folder Monitor, and then click OK.

    The new, blank Rule appears in the Rule Builder.

  3. In the Monitor folder Event, click [select]. The Monitor Folder dialog box appears.

  4. Next to the Folder box, click the folder icon  to specify a folder to monitor.

    To monitor a folder on a remote, non-EFT FTP server, supply the full UNC path to the network share. (The format for a UNC path is \\server\volume\directory and is not case-sensitive. For example: \\Shared1_svr\Shared1\WGroups\Network). Make sure that the EFT service has sufficient privileges to perform READ operations on the remote share. If you are using the "health check" feature, it must also have WRITE permissions. This is generally easiest if you set the EFT service to run as a domain account, or specify a dedicated “run as” account in the Monitor Folder dialog box. Wildcards are not supported.

  5. If you also want to monitor subfolders, select the Include subfolders check box. For example, if you are monitoring a user folder and the user has created subfolders, unless you select the Include subfolders check box, files added to or changed in subfolders do not trigger the Rule.

  6. If login credentials are required to access the folder and subfolders, select the Use the following credentials to access the monitored folder check box, then specify the username and password.

    The Microsoft definition of noninteractive login states: “Noninteractive authentication can only be used after an interactive authentication has taken place. During noninteractive authentication, the user does not input logon data; instead, previously established credentials are used. Noninteractive authentication is the mechanism at work when a user connects to multiple computers on a network without having to re-enter logon information for each computer.” In this case, EFT has joined the domain and/or the Server service runs as a domain user. You could supply different credentials to run as a different user for this Action.

  7. The Require Active Directory domain trust relationship check box specifies how the Folder Monitor Event Rule will log in to monitor remote folders. Selecting this check box indicates that Folder Monitor must establish a "trustful" connection to the system containing the folder(s) being monitored. This control is not enabled unless the Use the following credentials to access the monitored folder check box is selected. (Please also refer to the note above regarding this check box.)

  8. In the Triggers area, select the Trigger based on folder change notifications check box to cause Events to be set off by the receipt of directory change notifications (add, delete, and rename) generated by the system.

  9. To monitor the status of the network connection and report failures, select the Perform health check every check box, and specify an interval. An hour (60 minutes) is specified by default.

    10. When the check box is selected, EFT periodically writes a special file to the folder specified and then waits for the "file added" notification to verify that it can receive notifications of changes within the folder. When there is a loss of connectivity, EFT attempts to re-establish a link to the folder and triggers the Folder Monitor Failed Event internally. If you want to receive e-mail failure notifications (or other Actions) when the Folder Monitor health check returns a connection failure, create an additional Event Rule using the Folder Monitor Failed Event, and add the Send notification e-mail Action to it.

    The time EFT waits for the notification from Windows when a Folder Monitor health check file is created can be controlled by a registry value. Refer to the knowledgebase article at https://kb.globalscape.com/KnowledgebaseArticle10682.aspx.

  10. To enable Folder Sweep, select the Scan for files every check box and specify the frequency. The default is 30 minutes. A value between 1 and 9999 can be specified with units of seconds, minutes, or hours. The timer for the next sweep cycle is not started until all the files for the current sweep cycle have processed through all Event Rule Actions. Folder Sweep limits its processing to 1000 files at a time. If the monitored folder contains more than 1000 files, up to 1000 of the remaining files will be processed during the next sweep cycle. Selecting the Scan for files every check box will cause a Folder Monitor scan upon Event Rule start up (such as when you create the Rule and then click Apply). If you have Actions in the Rule, such as an e-mail notification, those Actions will be triggered. (This check box is not selected by default.) Selecting the Scan for files every check box causes the Event Rule's If File Change Condition to be set to does equal to added.

  11. All files in a monitored folder will be processed every sweep cycle so if a user neglects to remove processed files or if a Rule Action that was supposed to remove the file fails, the file will be reprocessed. In the Post Processing area, select the Once all actions are completed, archive any files still present in the monitored folder to avoid reprocessing check box, and then specify the name of the folder in which to archive any remaining files. The default is EFTArchive. The Archive subfolder will reside directly under the folder in which the file was added. The Archive subfolder name cannot contain any of the following characters:  | /  \ ? * < " : > + [  ] and is limited to 248 characters. (The total cannot exceed Windows path limit.)

    • Select the Include timestamp in archived filenames check box to avoid overwriting any files of the same name in the Archive subfolder. The file name will be appended using the Event Rule variables %EVENT.DATESTAMP% and %EVENT.TIMESTAMP_PRECISE% (time to the millisecond).

    • If Folder Sweep is enabled and you have specified an Archive subfolder, the Archive subfolder is ignored when Include subfolders is enabled.

    • If you change the name of the Archive subfolder, the existing Archive subfolders will be unaltered. If processing of subfolders is enabled, notifications and polling for contents of the former Archive subfolders will begin immediately upon applying the Rule changes.

  12. Click OK. If the Once all actions check box is selected and an invalid name or no name is given for the Archive subfolder, it will revert to the default name (EFTArchive) and a warning message appears.

  13. The If File Change Condition is added automatically to restrict the triggering of the Rule. Click the links in the If File Change Condition to specify whether the Rule should trigger when a file in the folder is or is not renamed, added, or removed. If Folder Sweep (the Scan for files every check box) is enabled (as described above), the If File Change Condition is forced to does equal to added because Folder Sweep only applies to files added to a folder or subfolders.

  14. Specify any Action/Conditions to occur when this Event is triggered:

  15. Click Apply to save the changes on EFT.

Folder Monitor Failure

To audit failures of Folder Monitor Rules, use the Folder Monitor Failed Event, then add the If Folder Monitored Failure reason Condition.

  • Click the reason link to specify a failure reason that will trigger the Rule: any failure, archive failure, health check failed.

Folder Monitor archive folder errors will also trigger this Event and write to the Windows Event log.