File Transfer Plug-Ins

See the admin manual for an overview of file transfer plug-ins; this part of the manual details the interface between HTCondor and a file transfer plug-in.

There are three versions of this interface. The first version, retroactively protocol version 1, invoked the plug-in once for each file being transferred; this was simple and easy, but not very efficient. In the second version – introducing protocol version 2 – “multi-file” plug-ins could transfer more than one file per invocation. Protocol version 4 extends version 2 with additional information in the plug-in’s input, allowing for more precise control (e.g., over which caches a particular transfer method uses). It also explicitly requires some behaviors intended to minimize the need for future protocol version changes.

Protocol Version 1

Although protocol version 1 is still supported, new plug-ins should not be written to use it; the example plug-in supports version 2, so the additional complexity should not be a burden.

Protocol Version 2

Presently, a file transfer plug-in may be invoked at six different times:

  1. When the EP initially starts up, to determine which protocols the EP should advertise as being supported.

  2. When the EP initially starts up, to test the plug-in, if that is enabled. (See <PLUGIN>_TEST_URL.)

  3. Just before file transfer begins for a job. This is a duplicate of item #1.

  4. Just after item #3. This is a duplicate of item #2.

  5. Once when doing input transfer, for all supported protocols.

  6. At least once for output tansfer, no more than once per supported protocol.

We call the interface for instances 1 and 3 the “query” interface, and the interface for instances 2, 4, 5, and 6 the “transfer” interface.

Plug-ins specified by transfer_plugins are not (can not be) run by instances 1 or 2.

The Query Interface (v2)

Instead of requiring the tedious (and static) specification of everything HTCondor needs to know about a file transfer plug-in in configuration, HTCondor invokes the plug-in with the -classad command-line argument and parses the ClassAd written to standard out. Presently, the ClassAd must be serialized in the “long” (line-oriented) format.

The ClassAd must contain four attributes, and should contain five. Extra attributes will be ignored, but preserved in the epoch log. Those attributes are:

  • MultipleFileSupport which must be the boolean True;

  • PluginVersion, an arbitrary string potentially useful for debugging;

  • PluginType, which must be the string value FileTransfer;

  • SupportedMethods, a string containing a comma-separated list of the method(s) (protocol(s), scheme(s)) the plug-in supports;

  • and PluginVersion, an integer which must be the integer 2.

This can be optionally followed by 2 or more attributes that give the plugin the ability to put arbitrary information, such as the plugin version, into the EP’s slot ads. One attribute must be StartdAttrs, a string containing a comma-separated list of one or more attribute names; the others must be the named attributes.

For example, the “curl” plug-in might produce the following output:

MultipleFileSupport = true
PluginVersion = "0.2"
PluginType = "FileTransfer"
ProtocolVersion = 2
SupportedMethods = "http,https,ftp,file,dav,davs"
StartdAttrs = "CurlPluginInfo"
CurlPluginInfo = [ CurlVersion = "7.15.1"; ]

The Transfer Interface (v2)

If a job specifies a URL transfer, the plug-in is invoked without the command line argument -classad. It will instead be given two other command line arguments, -infile <input-filename> and -outfile <output-filename>. The former specifies a path to file on disk which contains a series of input ads; the latter specifies the file to which the plug-in should write its output ads. Plug-ins must default to “download” (or “input”) mode, where URLs are transferred to local files; if -upload is specified, local files are transferred (“uploaded”) to URLs, instead.

The plug-in is expected to do the transfer, exiting with status 0 if the transfer was successful, and a non-zero status if the transfer was not successful. If the transfer was not succesful, the job will usually be placed on hold; but see The Output File (v2) below. Regardless, the job ClassAd attribute HoldReasonSubCode will be set to the exit status of the plug-in.

The Input File (v2)

The input file is a sequence of ClassAds serialized in the “new” format; see ClassAds: Old and New for details. There will be one ad for each URL (if downloading) or file (if uploading). Each ClassAd will contain the following attributes:

  • URL, a string containing the URL to transfer to/from.

  • LocalFileName, a string containing the path to transfer to/from.

(Remember that ClassAd attribute names are not case-sensitive.) Plug-ins must parse the complete ClassAd language, and not just the subset any particular version of HTCondor generates; see ASSUME_COMPATIBLE_MULTIFILE_PLUGINS; the example plug-in does this. Plug-ins must ignore any attribute(s) they don’t know how to deal with; it is HTCondor’s responsibility not to pass attributes that shouldn’t be ignored. (For example: if a later version of the protocol were to require plug-ins to honor an “untar” flag, HTCondor should not pass that flag to an earlier version, because the user’s job may depend on that URL being untarred.)

As a reminder: the input file does not specify which direction the transfer is going; that is done by the absence or presence of the -upload command line option.

By default, later versions of HTCondor may include additional attributes in each input ad, based on attributes (optionally) set in the job ad. This feature is not presently documented because we expect it to change – and conformant plug-ins will ignore the extra attributes.

The Output File (v2)

Like the input file, the output file is a sequence of ClassAds serialized in the “new” format; there must be one ad for each ad in the input file (and thus for each file or URL, depending on the direction of transfer).

Note

The output file is pre-allocated! When you open the output file, do not truncate it. The output file is pre-allocated so that if the transfer plug-in runs out of disk space, it still has room to tell HTCondor what happened. (If the plug-in cleans up partial transfers, or if the disk was full but space becomes available before HTCondor has a chance to look around, it might not be able to tell, so this ability is important.) Carefully consider allocating additional space to the output file (e.g., write more spaces) if your output comes close to filling it up.

The following attributes are required:

  • TransferFileName, a string. Must be a LocalFileName from an input ad.

  • TransferURL, a string. Must be a URL from the input ad.

  • TransferSuccess, a bool. True if and only if the transfer of this URL to/from this file name succeeded. If false, the TransferError attribute must also be set.

The TransferFileName / TransferURL pair must correspond to one of the LocalFileName / URL pairs from the input ads.

The following attributes are optional:

  • TransferTotalBytes, an integer. Specifies how many bytes were sent or received to perform the transfer, which may differ from the number of bytes read from or written to the file.

  • TransferError, a string. If TransferSuccess is false, this must be set. It should be meaningful to the job submitter.

  • DeveloperData, a ClassAd.

  • TransferErrorData, a list of ClassAds.

The DeveloperData ClassAd is copied to the epoch history log and otherwise ignored; it is intended to help plug-in developers debug problems.

Each ClassAd in TransferErrorData specifies additional data about the nature of the failed transfer(s); this data is intended to allow HTCondor to make better decisions(s) about how the handle the failure(s). For example, a plug-in may be told that a particular service is down but should be back up in a few minutes. The transfer fails, because that’s longer than the plug-in is willing to wait, but TransferErrorData allows the plug-in to tell HTCondor that the transfer might work if tried again in a few minutes – as opposed to putting the job on hold, as would probably otherwise be the case.

See Transfer Error Data for details.

Protocol Version 4

Protocol version 4 is a small evolution of protocol version 2. (There is no protocol version 3.)

The Query Interface (v4)

The query interface is identical, except that the plug-in must not add attributes to the query ad that might conflict with attributes defined in later versions of this document. We’ll try not to specify attribute names that begin with _, but the safest thing to do is not generate any attributes other than those named in this specification.

The Transfer Interface (v4)

The transfer interface is identical, with the following exceptions.

The Input File (v4)

  • Instead of duplicating information from the job ad into each individual input ad, HTCondor provides additional ads in the input file. These ads do not correspond to a particular input file (or URL). Each individual input ad may still have additional attributes, but they will be specific to that file. (For example, the user may wish tarball T to be untarred, but not zip file Z, because the former contains a compressed application and the latter is itself an input to that application.) This feature is not presently documented because we expect it to change – and conformant plug-ins will ignore the extra attributes.

  • Ads that don’t correspond to a particular input file (or URL) will not have the URL or LocalFileName attributes.

  • Ads that don’t correspond to a particular input file (or URL) may be present anywhere in the input file. However, these ads must only affect file ads after them in the input file.

The Output File (v4)

  • The TransferErrorData attribute is mandatory in version 4.

Further Protocol Versions

The intent is to change protocol version numbers sparingly, and only when plug-ins conformant to the previous version might break. To achieve this goal, we will from time to time specify additional (optional) attributes for the various ads that will change how HTCondor interacts with the plug-in. (To continue the previous example, we might add a ‘’HasUntar’’ boolean to the query interface which causes the ‘’MustUntar’’ flag to be set in some specific transfer ads; we might further expect the plug-in to explicitly note in the corresponding output ad if the untarring was succesful.)