7 The [Dataset Stager (afdsmgrd)](http://afdsmgrd.googlecode.com/) is
8 a daemon that coordinates the transfer of data from a remote storage
11 For each file to transfer, a script is called. The script can be
12 customized to support your source and destination protocol.
14 Staging requests are issued from the ROOT console, where you can also
15 control the progress of your staging. How to request stagings and how to
16 check current transfer progress from ROOT is explained in the [PROOF interface
17 to AliEn file catalog documentation](TDataSetManagerAliEn.html).
22 The Dataset Stager is distributed both on a repository on its own and as
23 part of ROOT. The easiest way to compile it is to do it inside ROOT.
28 When configuring the ROOT source, enable the Dataset Stager by adding
29 `--enable-afdsmgrd`. Check in the list of enabled features if you have
32 After running `make` (and, optionally, `make install`) you'll find the
33 daemon in the same directory of `root.exe`.
35 The configuration file and init.d startup script will be in
36 `$ROOTSYS/etc/proof`. The daemon can and **must** run as unprivileged
42 The Dataset Stager can share its configuration file with PROOF, as
43 some directives are the same and unknown directives are just ignored.
45 Directives are one per line and lines beginning with a pound sign (`#`)
46 are used for comments.
48 > The configuration file is automatically checked at each loop: this
49 > means you can change configuration without restarting the daemon or
50 > stopping your current transfers.
52 A detailed description of each directive follows.
55 : This statement will substitute every occurrence of `$VARIABLE` with
56 its *value* in the rest of the configuration file. You can have
57 multiple `set` statements.
59 xpd.stagereqrepo [dir:]*directory*
60 : This directive is shared with PROOF: *directory* is the full path to
61 the dataset repository. **Defaults to empty:** without this
62 directive the daemon is not operative.
64 The `dir:` prefix is optional.
66 dsmgrd.purgenoopds *true|false*
67 : Set it to *true* **(default is false)** to remove a dataset when no file to stage
68 is found. If no file to stage is found, but corrupted files exist, the
69 dataset is kept to signal failures. Used in combination with `xpd.stagereqrepo`
70 makes it "disposable": only the datasets effectively needed for signaling
71 the staging status will be kept, improving scalability and stability.
73 dsmgrd.urlregex *regex* *subst*
74 : Each source URL present in the datasets will be matched to *regex*
75 and substituted to *subst*. *regex* supports grouping using
76 parentheses, and groups can be referenced in order using the dollar
77 sign with a number (`$1` for instance) in *subst*.
79 Matching and substitution for multiple URL schemas are supported by
80 using in addition directives `dsmgrd.urlregex2` up to
81 `dsmgrd.urlregex4` which have the same syntax of this one.
83 Example of URL translation via regexp:
85 > - Configuration line:
87 > dsmgrd.urlregex alien://(.*)$ root://xrd.cern.ch/$1
91 > alien:///alice/data/2012/LHC12b/000178209/ESDs/pass1/12000178209061.17/AliESDs.root
95 > root://xrd.cern.ch//alice/data/2012/LHC12b/000178209/ESDs/pass1/12000178209061.17/AliESDs.root
97 dsmgrd.sleepsecs *secs*
98 : Seconds to sleep between each loop. The dataset stager checks at
99 each loop the status of the managed transfers. Defaults to **30
102 dsmgrd.scandseveryloops *n*
103 : Every `n` loops, the dataset repository is checked for newly
104 incoming staging requests. Defaults to **10**.
106 dsmgrd.parallelxfrs *n*
107 : Number of concurrent transfers. Defaults to **8**.
109 dsmgrd.stagecmd *shell\_command*
110 : Command to run in order to stage each file. It might be whatever you
111 want (executable, shell script...). If you add `$URLTOSTAGE` and/or
112 `$TREENAME` in the *shell\_command*, they'll be substituted
113 respectively with the destination URL and the default ROOT tree name
114 in the file (as specified in the dataset staging request from ROOT).
118 dsmgrd.stagecmd /path/to/afdsmgrd-xrd-stage-verify.sh "$URLTOSTAGE" "$TREENAME"
120 Return value of the command is ignored: standard output is
121 considered, as explained here.
123 Defaults to `/bin/false`.
125 dsmgrd.cmdtimeoutsecs *secs*
126 : Timeout on staging command, expressed in seconds: after this
127 timeout, the command is considered failed and it is killed (in first
128 place with `SIGSTOP`, then if it is unresponsive with `SIGKILL`).
129 Defaults to **0 (no timeout)**.
131 dsmgrd.corruptafterfails *n*
132 : Set this to a number above zero to tell the daemon to mark files as
133 corrupted after a certain number of either download or verification
134 failures. A value of **0 (default)** tells the daemon to retry
137 Configuring the MonALISA monitoring plugin
138 ------------------------------------------
140 The Dataset Stager supports generic monitoring plugins. The only plugin
141 distributed with the stager is the MonALISA monitoring plugin.
143 dsmgrd.notifyplugin */path/to/libafdsmgrd\_notify\_apmon.so*
144 : Set it to the path of the MonALISA plugin shared object. By default,
145 notification plugin is disabled.
147 dsmgrd.apmonurl *apmon://apmon.cern.ch*
148 : This variable tells the ApMon notification plugin how to contact one
149 or more MonALISA server(s) to activate monitoring via ApMon. It
150 supports two kinds of URLs:
152 - `http[s]://host/path/configuration_file.conf` (a remote file
153 where to fetch the list of servers from)
155 - `apmon://[:password@]monalisahost[:8884]` (a single server to
158 If the variable is not set, yet the plugin is loaded, MonALISA
159 monitoring is inhibited until a valid configuration variable is
162 dsmgrd.apmonprefix *MY::CLUSTER::PREFIX*
163 : Since MonALISA organizes information in "clusters" and "hosts", here
164 you can specify what to use as cluster prefix for monitoring
165 datasets information and daemon status. If this variable is not set,
166 MonALISA monitoring is inhibited. Please note that the suffix
167 `_datasets` or `_status` is appended for each of the two types of
170 A sample configuration file
171 ---------------------------
173 xpd.stagereqrepo /opt/aaf/var/proof/datasets
174 dsmgrd.purgenoopds true
175 dsmgrd.urlregex alien://(.*)$ /storage$1
177 dsmgrd.scandseveryloops 30
178 dsmgrd.parallelxfrs 10
179 dsmgrd.stagecmd /opt/aaf/bin/af-xrddm-verify.sh "$URLTOSTAGE" "$TREENAME"
180 dsmgrd.cmdtimeoutsecs 3600
181 dsmgrd.corruptafterfails 0
ROOT 6.10/02 - Reference Guide Generated on Thu Jul 13 2017 10:25:48 using Doxygen 1.8.11.
