Policies are system entities which generate rules or transfer requests based on matching particular dataset metadata at registration time. Polices are owned by an account and can only generate rules for that account. Policies may have a lifetime, after which they will expire.
An example of a policy is given below:
Attribute | Value |
---|---|
Owner | tzero |
match | project=data11 7TeV, dataType=RAW, stream=physics* |
rule | 1@CERNTAPE, 1@T1TAPE |
lifetime | 2012-01-01 00:00 |
Policies can also create transfer primitives, so generate extra copies of data as it is produced:
Attribute | Value |
---|---|
Owner | prod |
match | project=mc11 7TeV, dataType=merge.AOD, tag=*(p795|p796|p805)*, replicationPolicy=RPValue |
rule | 1@T1DISK, 1@T2DISK, |
transfer | 1@T1DISK, 2@T2DISK |
lifetime | 2011-12-01 00:00 |
In this case the transfer request is for extra copies, in addition to those set by rules. (This is different behaviour to that for rules themselves, which are always independent.)
Rucio Subscriptions / Policies (name TBD) exist for the purpose of making data placement decisions before the actual data has been created. In the current DQ2 system there are basically 3 applications which are responsible for this behavior:
See https://twiki.cern.ch/twiki/bin/viewauth/Atlas/DataTransferRequestInterface
DaTRI is a data-transfer tool which basically offers two functionalities:
Data is identified by dataset-names, patterns, container-names or patterns. Requests can either finish after a transfer is complete or stay active to look for newly-created data (which match a request pattern) to continously transfer the data to a site.
All requests (there are exceptions) have to be approved by either a DaTRI admin or a cloud coordinator.
There also exists a website which shows the status of all requests.
TODO
TODO
In order to represent all (or most) of this functonality in Rucio in a generic way, the respective steps of the workflow of the current applications have to be identified and described. At the moment we spotted three fundamental steps for all the applications which should be described in the following section.
Based on:
Specific destination endpoint; However, based on the endpoint the request has to be approved by an admin or cloud coordinator.
The output generation can be based on: