StoRM WebDAV User Guide

The WebDAV protocol
StoRM WebDAV
Examples

The WebDAV protocol

Web Distributed Authoring and Versioning (WebDAV) protocol consists of a set of methods, headers, and content-types ancillary to HTTP/1.1 for the management of resource properties, creation and management of resource collections, URL namespace manipulation, and resource locking. The purpose of this protocol is to present a Web content as a writable medium in addition to be a readable one. WebDAV on Wikipedia and the WebDAV website provide information on this protocol.

In a few words, the WebDAV protocol mainly abstracts concepts such as resource properties, collections of resources, locks in general, and write locks specifically. These abstractions are manipulated by the WebDAV-specific HTTP methods and the extra HTTP headers used with WebDAV methods. The WebDAV added methods include:

PROPFIND - used to retrieve properties, stored as XML, from a web resource. It is also overloaded to allow one to retrieve the collection structure (a.k.a. directory hierarchy) of a remote system.
PROPPATCH - used to change and delete multiple properties on a resource in a single atomic act.
MKCOL - used to create collections (a.k.a. a directory).
COPY - used to copy a resource from one URI to another.
MOVE - used to move a resource from one URI to another.
LOCK - used to put a lock on a resource. WebDAV supports both shared and exclusive locks.
UNLOCK - used to remove a lock from a resource.

While the status codes provided by HTTP/1.1 are sufficient to describe most error conditions encountered by WebDAV methods, there are some errors that do not fall neatly into the existing categories, so the WebDAV specification defines some extra status codes. Since some WebDAV methods may operate over many resources, the Multi-Status response has been introduced to return status information for multiple resources. WebDAV uses XML for property names and some values, and also uses XML to marshal complicated requests and responses.

StoRM WebDAV

From StoRM v.1.11.7 release, the StoRM service that provides valid WebDAV endpoints for each managed storage area is StoRM WebDAV.

StoRM WebDAV replaces the StoRM gridhttps service. All sites installing StoRM and providing HTTP and WebDAV endpoints should upgrade to the StoRM WebDAV service for improved performance and stability of the service as soon as possible.

Important: The StoRM WebDAV service is released and supported only on SL/CENTOS 6.

Installation and configuration

See the System Administration Guide to learn how to install and configure the service.

Endpoints

For each Storage Area, both/either a plain HTTP and/or a HTTP over SSL endpoint can be enabled. The default ports are 8085 (HTTP) and 8443 (HTTPS). All the following URLs are valid endpoints for a storage area:

http://example.infn.it:8085/storage_area_accesspoint
https://example.infn.it:8443/storage_area_accesspoint

To fully support the old StoRM GridHTTPs webdav endpoints, used until StoRM v1.11.6, all the URLs with webdav context path are accepted by StoRM WebDAV:

http://example.infn.it:8085/webdav/storage_area_accesspoint
https://example.infn.it:8443/webdav/storage_area_accesspoint

Authentication and authorization

Users authentication within StoRM WebDAV is made through a valid VOMS proxy. All the users that provide a valid x509 VOMS proxy are authorized to access all the content of the storage area in read/write mode.

The most common way to authenticate and be authorized to read/write data into a Storage Area is by providing the right VOMS credentials through a valid VOMS Proxy. Otherwise, through the definition of a VOMS map file, a Storage Area can be configure to accept the list of VO members as obtained by running the voms-admin list-users command. When VOMS mapfiles are enabled, users can authenticate to the StoRM webdav service using the certificate in their browser and be granted VOMS attributes if their subject is listed in one of the supported VOMS mapfile. For each supported VO, a file having the same name as the VO is put in the voms-mapfiles directory (/etc/storm/storm-webdav/vo-mapfiles.d).

Example: to generate a VOMS mapfile for the cms VO, run the following command

  voms-admin --host voms.cern.ch --vo cms list-users > cms

See more details here. Read permissions of the content of a storage area can also be extendend to anonymous user (it’s disabled by default).

Notes

Both the old storm-gridhttps-server and the new storm-webdav components implements WebDAV protocol by using Milton open source java library.

milton

Examples

The most common WebDAV clients are:

browsers
command-line tools like cURLs and davix
a third-party GUI

Currently, users are used to connect to a WebDAV endpoint providing a valid username and password or as anonymous users (if supported). But, as seen in the Authentication and authorization paragraph, in our case the most common use case is providing a valid VOMS proxy. The VOMS proxies are supported only by command-line tools. Browsers can be used to navigate into the storage area content in the some cases:

if VO users access through their x509 certificate is enabled (HTTPS endpoint)
if anonymous read-only access is enabled (HTTP endpoint)

The use of a third party client (in read only mode) can happen only if anonymous read is enabled.

You can also develop a client on your own, for example by using the Apache Jackrabbit API.

The following paragraphs will give an example for each WebDAV/HTTP method by using cURLS and DAVIX client command line tools. cURL is a command line tool for transferring data with URL syntax (see cURL website).

All the requests have been done:

against our WebDAV test endpoint https://omii006-vm03.cnaf.infn.it:8443/
using test0.p12 of our igi-test-ca credentials:

$ yum install igi-test-ca
$ cp /usr/share/igi-test-ca/test0.p12 $HOME
$ chmod 600 test0.p12

after the creation of a valid VOMS proxy for test.vo VO

$ cd $HOME
$ voms-proxy-init --voms test.vo --cert test0.p12

Download file

Having the remote file:

/test.vo/test.txt

Hello world

use:

$ curl https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
    --cert $X509_USER_PROXY \
    --capath /etc/grid-security/certificates \
    --cacert /usr/share/igi-test-ca/test0.cert.pem

$ davix-get -P Grid https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt

Hello world

Download multiple file ranges

Having the remote file:

/test.vo/test.txt

Hello world

use:

$ curl https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
    --cert $X509_USER_PROXY \
    --capath /etc/grid-security/certificates \
    --cacert /usr/share/igi-test-ca/test0.cert.pem \
    -H "Range: 0-1,3-4,8-10"

$ davix-get -P Grid https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
    -H "Range: 0-1,3-4,8-10"


--jetty1913974379i77mn8om
Content-Type: text/plain
Content-Range: bytes 0-1/12

He
--jetty1913974379i77mn8om
Content-Type: text/plain
Content-Range: bytes 3-4/12

lo
--jetty1913974379i77mn8om
Content-Type: text/plain
Content-Range: bytes 8-10/12

rld
--jetty1913974379i77mn8om--

Upload file

$ curl -T test.txt https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
    --cert $X509_USER_PROXY \
    --capath /etc/grid-security/certificates \
    --cacert /usr/share/igi-test-ca/test0.cert.pem

$ davix-put -P Grid text.txt https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt

With increased verbosity:

> PUT /test.vo/test.txt HTTP/1.1
> User-Agent: libdavix/0.4.0 neon/0.0.29
> Keep-Alive: 
> Connection: TE, Keep-Alive
> TE: trailers
> Host: omii006-vm03.cnaf.infn.it:8443
> Content-Length: 12
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 201 Created
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< X-Frame-Options: DENY
< Set-Cookie: JSESSIONID=abwhzs5fhrwyf3jn2fb9u1uf;Path=/;Secure
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Date: Fri, 13 Mar 2015 13:22:22 GMT
< Accept-Ranges: bytes
< ETag: "/storage/test.vo/test.txt_323800060"
< Transfer-Encoding: chunked
<

Check if resource exists

To check if a resource exists without download any data in case of a file, the HTTP HEAD method is used. HEAD acts like HTTP/1.1, so HEAD is a GET without a response message body.

$ curl -X HEAD https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
    --cert $X509_USER_PROXY \
    --capath /etc/grid-security/certificates \
    --cacert /usr/share/igi-test-ca/test0.cert.pem

$ davix-http -P Grid -X HEAD https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt

With increased verbosity:

< HTTP/1.1 200 OK
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< X-Frame-Options: DENY
< Set-Cookie: JSESSIONID=1g5r0pqndsonh3kvlcz6nqhrc;Path=/;Secure
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Content-Type: text/plain
< Last-Modified: Fri, 13 Mar 2015 13:22:22 GMT
< Content-Length: 12
< Accept-Ranges: bytes
<

Create directory

curl -X MKCOL https://omii006-vm03.cnaf.infn.it:8443/test.vo/testDir \
    --cert $X509_USER_PROXY \
    --capath /etc/grid-security/certificates \
    --cacert /usr/share/igi-test-ca/test0.cert.pem

$ davix-mkdir -P Grid https://omii006-vm03.cnaf.infn.it:8443/test.vo/testDir

With increased verbosity:

< HTTP/1.1 201 Created
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< X-Frame-Options: DENY
< Set-Cookie: JSESSIONID=1vaxmyqwph6bx17glea63khsfy;Path=/;Secure
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Transfer-Encoding: chunked
<

Delete file

curl -X DELETE https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
    --cert $X509_USER_PROXY \
    --capath /etc/grid-security/certificates \
    --cacert /usr/share/igi-test-ca/test0.cert.pem

$ davix-rm -P Grid https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt

With increased verbosity:

< HTTP/1.1 204 No Content
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< X-Frame-Options: DENY
< Set-Cookie: JSESSIONID=1xd4oo2aao53i1abg7b8no4nae;Path=/;Secure
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Date: Fri, 13 Mar 2015 13:55:07 GMT
< Accept-Ranges: bytes
< ETag: "/storage/test.vo/test.txt_0"
<

Copy or duplicate file

COPY method duplicates the resource identified by the Request-URI to the resource identified by the Destination header URI. Destination header MUST be present. If destination resource already exists the Overwrite header value is evaluated. Available values are T (true, default) and F (false).

$ curl -X COPY https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
--cert $X509_USER_PROXY \
--capath /etc/grid-security/certificates \
--cacert /usr/share/igi-test-ca/apostrofe.cert.pem \
-H "Destination: https://omii006-vm03.cnaf.infn.it:8443/test.vo/test3.txt"

$ davix-cp -P Grid https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
https://omii006-vm03.cnaf.infn.it:8443/test.vo/test2.txt

With increased verbosity:

> COPY /test.vo/test.txt HTTP/1.1
> User-Agent: libdavix/0.4.0 neon/0.0.29
> Keep-Alive: 
> Connection: TE, Keep-Alive
> TE: trailers
> Host: omii006-vm03.cnaf.infn.it:8443
> Destination: https://omii006-vm03.cnaf.infn.it:8443/test.vo/test2.txt
> X-Number-Of-Streams: 1
>
< HTTP/1.1 201 Created
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< X-Frame-Options: DENY
< Set-Cookie: JSESSIONID=dnyk3ni98vyj1bwe56n50fpif;Path=/;Secure
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Date: Fri, 13 Mar 2015 13:58:29 GMT
< Accept-Ranges: bytes
< ETag: "/storage/test.vo/test.txt_325854060"
< Transfer-Encoding: chunked

Move or rename file

As already seen for COPY method, also for every MOVE the Destination header MUST be present.

$ curl -X MOVE https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
--cert $X509_USER_PROXY \
--capath /etc/grid-security/certificates \
--cacert /usr/share/igi-test-ca/test0.cert.pem \
-H "Destination: https://omii006-vm03.cnaf.infn.it:8443/test.vo/test2.txt"

$ davix-mv -P Grid https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt https://omii006-vm03.cnaf.infn.it:8443/test.vo/test2.txt

With increased verbosity:

> MOVE /test.vo/test.txt HTTP/1.1
> User-Agent: libdavix/0.4.0 neon/0.0.29
> Keep-Alive: 
> Connection: TE, Keep-Alive
> TE: trailers
> Host: omii006-vm03.cnaf.infn.it:8443
> Destination: https://omii006-vm03.cnaf.infn.it:8443/test.vo/test2.txt
> 
< HTTP/1.1 201 Created
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< X-Frame-Options: DENY
< Set-Cookie: JSESSIONID=fcyai4anga5a73pcoyltszrl;Path=/;Secure
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Date: Mon, 16 Mar 2015 09:28:40 GMT
< Accept-Ranges: bytes
< ETag: "/storage/test.vo/test.txt_0"
< Transfer-Encoding: chunked
<

List directory

There are two ways to get the list of the resources into a remote directory:

the first is through an HTTP GET which returns a human readable version of all the content of the directory;
the second is through a WebDAV PROPFIND which returns a structured XML body.

The PROPFIND operation retrieves, in XML format, the properties defined on the resource identified by the Request-URI. Clients must submit a Depth header with a value of "0", "1", or "infinity" (default is "Depth: infinity").

Clients may submit through the body of the request a ‘propfind’ XML element. It’s used to describe what information is being requested:

a particular property value (by using the ‘prop’ element)

<?xml version="1.0" encoding="utf-8" ?>
  <D:propfind xmlns:D="DAV:">
    <D:prop xmlns:R="http://ns.example.com/boxschema/">
      <R:author/>
      <R:title/>
    </D:prop>
  </D:propfind>

In this example, the propfind XML element specifies the name of two properties whose values are being requested.

all property values (by using the ‘allprop’ element);

<?xml version="1.0" encoding="utf-8" ?>
  <D:propfind xmlns:D="DAV:">
    <D:allprop/>
  </D:propfind>

In this example, the request should return the name and value of all the properties defined by WebDAV specification plus the user defined properties.

the list of names of all the properties defined on the resource (by using the ‘propname’ element).

<?xml version="1.0" encoding="utf-8" ?>
  <propfind xmlns="DAV:">
    <propname/>
  </propfind>

To list all the content of a remote directory we can use the ‘allprop’ XML body, with depth equal to 1:

$ curl -X PROPFIND https://omii006-vm03.cnaf.infn.it:8443/test.vo \
   --cert $X509_USER_PROXY \
   --capath /etc/grid-security/certificates \
   --cacert /usr/share/igi-test-ca/test0.cert.pem \
   --data "<?xml version='1.0' encoding='utf-8' ?><D:propfind xmlns:D='DAV:'><D:allprop/></D:propfind>"

$ davix-http -P Grid -X PROPFIND https://omii006-vm03.cnaf.infn.it:8443/test.vo \
    --data "<?xml version='1.0' encoding='utf-8' ?><D:propfind xmlns:D='DAV:'><D:allprop/></D:propfind>"

The XML body received:

<?xml version="1.0" encoding="utf-8" ?>
<d:multistatus xmlns:d="DAV:" xmlns:cal="urn:ietf:params:xml:ns:caldav" xmlns:card="urn:ietf:params:xml:ns:carddav" xmlns:cs="http://calendarserver.org/ns/" xmlns:ns1="http://storm.italiangrid.org/2014/webdav">
   <d:response>
      <d:href>/test.vo/</d:href>
      <d:propstat>
         <d:prop>
            <d:supported-report-set />
            <d:isreadonly>FALSE</d:isreadonly>
            <d:iscollection>TRUE</d:iscollection>
            <d:name>test.vo</d:name>
            <d:getcontenttype />
            <d:resourcetype>
               <d:collection />
            </d:resourcetype>
            <d:getlastmodified>Mon, 30 Mar 2015 17:37:29 GMT</d:getlastmodified>
            <d:getcontentlength />
            <d:getcreated>2015-03-30T17:37:29Z</d:getcreated>
            <d:creationdate>2015-03-30T17:37:29Z</d:creationdate>
            <d:getetag>"/storage/test.vo_1807906532"</d:getetag>
            <d:displayname>test.vo</d:displayname>
         </d:prop>
         <d:status>HTTP/1.1 200 OK</d:status>
      </d:propstat>
   </d:response>
   <d:response>
      <d:href>/test.vo/test2.txt</d:href>
      <d:propstat>
         <d:prop>
            <ns1:Checksum>1cf20447</ns1:Checksum>
            <d:supported-report-set />
            <d:isreadonly>TRUE</d:isreadonly>
            <d:iscollection>FALSE</d:iscollection>
            <d:name>test2.txt</d:name>
            <d:getcontenttype>text/plain</d:getcontenttype>
            <d:resourcetype />
            <d:getlastmodified>Mon, 16 Mar 2015 07:27:35 GMT</d:getlastmodified>
            <d:getcontentlength>12</d:getcontentlength>
            <d:getcreated>2015-03-16T07:27:35Z</d:getcreated>
            <d:creationdate>2015-03-16T07:27:35Z</d:creationdate>
            <d:getetag>"/storage/test.vo/test2.txt_561712916"</d:getetag>
            <d:displayname>test2.txt</d:displayname>
         </d:prop>
         <d:status>HTTP/1.1 200 OK</d:status>
      </d:propstat>
   </d:response>
   <d:response>
      <d:href>/test.vo/testDir/</d:href>
      <d:propstat>
         <d:prop>
            <d:supported-report-set />
            <d:isreadonly>FALSE</d:isreadonly>
            <d:iscollection>TRUE</d:iscollection>
            <d:name>testDir</d:name>
            <d:getcontenttype />
            <d:resourcetype>
               <d:collection />
            </d:resourcetype>
            <d:getlastmodified>Fri, 13 Mar 2015 13:53:21 GMT</d:getlastmodified>
            <d:getcontentlength />
            <d:getcreated>2015-03-13T13:53:21Z</d:getcreated>
            <d:creationdate>2015-03-13T13:53:21Z</d:creationdate>
            <d:getetag>"/storage/test.vo/testDir_325658916"</d:getetag>
            <d:displayname>testDir</d:displayname>
         </d:prop>
         <d:status>HTTP/1.1 200 OK</d:status>
      </d:propstat>
   </d:response>
   <d:response>
      <d:href>/test.vo/test.txt</d:href>
      <d:propstat>
         <d:prop>
            <ns1:Checksum />
            <d:supported-report-set />
            <d:isreadonly>TRUE</d:isreadonly>
            <d:iscollection>FALSE</d:iscollection>
            <d:name>test.txt</d:name>
            <d:getcontenttype>text/plain</d:getcontenttype>
            <d:resourcetype />
            <d:getlastmodified>Mon, 16 Mar 2015 10:04:56 GMT</d:getlastmodified>
            <d:getcontentlength>12</d:getcontentlength>
            <d:getcreated>2015-03-16T10:04:56Z</d:getcreated>
            <d:creationdate>2015-03-16T10:04:56Z</d:creationdate>
            <d:getetag>"/storage/test.vo/test.txt_571153420"</d:getetag>
            <d:displayname>test.txt</d:displayname>
         </d:prop>
         <d:status>HTTP/1.1 200 OK</d:status>
      </d:propstat>
   </d:response>
</d:multistatus>

To list all the properties of a remote file we can use the same ‘allprop’ XML body:

$ curl -X PROPFIND https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
   --cert $X509_USER_PROXY \
   --capath /etc/grid-security/certificates \
   --cacert /usr/share/igi-test-ca/test0.cert.pem \
   --data "<?xml version='1.0' encoding='utf-8' ?><D:propfind xmlns:D='DAV:'><D:allprop/></D:propfind>"

$ davix-http -P Grid -X PROPFIND https://omii006-vm03.cnaf.infn.it:8443/test.vo/test.txt \
    --data "<?xml version='1.0' encoding='utf-8' ?><D:propfind xmlns:D='DAV:'><D:allprop/></D:propfind>"

The XML body received:

<?xml version="1.0" encoding="UTF-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:cal="urn:ietf:params:xml:ns:caldav" xmlns:card="urn:ietf:params:xml:ns:carddav" xmlns:cs="http://calendarserver.org/ns/" xmlns:ns1="http://storm.italiangrid.org/2014/webdav">
   <d:response>
      <d:href>/test.vo/test.txt</d:href>
      <d:propstat>
         <d:prop>
            <ns1:Checksum />
            <d:supported-report-set />
            <d:isreadonly>TRUE</d:isreadonly>
            <d:iscollection>FALSE</d:iscollection>
            <d:name>test.txt</d:name>
            <d:getcontenttype>text/plain</d:getcontenttype>
            <d:resourcetype />
            <d:getlastmodified>Mon, 16 Mar 2015 10:04:56 GMT</d:getlastmodified>
            <d:getcontentlength>12</d:getcontentlength>
            <d:getcreated>2015-03-16T10:04:56Z</d:getcreated>
            <d:creationdate>2015-03-16T10:04:56Z</d:creationdate>
            <d:getetag>"/storage/test.vo/test.txt_571153420"</d:getetag>
            <d:displayname>test.txt</d:displayname>
         </d:prop>
         <d:status>HTTP/1.1 200 OK</d:status>
      </d:propstat>
   </d:response>
</d:multistatus>