StoRM Info Provider

StoRM Info Provider is the StoRM component that manages how and what information are published on the BDII.

The BDII service

The WLCG Information System is used to discover services and get status information about WLCG resources. The BDII (Berkeley Database Information Index) is a Perl/BDB ‘glue’ used to manage LDAP updates. See Grid Information System page for more details.

By default, the BDII uses three directories to obtain information sources:

ldif: static LDIF files should be placed in this directory;
provider: here information providers are placed and run once at BDII startup;
plugin: scripts periodically run to update information.

These directories are located by default into /var/lib/bdii/gip.

Configuration

StoRM Info Provider has not a configuration file, its behavior and outputs depend on the site configuration which is processed by YAIM and stored into /etc/storm/info-provider/storm-yaim-variables.conf.

There are some YAIM variables used both to configure StoRM services and Info Provider and some other specific variables which affects only the published information (e.g. the SRM and WebDAV endpoints).

Var. Name	Description
`SITE_NAME`	It’s the human-readable name of your site.

`STORM_ENDPOINT_QUALITY_LEVEL`	Endpoint maturity level to be published. Optional variable. Available values are: 0 (development), 1 (testing), 2 (pre-production) or 3 (production). Default value: 2
`STORM_ENDPOINT_SERVING_STATE`	Endpoint serving state to be published. Optional variable. Available values are: 1, (closed) or 4 (production). Default value: 4
`STORM_ENDPOINT_CAPABILITY`	Capability according to OGSA to be published. Optional variable. Default value: data.management.storage

`STORM_FRONTEND_PATH`	StoRM Frontend service path. Optional variable. Default value: /srm/managerv2
`STORM_FRONTEND_PORT`	StoRM Frontend service port. Optional variable. Default value: 8444
`STORM_FRONTEND_PUBLIC_HOST`	StoRM Frontend service public host. It’s used by StoRM Info Provider to publish the SRM endpoint into the Resource BDII. Mandatory variable. Default value: STORM_BACKEND_HOST

`STORM_INFO_FILE_SUPPORT`	Trigger the `file` protocol to be published as transfer protocol. Optional variable. There’s also a variable for a more specific Storage Area configuration. Default value: true
`STORM_INFO_GRIDFTP_SUPPORT`	Trigger the `gsiftp` protocol to be published as transfer protocol. Optional variable. There’s also a variable for a more specific Storage Area configuration. Default value: true
`STORM_INFO_RFIO_SUPPORT`	Trigger the `rfio` protocol to be published as transfer protocol. Optional variable. There’s also a variable for a more specific Storage Area configuration. Default value: false
`STORM_INFO_ROOT_SUPPORT`	Trigger the `root` (or `xroot`) protocol to be published as transfer protocol. Optional variable. There’s also a variable for a more specific Storage Area configuration. Default value: false
`STORM_INFO_HTTP_SUPPORT`	Trigger the `HTTP` protocol to be published as transfer protocol. Optional variable. There’s also a variable for a more specific Storage Area configuration. Default value: false
`STORM_INFO_HTTPS_SUPPORT`	Trigger the `HTTPS` protocol to be published as transfer protocol. Optional variable. There’s also a variable for a more specific Storage Area configuration. Default value: false

STORM_WEBDAV_POOL_LIST StoRM WebDAV endpoints. It’s used by StoRM Info Provider to publish the WebDAV endpoints into the Resource BDII.
Optional variable, mandatory if the administrator wants to publish a WebDAV endpoint.

Publish the SRM endpoint

StoRM Info Provider allows to publish one and unique SRM endpoint. The endpoint published will be:

httpg://${STORM_FRONTEND_PUBLIC_HOST}:${STORM_FRONTEND_PORT}${STORM_FRONTEND_PATH}

Publish one or more WebDAV endpoints

From version 1.8.2 StoRM Info Provider allows to publish more than one WebDAV endpoint. Use STORM_WEBDAV_POOL_LIST to share the endpoints.

It’s recommended to migrate to the new YAIM variable if you were publishing WebDAV endpoint info with STORM_GRIDHTTPS_PUBLIC_HOST variable. Old configuration variables are still supported within StoRM v1.11.15, but the upcoming versions won’t support them.

For example, if:

your WebDAV endpoint’s hostname is storm-webdav.example.org
it’s detached from Backend’s hostname
you kept default http and https ports

you should have set:

STORM_GRIDHTTPS_ENABLED=true
STORM_GRIDHTTPS_PUBLIC_HOST=storm-webdav.example.org

If you changed the default ports you should also have values for:

STORM_GRIDHTTPS_HTTP_PORT=...
STORM_GRIDHTTPS_HTTPS_PORT=...

Your published endpoints will be:

http://storm-webdav.example.org:8085
https://storm-webdav.example.org:8443

Then you can copy this endpoints and paste them into the following YAIM variable:

STORM_WEBDAV_POOL_LIST=http://storm-webdav.example.org:8085,https://storm-webdav.example.org:8443

and then remove STORM_GRIDHTTPS_ENABLED, STORM_GRIDHTTPS_PUBLIC_HOST and also STORM_GRIDHTTPS_HTTP_PORT and STORM_GRIDHTTPS_HTTPS_PORT, if defined.

Usage

/usr/libexec/storm-info-provider -h

usage: storm-info-provider [-h] [-v LOG_LEVEL] [-o LOG_FILENAME]
                       {configure,get-static-ldif,get-update-ldif} ...

Options:

-v: LOG_LEVEL can be 10 (DEBUG), 20 (INFO - default), 30 (WARNING) and 40 (ERROR)
-o: all the log messages are printed on stderr by default but they can be redirected to an external LOG_FILENAME by specifying this option

Example of usage with configure:

/usr/libexec/storm-info-provider configure -h

usage: storm-info-provider configure [-h] [-f FILEPATH]
                                 [-g {glue13,glue2,all}]
optional arguments:
  -h, --help            show this help message and exit
  -f FILEPATH
  -g {glue13,glue2,all}

Options:

-f: the path of the file which contains all the StoRM related YAIM variables with their key-value pairs (default is /etc/storm/info-provider/storm-yaim-variables.conf which is the file created by yaim-storm)
-g: GLUE version selector (default: all)

During configuration, yaim-storm creates storm-yaim-variables.conf and runs the StoRM Dynamic Info Provider script as follow:

/usr/libexec/storm-info-provider -v LOG_LEVEL configure -g all -f /etc/storm/info-provider/storm-yaim-variables.conf

Example of output with LOG_LEVEL=20:

/usr/libexec/storm-info-provider configure

2014-09-04 10:40:34,271 root        : INFO Successfully created /etc/storm/info-provider/glite-info-glue13-service-storm.conf !
2014-09-04 10:40:34,271 root        : INFO Successfully created /var/lib/bdii/gip/provider/storm-glue13-provider !
2014-09-04 10:40:34,272 root        : INFO Successfully created /var/lib/bdii/gip/plugin/storm-glue13-plugin !
2014-09-04 10:40:34,321 root        : INFO Successfully created /var/lib/bdii/gip/ldif/storm-glue13-static.ldif !
2014-09-04 10:40:34,322 root        : INFO Successfully created /etc/storm/info-provider/glite-info-glue2-service-storm.conf !
2014-09-04 10:40:34,322 root        : INFO Successfully created /etc/storm/info-provider/glite-info-glue2-service-storm-endpoint-srm.conf !
2014-09-04 10:40:34,322 root        : INFO Successfully created /var/lib/bdii/gip/provider/storm-glue2-provider !
2014-09-04 10:40:34,323 root        : INFO Successfully created /var/lib/bdii/gip/plugin/storm-glue2-plugin !
2014-09-04 10:40:34,365 root        : INFO Successfully created /var/lib/bdii/gip/ldif/storm-glue2-static.ldif !
2014-09-04 10:40:34,365 root        : INFO Received configure - It took 0.04 sec

Example of usage with get-static-ldif:

/usr/libexec/storm-info-provider get-static-ldif -h

usage: storm-info-provider get-static-ldif [-h] [-f FILEPATH]
                                           [-g {glue13,glue2}]
optional arguments:
  -h, --help         show this help message and exit
  -f FILEPATH
  -g {glue13,glue2}

Options:

-f: the path of the file which contains all the StoRM related YAIM variables with their key-value pairs (default is /etc/storm/info-provider/storm-yaim-variables.conf which is the file created by yaim-storm)
-g: GLUE version selector (default: glue2)

Example of a filtered output to obtain only the dn of the generated entries:

/usr/libexec/storm-info-provider get-static-ldif -g glue13 2>/dev/null | grep dn

dn: GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSALocalID=tape:custodial:nearline,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueVOInfoLocalID=testers.eu-emi.eu,GlueSALocalID=tape:custodial:nearline,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSALocalID=igi:replica:online,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSALocalID=noauth:replica:online,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSALocalID=nested:replica:online,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueVOInfoLocalID=testers.eu-emi.eu,GlueSALocalID=nested:replica:online,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSALocalID=dteam:replica:online,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueVOInfoLocalID=dteam,GlueSALocalID=dteam:replica:online,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSALocalID=testerseuemieu:replica:online,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueVOInfoLocalID=testers.eu-emi.eu,GlueSALocalID=testerseuemieu:replica:online,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSEControlProtocolLocalID=srm_v2.2,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSEAccessProtocolLocalID=file,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSEAccessProtocolLocalID=gsiftp,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSEAccessProtocolLocalID=xroot,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSEAccessProtocolLocalID=http,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSEAccessProtocolLocalID=https,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid
dn: GlueSEAccessProtocolLocalID=webdav,GlueSEUniqueID=cloud-vm45.cloud.cnaf.infn.it,mds-vo-name=resource,o=grid

The action get-static-ldif is not used by the installed scripts. However its functionality is internally used by the configure action to generate the static ldif files.

Example of usage with get-update-ldif:

/usr/libexec/storm-info-provider get-update-ldif -h
usage: storm-info-provider get-static-ldif [-h] [-f FILEPATH]
                                           [-g {glue13,glue2}]
optional arguments:
  -h, --help         show this help message and exit
  -f FILEPATH
  -g {glue13,glue2}

Options:

-f: the path of the file which contains all the StoRM related YAIM variables with their key-value pairs (default is /etc/storm/info-provider/storm-yaim-variables.conf which is the file created by yaim-storm)
-g: GLUE version selector (default: glue2)

The plugin files created during configure phase runs StoRM DIP get-update-ldif. If StoRM service is down, an error is logged and user obtains the LDIF output useful to update the serving-state value of the endpoints.

Example of a filtered output to obtain only the dn of the generated entries:

/usr/libexec/storm-info-provider get-update-ldif -g glue2 2>/dev/null | grep dn

dn: GLUE2EndpointID=cloud-vm45.cloud.cnaf.infn.it/storage/endpoint/SRM,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2EndpointID=cloud-vm45.cloud.cnaf.infn.it/storage/endpoint/HTTP,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2EndpointID=cloud-vm45.cloud.cnaf.infn.it/storage/endpoint/HTTPS,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2StorageServiceCapacityID=cloud-vm45.cloud.cnaf.infn.it/storage/capacity/online,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2StorageServiceCapacityID=cloud-vm45.cloud.cnaf.infn.it/storage/capacity/nearline,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/tape,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2StorageShareCapacityID=cloud-vm45.cloud.cnaf.infn.it/storage/share/tape/capacity/online,GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/tape,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2StorageShareCapacityID=cloud-vm45.cloud.cnaf.infn.it/storage/share/tape/capacity/nearline,GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/tape,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/igi,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2StorageShareCapacityID=cloud-vm45.cloud.cnaf.infn.it/storage/share/igi/capacity/online,GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/igi,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/noauth,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2StorageShareCapacityID=cloud-vm45.cloud.cnaf.infn.it/storage/share/noauth/capacity/online,GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/noauth,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/nested,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2StorageShareCapacityID=cloud-vm45.cloud.cnaf.infn.it/storage/share/nested/capacity/online,GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/nested,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/dteam,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2StorageShareCapacityID=cloud-vm45.cloud.cnaf.infn.it/storage/share/dteam/capacity/online,GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/dteam,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/testerseuemieu,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
dn: GLUE2StorageShareCapacityID=cloud-vm45.cloud.cnaf.infn.it/storage/share/testerseuemieu/capacity/online,GLUE2ShareID=cloud-vm45.cloud.cnaf.infn.it/storage/share/testerseuemieu,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue

Example of the output generated when StoRM service is down:

service storm-backend-server stop

Stopping storm-backend-server                              [  OK  ]

/usr/libexec/storm-info-provider get-update-ldif -g glue2 2>/dev/null

dn: GLUE2EndpointID=cloud-vm45.cloud.cnaf.infn.it/storage/endpoint/SRM,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
GLUE2EndpointServingState: closed
dn: GLUE2EndpointID=cloud-vm45.cloud.cnaf.infn.it/storage/endpoint/HTTP,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
GLUE2EndpointServingState: closed
dn: GLUE2EndpointID=cloud-vm45.cloud.cnaf.infn.it/storage/endpoint/HTTPS,GLUE2ServiceID=cloud-vm45.cloud.cnaf.infn.it/storage,GLUE2GroupID=resource,o=glue
GLUE2EndpointServingState: closed