Supported by the National Science Foundation Collaborator: University of Michigan Collaborator: Michigan State University Collaborator: Wayne State University Collaborator: Indiana University

We’ve included here instructions for OSiRIS users as well as for other sites that are interested in setting up an installation similar to OSiRIS.


User Documentation

Enrolling in OSiRIS: How to enroll in your OSiRIS COU and obtain access tokens for use with OSiRIS Storage

Uploading an SSH key: How to upload your ssh key so you can use OSiRIS ssh/scp/sftp gateways

Using OSiRIS S3: How to obtain credentials and use OSiRIS S3 gateways

Using Globus with OSiRIS CephFS: How to get setup to use Globus with OSiRIS CephFS

Using NFS with OSiRIS: How to access NFS exports of OSiRIS CephFS on U-M or MSU campuses

Using RADOS: How to use OSiRIS Ceph object store directly with rados libs and utils (requires configuration, please contact us)

Managing OSiRIS Groups: How to create new posix groups and manage memberships

Configuring an S3 fuse mount: The S3 fuse program will present S3 buckets as mounted filesystems though the standard http-based S3 protocol.

OSiRIS Setup Documentation

At this time we do not have detailed setup documentation for OSiRIS. However, all of the components used are publicly available. Here is a quick overview.

Puppet Modules

Generally we use Puppet to manage setup and configuration. The following puppet modules were created or forked from other modules and modified for OSiRIS usage. Documentation on using them is included in the repository README file.

puppet-ceph: OSiRIS storage is provided by Ceph. This puppet module is used to deploy and manage all ceph components. It was recently updated to deploy Bluestore OSD. Our version is forked from openstack/puppet-ceph

puppet-ds389: OSiRIS backend directory services are provided by 389 Directory server in a multi-master replicated configuration. This module is used to deploy/manage that configuration and additional schema required for OSiRIS.

puppet-grouper: OSiRIS Posix groups are managed and provisioned to LDAP by Internet2 Grouper. Grouper could also be extended with additional provisioning targets to manage non-LDAP groups or to translate group memberships to other models such as S3 bucket ACL users but we haven’t explored this. This puppet module manages Grouper config as used by OSiRIS but requires some pre-setup of Grouper.

puppet-shibboleth: Our web services are authenticated by Shibboleth using InCommon meta-data. We use this puppet module to manage the configuration. It is forked from Aethylred/puppet-shibboleth.

Many other internal components are managed by puppet modules also available from our Github repository. These include pdsh for distributed command execution, LLDP for network link information, Rancid network config version control, and a Shibboleth auth module for our Dokuwiki internal wiki. Further information on any module should be in the repository README. We also leverage a large number of modules from Puppet Forge for basic system configuration.


OSiRIS identity management and provisioning is handled by Internet2 COmanage. Plugins to provision user information from COmanage to LDAP and to Grouper are part of the COmanage release. Plugins related to Ceph we had to write. Each plugin is developed on a git branch and merged into a master branch that reflects our current in-use version of COmanage. We tend to track the develop branch of COmanage.

CephProvisioner: This plugin provisions COmanage identities to Ceph. It covers several provisioning operations:

  • COU - When new COU are created the plugin creates data pools in Ceph for that COU and associates them with the appropriate application (CephFS, RGW). It also creates data placement targets and tags for the RGW pool.
  • RGW - A RGW user is created for each unique combination of uid identifier and COU name (ie, username_somecou). This decision was made so we could support setting default data placement targets for each user associated to per-COU data pools. When the user is provisioned they are configured with default placement and placement tags corresponding to their COU data placement targets. Users can retrieve their LDAP RGW tokens from the COmanage service token page (modifications were made to the CoServiceToken plugin)
  • CephFS - COmanage provisions LDAP posixUser information used by systems which mount CephFS, and it provisions default COU groups to Grouper where they they are provisioned to LDAP along with any user-created groups. This plugin creates a ceph client key with appropriate access capabilities to access the COU directory path and COU data pool dedicated to CephFS. We use an external script to create a directory for each COU and set the file layout for the directory to be the COU CephFS data pool. The key created by this plugin also includes capabilities to access COU data pools for rados and rbd. Users can retrieve this key from the COmanage service tokens page (more modifications in CoServiceToken plugin)

CephRgwLdapTokenProvisioner: For the RGW case we use LDAP authentication. This plugin provisions a simpleSecurityObject into an LDAP OU. The object uid is a unique combination of uid identifier and COU name, and one object is provisioned for each COU a user might belong to. The password is set from a COmanage service token type which we added to the CoServiceToken plugin. The LDAP users match up with userid provisioned by the CephProvisioner plugin in radosgw.

CoServiceToken: This plugin is already part of COmanage but we made some modifications to save and display Ceph RGW ldap tokens and Ceph cluster auth keys. Like other COmanage token types the user can regenerate the token on demand and retrieve it from the service token view. RGW tokens generated by this plugin are saved and propogated to the LDAP directory by the CephRgwLdapTokenProvisioner. Ceph cluster keys are generated with Ceph cluster commands from this plugin and saved for retrieval by the user. It relies on the keys already being provisioned by the CephProvisioner plugin to generate a new key with identical caps.

LdapUserPosixGroupProvisioner: A simple plugin that provisions a posixGroup with gid matching every posixUser uid. Possibly will be obsoleted by including this feature in the core LdapProvisioner plugin but nonetheless we needed something to do this.

Stable code from of all of these plugins is combined on the osiris_master Git branch within our fork of the Internet2 COmanage repository. Other miscellanous changes to COmanage are also included on this branch but they are non-essential for recreating our functionality. From time to time we have made PR to the upstream repo with small changes that are applicable to general use, and may at some point make an effort to include our other plugins in the upstream release if there is interest.