Data sharing

Sometimes two or more MetaCentrum users need to work on the same files and directories.

Since setting whole home directory as accessible to anyone is considered a security vulnerability (due to the possibility to manipulate sensitive files like .k5login, .bashrc by a third party) such settings are discouraged and are automatically reset.

To make sharing of data possible and at the time safe, we offer two options varying slightly with respect to the expected volume of shared data, namely:

1. sharing based on a group (moderate amount of data)
2. sharing based on project directory (large amount of data)

Group setup

1. Choose a name for your group and ask us at meta@cesnet.cz to create it
2. Include a list of members of the group and choose at least one group manager

The group manager can later add/remove members of this group through the Perun interface.

After the group is created, you will be notified.

Finally, create a directory dedicated for shared data, e.g. mkdir /storage/CITY/home/USER/shared.

Project directory setup

1. Choose a name for your group and ask us at meta@cesnet.cz to create it together with a project directory
2. Include list of members of the group and choose at least one group manager

The group manager can later add/remove members of this group through the Perun interface.

After the group is created, you will be notified.

There will be also a /storage/projects/GROUP_NAME directory created to store the data in.

The Unix file permissions for project directory are 770.

Using the shared directory

The usage of shared directory is the same for both shared directories in /storage/projects and in /storage/CITY/home/USER/.

1. assign the directory to your group (denoted by “MYGROUP”)

chgrp -R MYGROUP /storage/CITY/home/USER/shared

2. set a “sticky bit” (permission atribute) to the directory, so that all the data created within it will belong to the group

chmod g+s /storage/CITY/home/USER/shared

3. Ensure the correct identity within the system/job environment

When creating files, the system has to know under which group identity the files should be created.

Here, it is necessary to distinguish between the work on frontends and work within running jobs.

Identity on frontends

To create files when working on frontends, it is necessary to change your primary group to the requested one:

newgrp MYGROUP # all the created files will be owned by the MYGROUP group
umask 002 # will set the access rights of all the newly created files to rwxrwxr-x

You can include the above lines into system start-up scripts /storage/*/home/USER/.profile or equivalent file (e.g. /storage/*/home/USER/.bash_profile) so that the correct group is set automatically.

Alternatively, you can ask us to change your primary group within the whole MetaCentrum infrastructure.

Identity in running jobs

When creating files within running jobs, it is necessary to have the following lines within your batch script:

#!/bin/bash
#PBS -W umask=002
#PBS -W group_list=MYGROUP

• option -W umask=002 ensures the correct access rights to created files (rwxrwxr-x)
• option -W group_list=MYGROUP ensures that all the processes will run under the group MYGROUP

these options further ensure an availability of the $SCRATCHDIR directory to the group members as well - the directory will be owned by the requested group and the access rights will be rwx:

$ qsub -I -W group_list=einfra -W umask=002
qsub: waiting for job 4025666.pbs-m1.metacentrum.cz to start
qsub: job 4025666.pbs-m1.metacentrum.cz ready
 
konos6$ ls -ld $SCRATCHDIR
drwxrwxr-x 2 makub einfra 6 Feb 13 10:50 /scratch/makub/job_4025666.pbs-m1.metacentrum.cz
 
konos6$ id
uid=13153(makub) gid=10002(einfra) groups=10000(meta),10002(einfra),10100(storage)

chgrp -R MYGROUP DIRECTORY

(Otherwise, the data will be saved under your primary group.) Alternatively, you can ask us for changing your primary group throughout the whole MetaCentrum infrastructure.

sync_with_group usage

Some users find changing umask inconvenient. For these users we recommend another approach.

1. Copy the files you need to work with from the shared directory elsewhere.
2. Process the data, create new files etc.
3. When ready to share the data, send them back into the shared directory by a command

sync_with_group group_name source_dir target_dir 

where

group_name = name of the group the project directory belongs to
source_dir = the working directory
target_dir = shared project directory

NFSv4 usage

Common user requirement is the need to share a certain directory with other users(s). For more extensive co-operations we recommend setting up a shared project directory, for smaller-scaled collaborations it is often sufficient to set up access rights accordingly.

This is an alternative way of access control to the above described methods.

Before you start

Below is a couple of things you should be aware of before you start.

• Any files you create, no matter whether inside or outside your home directory tree, are still yours and will be included in your quotas. If you produce data in other user’s directory, you may run out of your quota without seeing any obvious problem in your home.
• Within the shared directory, any user to whom you grant access can change, delete or rename the files and directories regardless of their owner. It is upon the collaborating group to ensure all users are able to manipulate data safely and to prevent the risks of unwanted loss of data.
• If you modify access permissions by running chmod or similar command please note that doing so is extremely risky, as you grant access to all users and anyone can, if only by accident, delete your files.
• The ls -l command will not list full ACL. You have to use nfs4_getfacl command instead. Furthermore, don’t use commands like chmod (POSIX commands) together with nfs4_setfacl.

NFSv4 wrapper

The NFSv4 ACL usage is faciliated by a share_directory.sh script which is automatically loaded into MetaCentrum user environment.

If you are interested in what’s inside, you can get the script at /afs/ics.muni.cz/software/meta-utils/public:

(BOOKWORM)user_123@skirit:~$ whereis share_directory.sh
share_directory.sh: /afs/ics.muni.cz/software/meta-utils/public/share_directory.sh
(BOOKWORM)user_123@skirit:~$ 

Set up a shared directory

First create the directory to be shared:

(BUSTER)melounova@skirit:~$ umask 0002 # set umask to 0002
(BUSTER)melounova@skirit:~$ mkdir /storage/brno2/home/melounova/shared # create the directory to be shared

Add group of users

If you wish to share the directory within a group of users, run the script as

(BUSTER)melounova@skirit:~$ share_directory.sh -g GROUPNAME /storage/brno2/home/melounova/shared

In the (improbable) case you yourself do not belong to the group, add also yourself as a particular user:

(BUSTER)melounova@skirit:~$ share_directory.sh -u melounova /storage/brno2/home/melounova/shared

Add particular users

If you want to share the directory only with selected user, proceed as follows:

(BUSTER)melounova@skirit:~$ share_directory.sh -u dummy1 /storage/brno2/home/melounova/shared
(BUSTER)melounova@skirit:~$ share_directory.sh -u dummy2 /storage/brno2/home/melounova/shared
...
(BUSTER)melounova@skirit:~$ share_directory.sh -u dummyN /storage/brno2/home/melounova/shared

Strange as it may seem, you need to grant yourself privileged access to be able to manipulate with files created by another user.

(BUSTER)melounova@skirit:~$ share_directory.sh -u melounova /storage/brno2/home/melounova/shared

Use the shared directory

Work with files and directories

Always set your umask to 0002 when working in shared directory:

(BUSTER)melounova@skirit:~/shared$ umask 0002 
(BUSTER)melounova@skirit:~/shared$ mkdir dir1

The same is true for any of your “hosting users”:

(BUSTER)dummy1@skirit:/storage/brno2/home/melounova/shared$ umask 0002
(BUSTER)dummy1@skirit:/storage/brno2/home/melounova/shared$ cd dir1 ; touch file001

What to do if forgot to set umask to 0002?

If you work in the shared directory with mask set to 0022 (which is the default), the newly created files and directories will not be writable to other users.

Unfortunately the nfs4_setfacl seems not to work correctly in recursive mode (there seems to be a bug in nfs4_setfacl), so the obvious solution like running nfs4_setfacl -R -a A::dummy1@META:RWX my_shared_dir recursively will not work in case you have both subdirectories and files in your shared directory. Therefore you have to circumvent the problem by setting the ACL rights for directories and files separately, as in this script:

#!/bin/bash
 
basedir="path/to/shared/directory" 
userlist="dummy1 dummy2... dummyN" # list of users to allow access to
 
# setback the RWX right for all files
for a in `find ${basedir} -type f` ; do
  for user in ${userlist} ; do
    nfs4_setfacl -a A::${user}@META:RWX ${a}
  done
done
 
# setback the RWX right for all directories
for a in `find ${basedir} -type d` ; do
  for user in ${userlist} ; do
    nfs4_setfacl -a A:fd:${user}@META:RWX ${a}
  done
done

Submit jobs

If you submit jobs and wish to copy results to the shared directory, set umask 0002 somewhere at the beginning of the script.