Storage Department services
The CESNET Storage Department provides a backup and archiving service. It is available to all users with MetaCentrum login and password.
Storage Department data policies will be described to a certain level at this page. For more detailed information, users should however navigate the Storage Department documentation pages.
Tape storage
Data archiving and backup is not MetaCentrum service
From within MetaCentrum frontend, users can ssh/scp to Storage Department infrastructure. Information in this section is a rough overview of data services provided by Cesnet Storage Department. In case of problems/questions, we recommend to contact Cesnet Storage Department.
Underlying infrastructure in Storage Department are servers with hierarchical storage policy. To increase redundancy of data, these discs contain several layers of storage media: first layer is a disk array, lower layers are made of MAIDs (Massive Array of Idle Drives) or magnetic tape libraries.
- Lower layers have bigger capacity but slower access times.
- Data are moved automatically among these layers based on their last usage.
- The most important consequence from the user's point of view is that the access to unused data may be slower than to the recently used ones.
Current Storage Department servers
| Server name | Mounted at | Note |
|---|---|---|
| storage-du-cesnet.metacentrum.cz | /storage/du-cesnet/ | for all Metacentrum users |
| storage-brno14.ceitec.metacentrum.cz | /storage/brno14-ceitec/ | for NCBR/CEITEC users only |
Warning
Never leave data directly in the home, i.e. in/storage/du-cesnet/home/META_username/. The home directory should serve only to keep SSH keys, making links to directories with the actual data and other configuration files. To enforce this, there is tiny quota set on home directory (see further info on Cesnet data storage service pages).
Backup
/storage/du-cesnet/home/META_username/VO_metacentrum-tape_tape/- Use this option to backup data to protect data in case primary data are lost.
- Typically these data need not to be kept for a very long time.
- Files older than 12 months are automatically removed (they are considered as "expired").
Archiving
/storage/du-cesnet/home/META_username/VO_metacentrum-tape_tape-archive/- Use this option to archive data you want to keep "forever".
- The directory has user quota set for volume of data and/or number of files.
- The data are not removed after a time (they do not "expire").
Object storage
Warning
This section is under construction. A complete guide can be found in the Storage Department documentation.
Access
First you need to contact data storage support du-support@cesnet.cz and request them to obtain s3 credenials.
To work with S3 storage, you need to obtain S3 credentials aws_access_key_id (aka "access key") and aws_secret_access_key (aka "secret key") from Storage Department admins.
Clients
There are more clients to work with s3 storages - see guide on Storage Department pages.
s3cmd
Create a configuration file /storage/brno2/home/user123/.s3cfg:
[default]
host_base = https://s3.cl4.du.cesnet.cz
use_https = True
access_key = XXXXXXXXXXXXXXXXXXXX
secret_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
host_bucket = s3.cl4.du.cesnet.cz
| Basic s3cmd commands | |
|---|---|
| Bucket operations | |
| s3cmd ls | List all s3 buckets |
| s3cmd mb s3://newbucket | Create of new s3 bucket |
| s3cmd rb s3://newbucket | Remove s3 bucket |
| s3cmd du s3://newbucket/ | List s3 bucket size |
| Files and directories operations | |
| s3cmd ls s3://newbucket/ | List content of s3 bucket |
| s3cmd put file.txt s3://newbucket/ | Upload a file |
| s3cmd put -r directory s3://newbucket/ | Upload directory |
| s3cmd get s3://newbucket/file.txt | Download file from s3 bucket |
| s3cmd del s3://newbucket/file.txt $ s3cmd del s3://newbucket/directory |
Delete data from s3 bucket |
| s3cmd sync /local/path/ s3://newbucket/backup/ | Data sync into s3 bucket |
| s3cmd sync s3://newbucket/backup/ ~/restore/ | Data sync from s3 bucket |
aws
Create a configuration file /storage/brno2/home/user123/.aws/credentials:
###
[my-profile]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
max_concurrent_requests = 200
max_queue_size = 20000
multipart_threshold = 128MB
multipart_chunksize = 32MB
###
| Basic aws commands | |
|---|---|
| Bucket operations | |
| aws s3 --profile my-profile --endpoint-url https://s3.cl4.du.cesnet.cz ls | List all s3 buckets |
| aws s3 --profile my-profile --endpoint-url https://s3.cl4.du.cesnet.cz mb s3://test1 | Create of new s3 bucket |
| aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz rb s3://test1 | Remove s3 bucket |
| Files and directories operations | |
| aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz cp C:/Users/User/Desktop/test-file.zip s3://test1 | Upload a file |
| $ aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz cp s3://test1/test-file.zip C:\Users\User\Downloads\ | Download file from s3 bucket |
| $ aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz rm s3://test1/test-file.zip | Delete data from s3 bucket |