Skip to content

Storage Department services

The CESNET Storage Department provides a backup and archiving service. It is available to all users with MetaCentrum login and password.

Storage Department data policies will be described to a certain level at this page. For more detailed information, users should however navigate the Storage Department documentation pages.

Tape storage

Data archiving and backup is not MetaCentrum service

From within MetaCentrum frontend, users can ssh/scp to Storage Department infrastructure. Information in this section is a rough overview of data services provided by Cesnet Storage Department. In case of problems/questions, we recommend to contact Cesnet Storage Department.

pic

Underlying infrastructure in Storage Department are servers with hierarchical storage policy. To increase redundancy of data, these discs contain several layers of storage media: first layer is a disk array, lower layers are made of MAIDs (Massive Array of Idle Drives) or magnetic tape libraries.

  • Lower layers have bigger capacity but slower access times.
  • Data are moved automatically among these layers based on their last usage.
  • The most important consequence from the user's point of view is that the access to unused data may be slower than to the recently used ones.

Current Storage Department servers

Server name Mounted at Note
storage-du-cesnet.metacentrum.cz /storage/du-cesnet/ for all Metacentrum users
storage-brno14.ceitec.metacentrum.cz /storage/brno14-ceitec/ for NCBR/CEITEC users only

Warning

Never leave data directly in the home, i.e. in/storage/du-cesnet/home/META_username/. The home directory should serve only to keep SSH keys, making links to directories with the actual data and other configuration files. To enforce this, there is tiny quota set on home directory (see further info on Cesnet data storage service pages).

Backup

  • /storage/du-cesnet/home/META_username/VO_metacentrum-tape_tape/
  • Use this option to backup data to protect data in case primary data are lost.
  • Typically these data need not to be kept for a very long time.
  • Files older than 12 months are automatically removed (they are considered as "expired").

Archiving

  • /storage/du-cesnet/home/META_username/VO_metacentrum-tape_tape-archive/
  • Use this option to archive data you want to keep "forever".
  • The directory has user quota set for volume of data and/or number of files.
  • The data are not removed after a time (they do not "expire").

Object storage

Warning

This section is under construction. A complete guide can be found in the Storage Department documentation.

Access

First you need to contact data storage support du-support@cesnet.cz and request them to obtain s# credenials.

To work with S3 storage, you need to obtain S3 credentials aws_access_key_id (aka "access key") and aws_secret_access_key (aka "secret key") from Storage Department admins.

Clients

There are more clients to work with s3 storages - see guide on Storage Department pages.

s3cmd

Create a configuration file /storage/brno2/home/user123/.s3cfg:

[default]
host_base = https://s3.cl4.du.cesnet.cz
use_https = True
access_key = XXXXXXXXXXXXXXXXXXXX  
secret_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
host_bucket = s3.cl4.du.cesnet.cz
Basic s3cmd commands
Bucket operations
s3cmd ls List all s3 buckets
s3cmd mb s3://newbucket Create of new s3 bucket
s3cmd rb s3://newbucket Remove s3 bucket
s3cmd du s3://newbucket/ List s3 bucket size
Files and directories operations
s3cmd ls s3://newbucket/ List content of s3 bucket
s3cmd put file.txt s3://newbucket/ Upload a file
s3cmd put -r directory s3://newbucket/ Upload directory
s3cmd get s3://newbucket/file.txt Download file from s3 bucket
s3cmd del s3://newbucket/file.txt
$ s3cmd del s3://newbucket/directory
Delete data from s3 bucket
s3cmd sync /local/path/ s3://newbucket/backup/ Data sync into s3 bucket
s3cmd sync s3://newbucket/backup/ ~/restore/ Data sync from s3 bucket

aws

Create a configuration file /storage/brno2/home/user123/.aws/credentials:

###
[my-profile]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
max_concurrent_requests = 200
max_queue_size = 20000
multipart_threshold = 128MB
multipart_chunksize = 32MB
###
Basic aws commands
Bucket operations
aws s3 --profile my-profile --endpoint-url https://s3.cl4.du.cesnet.cz ls List all s3 buckets
aws s3 --profile my-profile --endpoint-url https://s3.cl4.du.cesnet.cz mb s3://test1 Create of new s3 bucket
aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz rb s3://test1 Remove s3 bucket
Files and directories operations
aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz cp C:/Users/User/Desktop/test-file.zip s3://test1 Upload a file
$ aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz cp s3://test1/test-file.zip C:\Users\User\Downloads\ Download file from s3 bucket
$ aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz rm s3://test1/test-file.zip Delete data from s3 bucket