Storage Department services
The CESNET Storage Department provides a backup and archiving service. It is available to all users with MetaCentrum login and password.
Storage Department data policies will be described to a certain level at this page. For more detailed information, users should however navigate the Storage Department documentation pages.
Data storage technology in Storage department has changed by May 2024
For a long time the data were stored on hierarchical storage machine ("hsm" for short) with a directory structure accessible from /storage/du-cesnet.
Due to end-of-life state and multiple hardware failures the HSM storages were disconnected. User data have been transferred to machines with Object storage technology.
Object storage is successor of HSM with slightly different set of commands, i.e. it does not work in the precisely same way (sadly).
Ask du-support@cesnet.cz for help, they are prepared for many questions from users
Object storage is a different paradigm than good old Linux filesystem, so yes, this is a big change for most of our users and not a very easy one to grasp. The people in Storage department know this and they are prepared to answer your questions.
Please be aware that MetaCentrum user support (meta@cesnet.cz) can provide only limited advice as the data storage is out of the scope of MetaCentrum services.
Below you will find short description of main object storage (S3) command and usage.
Object storage
Access
First you need to contact data storage support du-support@cesnet.cz and request them to obtain s3 credenials.
To work with S3 storage, you need to obtain S3 credentials aws_access_key_id (aka "access key") and aws_secret_access_key (aka "secret key") from Storage Department admins.
Clients
There are more clients to work with s3 storages - see guide on Storage Department pages.
s3cmd
Create a configuration file /storage/brno2/home/user123/.s3cfg:
[default]
host_base = https://s3.cl4.du.cesnet.cz
use_https = True
access_key = XXXXXXXXXXXXXXXXXXXX
secret_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
host_bucket = s3.cl4.du.cesnet.cz
| Basic s3cmd commands | |
|---|---|
| Bucket operations | |
| s3cmd ls | List all s3 buckets |
| s3cmd mb s3://newbucket | Create of new s3 bucket |
| s3cmd rb s3://newbucket | Remove s3 bucket |
| s3cmd du s3://newbucket/ | List s3 bucket size |
| Files and directories operations | |
| s3cmd ls s3://newbucket/ | List content of s3 bucket |
| s3cmd put file.txt s3://newbucket/ | Upload a file |
| s3cmd put -r directory s3://newbucket/ | Upload directory |
| s3cmd get s3://newbucket/file.txt | Download file from s3 bucket |
| s3cmd del s3://newbucket/file.txt $ s3cmd del s3://newbucket/directory |
Delete data from s3 bucket |
| s3cmd sync /local/path/ s3://newbucket/backup/ | Data sync into s3 bucket |
| s3cmd sync s3://newbucket/backup/ ~/restore/ | Data sync from s3 bucket |
aws
Create a configuration file /storage/brno2/home/user123/.aws/credentials:
###
[my-profile]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
max_concurrent_requests = 200
max_queue_size = 20000
multipart_threshold = 128MB
multipart_chunksize = 32MB
###
| Basic aws commands | |
|---|---|
| Bucket operations | |
| aws s3 --profile my-profile --endpoint-url https://s3.cl4.du.cesnet.cz ls | List all s3 buckets |
| aws s3 --profile my-profile --endpoint-url https://s3.cl4.du.cesnet.cz mb s3://test1 | Create of new s3 bucket |
| aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz rb s3://test1 | Remove s3 bucket |
| Files and directories operations | |
| aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz cp C:/Users/User/Desktop/test-file.zip s3://test1 | Upload a file |
| $ aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz cp s3://test1/test-file.zip C:\Users\User\Downloads\ | Download file from s3 bucket |
| $ aws s3 --profile my-profile --endpoint-url https://s3.cl2.du.cesnet.cz rm s3://test1/test-file.zip | Delete data from s3 bucket |