diff --git a/docs/apps/r-env.md b/docs/apps/r-env.md index 1a2c6ed72e..b94026170b 100644 --- a/docs/apps/r-env.md +++ b/docs/apps/r-env.md @@ -779,7 +779,7 @@ When prompted about an existing LaTeX distribution, answer `yes` to continue the The `r-env` module comes with the [`aws.s3`](https://cran.r-project.org/web/packages/aws.s3/) package for working with S3 storage, which makes it possible to use the Allas storage system directly from an R script. See [here](https://github.com/csc-training/geocomputing/blob/master/R/allas/working_with_allas_from_R_S3.R) for a practical example involving raster data. -Accessing Allas via the `r-env` module can be done as follows. First configure [Allas connection for S3](../data/Allas/using_allas/python_boto3.md#configuring-s3): +Accessing Allas via the `r-env` module can be done as follows. First configure [Allas connection for S3](../data/Allas/using_allas/allas-conf.md#s3-connection): ```bash module load allas diff --git a/docs/computing/allas.md b/docs/computing/allas.md deleted file mode 100644 index efaba3e594..0000000000 --- a/docs/computing/allas.md +++ /dev/null @@ -1,68 +0,0 @@ -# Using Allas object storage service from CSC Supercomputing environment - -The disk environments of CSC supercomputers, Puhti and Mahti, are not intended for storing research data that is -not actively used for computing. The data that needs to be stored for a longer time than just few weeks should be copied to Allas object storage service. - -Allas provides a platform that you can use to store your data as long as your CSC project is active. -In addition to storing, Allas can be used for transporting data between different servers and sharing data -with other users. - -## Getting access to Allas - -By default, CSC computing projects do not have access to Allas. Thus, the first thing is to make sure that your project has access to Allas. -This can be done in the [MyCSC interface](https://my.csc.fi). Note that only the project manager can -add new services for a project. Once Allas access is activated, all project members must visit the MyCSC and accept the terms -of use for Allas before they can use the Allas storage area. - -The default storage quota in Allas is 10 TB. As this space is shared with all project members, it is -possible that the space is not sufficient. In that case, you should estimate how much space is needed -and request more space. The request should be sent to servicedesk@csc.fi. - -## Connecting Allas - -In order to use Allas in Puhti or Mahti, first load the module _allas_: -```text -module load allas -``` -Allas access for a specific project can then be enabled with command: -```text -allas-conf -``` -or -```text -allas-conf project_name -``` -The _allas-conf_ command prompts for your CSC password. It lists your Allas projects and asks you to define a project (if not already defined as an argument). After that _allas-conf_ generates configuration files for several Allas clients and authenticates connection to Allas for the selected project. - -By default the _allas-conf_ activates tools that use Allas with **swift protocol**. -You can alternatively use S3 protocol too, however in this document we -discuss only _swift_ based Allas usage. - -Authentication is session specific and valid for eight hours at a time. -You can be connected to only one Allas project at a time in one terminal session. However, simutaneous terminal sessions -can use different Allas projects. The project you are using in Allas does not need to match the project you are using in -Puhti or Mahti. You can refresh the authentication or change the target project at any time my running _allas-conf_ again. - - -**Allas client software options for Puhti and Mahti and other linux servers** - -The _allas_ module provides several tools that you can use to move data between Allas and the computing server. -You can cross-use the Allas clients as long as you access Allas with the same protocol (swift or S3). -Below is a list of the Allas clients that are most commonly used in Puhti and Mahti: - -* **a-commands:** (Swift, optionally S3) [Easy and safe: a-commands](../data/Allas/using_allas/a_commands.md) -* **rclone:** (Swift, optionally S3) [Advanced tool: rclone](../data/Allas/using_allas/rclone.md) -* **swift python client:** (Swift) [Native Swift client](../data/Allas/using_allas/swift_client.md) -* **s3cmd:** (S3) [S3 client](../data/Allas/using_allas/s3_client.md#configuring-s3-connection-in-supercomputers) - -More information about using Allas can be found from the Allas documentation: - -* [Allas](../data/Allas/index.md) - -The Allas documentation includes two tutorials that are especially designed for Puhti and Mahti users: - -* [Examples for using Allas in CSC supercomputers](../data/Allas/allas-examples.md) - -* [Using Allas in batch jobs](../data/Allas/allas_batchjobs.md) - - diff --git a/docs/computing/webinterface/file-browser.md b/docs/computing/webinterface/file-browser.md index e891c625ef..742459593e 100644 --- a/docs/computing/webinterface/file-browser.md +++ b/docs/computing/webinterface/file-browser.md @@ -27,7 +27,7 @@ editor: ## Accessing Allas and LUMI-O -The **[Allas object storage service](../../computing/allas.md)** can also be +The **[Allas object storage service](../../data/Allas/index.md)** can also be accessed from the web interfaces using the file browser. To configure authentication for Allas: diff --git a/docs/data/Allas/accessing_allas.md b/docs/data/Allas/accessing_allas.md index bfcc908a8b..f04b864928 100644 --- a/docs/data/Allas/accessing_allas.md +++ b/docs/data/Allas/accessing_allas.md @@ -1,152 +1,117 @@ # Accessing Allas -## Gaining access -**Allas** access is based on CSC's customer projects. To be able to use Allas, you need to be a member of -a CSC project that has the permission to use Allas. If you do not have a CSC account, you must first register as a CSC user -and join or start a computing project for which Allas has been enabled. This can be done in the -MyCSC user portal: [https://my.csc.fi]( https://my.csc.fi). +![Allas access clients](img/allas-access-flavors.png) -Once you have Allas enabled you can access it from any machine or server that is connected to internet. This can be a your laptop, supercomputer at CSC, virtual machine in cloud or enven your phone. +The four main options for accessing Allas are: +* Web browser interfaces +* Command-line tools +* Graphical tools +* Other tools: Python and R libraries etc -## Accessing Allas from the web browser +The tool lists below are not complete or exclusive. Any tool that supports Swift or S3 protocols can in principle use Allas. +You can cross-use the Allas clients as long as you access Allas with the same protocol (Swift or S3). -At the moment CSC provides several web browser interfaces for Allas: - -**Allas Web UI** is a web-based interface designed to simplify the management of object storage in Allas. It provides an intuitive way to interact with your data without needing command-line tools. -It is an ideal option for users who prefer a visual interface over command-line tools for basic object storage operations. - -* [Allas Web UI Guide](./using_allas/allas-ui.md) -* [Access Allas Web UI](https://allas.csc.fi) - - -The **WWW interfaces of Puhti and Mahti** are connected to Allas. -These interfaces allow you to transfer files and directories between your local computer and Allas as well as -between CSC supercomputers and Allas. - -* [Instructions for using Allas in Puhti and Mahti web interfaces](../../computing/webinterface/file-browser.md) -* [Puhti web interface](https://www.puhti.csc.fi) -* [Mahti web interface](https://www.mahti.csc.fi) - - -The OpenStack Horizon web interface in **cPouta** provides easy-to-use basic functions for data management in Allas. This interface can only be used for files smaller than 5 GB. - -* [Web client – OpenStack Horizon Dashboard](./using_allas/web_client.md) -* [cPouta Web Interface](https://pouta.csc.fi) - - -**SD Connect** provides an interface for storing and sharing sensitive data. -This service is based on Allas but we don't recommend it for other than sensitive data. - -* [SD Connect instructions](../sensitive-data/sd_connect.md) -* [SD Connect interface](https://sd-connect.csc.fi) - - - -## Accessing Allas in the CSC computing environment and other Linux platforms +When choosing the tool for accessing Allas, consider: -CSC supercomputers Puhti and Mahti support many different command line tools for using Allas, These include +* Ease of getting started: web interfaces do not need installation and the connection configuration is easy. +* Ease of use: web interface and graphical tools are in general easier to get started for basic tasks. +* The amount of data you have to move, the web interfaces are not suitable for for big data transfers. +* Your other workflow, Python or R libraries may be useful, if you use these programming languages already for other tasks. +* The operating system of your local machine, some command-line and graphical tools support only Linux/Mac or Windows. +* [Allas protocol](introduction.md#protocols) of your choice, many of the command-line and graphical tools support only Swift or S3. +* [Packaging of files](introduction.md#file-sizes-and-packaging), in case of moving many files, `a-tools` packages them by default to a .tar file and adds metadata, other tools usually move files as they are. +* Sensitity of your data, for sensitive data use [tools that support client side encryption](allas_encryption.md). -* [**a-tools**](./using_allas/a_commands.md) for basic use: (Swift, optionally S3) -* [**rclone**](./using_allas/rclone.md) providing some advanced functions:** (Swift, optionally S3) -* [**swift**](./using_allas/swift_client.md) python client that provides wide range of functionalities (Swift) -* [**s3cmd**](./using_allas/s3_client.md) an S3 client and persistent Allas connections:** (S3) +To use Allas from Puhti or Mahti, see [Tutorial for using Allas in CSC supercomputers](allas-hpc.md). -Note that the tools listed above utilize two different protocols: _Swift_ and _S3_. Data uploaded using one protocol is not necessary compatible with another protocol. - -The software listed above can also be used on other devices, for example a virtual machine running in cPouta or your own laptop. - -In Puhti and Mahti the Allas tools listed above are installed by CSC and provided through _allas_ module. -In order to use Allas in Puhti or Mahti, first load the Allas module: -```text -module load allas -``` -Allas access for a specific project using swift protocol can then be enabled: -```text -allas-conf -``` -To enable S3 protocol, use option `-m S3` -```text -allas-conf -m S3 -``` -The `allas-conf` command prompts for your CSC password (the same that you use to login to CSC servers). It lists your Allas projects and asks you to define a project (if not already defined as an argument). `allas-conf` generates an `rclone` configuration file for the Allas service and authenticates the connection to the selected project. `allas-conf` enables you to use only one Allas project at a time in one session. The project you are using in Allas does not need to match the project you are using in Puhti or Mahti, and you can switch to another project by running `allas-conf` again. - -In the case of the Swift protocol, the authentication information is stored in the `OS_AUTH_TOKEN` and `OS_STORAGE_URL` environment variables and is valid for up to eight hours. However, you can refresh the authentication at any time by running `allas-conf` again. The environment variables are set only for the current login session, so you need to configure authentication individually for each shell with which you wish to access Allas. - -In the case of the S3 protocol, the authentication information is stored in configuration files located in your home directory on the device. The same authentication is used for all login sessions and it does not have an expiration time. - -Once an Allas connection is configured, you can start using the object storage with the tools listed above. - -Basic Allas operations with different tools. - -| Tool | List objects in bucket _buck_123_ | Upload file _data1.txt_ to bucket _buck_123_ | Download file _data1.txt_ from bucket _buck_123_ | -|-------|-----------------------------------|----------------------------------------------|-------------------------------------------------| -| [a-commands](using_allas/a_commands.md) |`a-list buck_123` | `a-put data1.txt -b buck_123` | `a-get buck_123/data1.txt.zst` | -| [rclone (swift)](using_allas/rclone.md) |`rclone ls allas:buck_123` | `rclone copy data1.txt allas:buck_123/` | `rclone copy allas:buck_123/data1.txt ./`| -| [rclone (S3)](using_allas/rclone.md) |`rclone ls s3allas:buck_123` | `rclone copy data1.txt s3allas:buck_123/` | `rclone copy s3allas:buck_123/data1.txt ./`| -| [Swift](using_allas/swift_client.md) |`swift list buck_123` | `swift upload buck_123 data1.txt` | `swift download buck_123 data1.txt` | -| [s3cmd](using_allas/s3_client.md)\* |`s3cmd ls s3://buck_123` | `s3cmd put data1.txt s3://buck_123/` | `s3cmd get s3://buck_123/data1.txt` | +## Web browser interfaces +At the moment CSC provides several web browser interfaces for Allas: +| Web interface | Instructions |SWIFT support | S3 support | Use cases | Limits | +| ----- | ------------- | ---------- | --------- | ------- | ------- | +| [Allas web UI](https://allas.csc.fi) | [Instructions](./using_allas/allas-ui.md) | ✔ | | General first choice, share data with another project | Max 5 GB files) +| [Puhti web UI](https://www.puhti.csc.fi) | [Instructions](../../computing/webinterface/file-browser.md) | ✔ | ✔ | Moving data to/from Puhti or local, also S3 usage and LUMI-O | Max 10 GB file uploads from local | +| [Mahti web UI](https://www.mahti.csc.fi) | [Instructions](../../computing/webinterface/file-browser.md) | ✔ | ✔ | Moving data to/from Mahti or local, also S3 usage and LUMI-O | Max 10 GB file uploads from local | +| [cPouta web UI](https://pouta.csc.fi) | [Instructions](./using_allas/web_client.md) | ✔ | | Make your bucket public | Max 5 GB files, uploading/downloading only a single file at a time. | +| [SD Connect ](https://sd-connect.csc.fi) | [Instructions](../sensitive-data/sd_connect.md) | ✔ | | Sensitive data | | -## Accessing Allas with Windows or Mac +## Commandline tools -In addition to the Web interfaces listed above, you can access Allas from you Windows or Mac computer with locally installed client software. -For example following tools can be used: +To access Allas with **command line commands**, client software supporting the _Swift_ or _S3_ protocol is required. This is the most flexible way to access Allas, but it is a little bit more complicated to get started. -* [Cyberduck](./using_allas/cyberduck.md) provides easy to use graphical interface for moving data between local computer and Allas. -* [Rclone](./using_allas/rclone_local.md) is a command line tool that provides a very effective way to use Allas on any operating system. -* [a-tools](./using_allas/a_commands.md) are Allas-specific commands that can be installed on macOS and Linux devices, but not ones running a Windows operating system. +| Tools | SWIFT support | S3 support | Linux/Mac | Windows | +| ----- | ------------- | ---------- | --------- | ------- | +| [a-commands](./using_allas/a_commands.md) | ✔ | ✔ | ✔ | - | +| [rclone](./using_allas/rclone.md) | ✔ | ✔ | ✔ | ✔ | +| [swift python-swiftclient](./using_allas/swift_client.md) | ✔ | | ✔ | | +| [s3cmd](./using_allas/s3_client.md) | | ✔ | ✔ | | +| [aws-cli](https://s3browser.com/) | | ✔ | ✔ | ✔ | -The list above is not complete or exclusive. Any tool that supports Swift or S3 protocols can in principle use Allas. +Additionally for exmple `curl` and `wget` can be used for downloading public objects or objects with temporary URLs. -## Copying files directly between object storages +## Graphical tools -Rclone can also be used to directly copy files from another object storage (e.g. [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html), [Google cloud](https://cloud.google.com/learn/what-is-object-storage), [CREODIAS](https://creodias.eu/cloud/cloudferro-cloud/storage-2/object-storage/),...) to Allas. For this both credentials need to be stored in a Rclone configuration file in the users home directory (`.config/rclone/rclone.conf`). An example is shown below: +| Tools | SWIFT support | S3 support | Linux/Mac | Windows | +| ----- | ------------- | ---------- | --------- | ------- | +| [Cyberduck](./using_allas/cyberduck.md) | ✔ | ✔ | ✔ | ✔ | +| [WinSCP](https://winscp.net/eng/index.php) | | ✔ | | ✔ | +| [S3browser](https://s3browser.com/) | | ✔ | | ✔ | -``` -[s3allas] -type = s3 -provider = Other -env_auth = false -access_key_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -endpoint = a3s.fi -acl = private +WinSCP has generally rather slow data transfer speed for S3, so likely not suitable for bigger amounts of data. -[otherobjectstorage] -type = s3 -provider = Other -env_auth = false -access_key_id = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy -secret_access_key = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy -endpoint = yourotherendpoint.com -acl = private -``` -The configuration for Allas is added automatically when configuring Allas in s3 mode +## Other tools: Python and R libraries etc -`source allas_conf --mode s3cmd` . +* Python: + * [Python with SWIFT](using_allas/python_swift.md) + * [Python with S3 with `boto3`](using_allas/python_boto3.md). + * [Geoscience related examples how Allas can be used in Python scripts](https://github.com/csc-training/geocomputing/tree/master/python/allas) +* R + * [aws.s3 R package](https://cloud.r-project.org/web/packages/aws.s3/index.html) can be used for working with Allas with S3 protocol + * [Geoscience related example how Allas can be used in R scripts](https://github.com/csc-training/geocomputing/blob/master/R/allas/working_with_allas_from_R_S3.R), inc. aws.s3 set up. +* [Nextcloud front end](allas-nextcloud.md) Can be set up in Pouta to get additional functionality. -After creating/updating this file, Rclone can be used to copy files +These Python and R libraries can be installed to all operating systems. -`rclone copy otherobjectstorage:bucket-x/object-y s3allas:bucket-z/object-a` +## Clients comparison -or list files from either Allas or the other object storage by using the respective name +A _web client_ is suitable for using the basic functions. *a-commands* offer easy-to-use functions for using Allas either via a personal computer or supercomputer. Power users might want to consider the clients _rclone_, _Swift_ and _s3cmd_. The table displays the core functions of the power clients concerning data management in Allas. -`rclone lsf otherobjectstorage: `. +| | Allas Web UI | a-commands | rclone | Swift | s3cmd | +| :----- | :-----: | :----: | :----: | :-----: | :----: | +| Usage | _Basic_ | _Basic_ | _Power_ |_Power_ | _Power_ | +| **Create buckets** | ✔ |✔ | ✔ | ✔ | ✔ | +| **Upload objects** | ✔ | ✔ | ✔ | ✔ | ✔ | +| **List** | | | | | | +|        objects | ✔ | ✔ | ✔ | ✔ | ✔ | +|        buckets | ✔ | ✔| ✔ | ✔ | ✔ | +| **Download** | | | | | +|        objects | ✔ |✔ | ✔ | ✔ | ✔ | +|        buckets | ✔ | |✔ | ✔ | ✔ | +| **Remove** | | | | | | +|        objects | ✔ | ✔ | ✔ | ✔ | ✔ | +|        buckets | ✔•• | ✔ | ✔ | ✔ | ✔•• | +| **Managing access rights** | | | | | +|        public/private | | ✔| | ✔ | ✔ | +|        read/write access
       to another project | ✔ | ✔ | | ✔| ✔ | +|        temp URLs | | | | ✔ | ✔ | +| **Set lifecycle policies** | | | | | ✔ | +| **Move objects** | | | | ✔ | ✔ | +| **Edit metadata** | | | | ✔ | ✔ | +| **Download whole project** | | | ✔ | ✔ | | +| **Remove whole project** | | | ✔ | ✔ | | -## Other ways of accessing Allas +
•• Only empty buckets
-* Python: - * [Python with SWIFT](using_allas/python_swift.md) - * [Python with S3 with `boto3`](using_allas/python_boto3.md). - * [Geoscience related examples how Allas can be used in Python scripts](https://github.com/csc-training/geocomputing/tree/master/python/allas) -* [Nextcloud front end](allas-nextcloud.md) Can be set up in Pouta to get additional functionality. -* R - * [aws.s3 R package](https://cloud.r-project.org/web/packages/aws.s3/index.html) can be used for working with Allas with S3 protocol - * [Geoscience related example how Allas can be used in R scripts](https://github.com/csc-training/geocomputing/blob/master/R/allas/working_with_allas_from_R_S3.R), inc. aws.s3 set up. +## Files larger than 5 GB +Files larger than 5 GB are divided into smaller segments during upload. +* Most tools split large files automatically +* With _Swift_, you can use the _Static Large Object_: [swift with large files](using_allas/swift_client.md#files-larger-than-5-gb) +After upload, s3cmd connects these segments into one large object, but in case of swift based uploads (a-put, rclone , swift) the large files are also stored as several objects. This is done automatically to a bucket that is named by adding extension `_segments` to the original bucket name. For example, if you would use _a-put_ to upload a large file to bucket _123-dataset_ the actual data would be stored as several pieces into bucket _123-dataset_segments_. The target bucket _123_dataset_ would contain just a front object that contains information what segments make the stored file. Operations performed to the front object are automatically reflected to the segments. Normally users don't need to operate with the _segments_ buckets at all and objects inside these buckets should not be deleted or modified. +It is important not to mix Swift and S3, as these protocols are not fully mutually compatible. diff --git a/docs/data/Allas/allas-examples.md b/docs/data/Allas/allas-hpc.md similarity index 93% rename from docs/data/Allas/allas-examples.md rename to docs/data/Allas/allas-hpc.md index ab03a2fda2..70138964d3 100644 --- a/docs/data/Allas/allas-examples.md +++ b/docs/data/Allas/allas-hpc.md @@ -1,17 +1,34 @@ -# Examples for using Allas in CSC supercomputers +# Tutorial for using Allas in CSC supercomputers CSC supercomputers, Puhti and Mahti, do not provide permanent storage space for -research data. Data that needs to be stored for a longer time than just a few +research data. The supercomputer's own storage has a policy to [delete idle data](../../computing/usage-policy.md#disk-cleaning), so the data must be moved to Allas after computing. + +Data that needs to be stored for a longer time than just a few weeks should be copied to Allas object storage service. Allas provides a platform that you can use to store your data as long as your CSC project is active. In addition to storage, Allas can be used for transporting data between different servers and sharing data with other users. -* [Allas user guide](index.md) +One of the main use cases of Allas is to store data while it is not actively used in the CSC supercomputers. When you start +working, you stage in the data from Allas. And when the data is no longer actively used, it can be staged out to Allas. + +If you have not used Allas before, then start with reading **[Allas general introduction](introduction.md)**, which includes many important things to know. + +Additional information regarding: + +* [Using Allas in batch jobs](allas_batchjobs.md) +* [Using Allas and LUMI-O from LUMI supercomputer](allas_lumi.md) + +To upload or download data between Allas and supercomputers there are a lot of different tools. You can use [Puhti or Mahti web interface](accessing_allas.md#web-browser-interfaces), [commandline tools](accessing_allas.md#commandline-tools) or [Python, R or other tools](accessing_allas.md#graphical-tools). In Puhti and Mahti, the Allas commandline tools are installed by CSC and provided through **allas module**. + +By default, CSC computing projects do not have access to Allas. The project manager has to additionally apply for Allas service in [MyCSC](https://my.csc.fi/). +[Allas introduction page](introduction.md#) describes how to do it and also about default quotas and how to apply for more storage space. + +## Examples This tutorial provides four examples for using Allas on Puhti and Mahti. The -examples are based on interactively executed commands, and thus apply only for -relatively small datasets (max. some hundreds of GBs). +examples are based on interactively executed commands, and thus examples 1, 2 and 4 apply only for +relatively small datasets (max. some hundreds of GBs). The third example suits also for bigger datasets. 1. [The first example](#example-1-using-allas-with-a-commands) uses the *a-commands* (`a-put`, `a-get`) for uploading data from Mahti to Allas, and @@ -28,27 +45,6 @@ The a-commands are better suited for cases where the data is mainly used within the CSC computing environment (Puhti, Mahti). The second option, Rclone, is good for cases when the data will be used outside CSC too. -## Getting access to Allas - -By default, CSC computing projects do not have access to Allas. Thus, the first -thing is to -[add the Allas service to your project](../../accounts/how-to-add-service-access-for-project.md). -This is done in the [MyCSC](https://my.csc.fi) portal. Note that only the -project manager can apply for access. - -The default storage quota in Allas is 10 TB. As this space is shared with all -project members, it is possible that the space is not sufficient. In that case, -you should estimate how much space is needed and request more space. The -request should be sent to [CSC Service Desk](../../support/contact.md). Please -include in your quota request: - -1. The ID/name of your project -2. The amount of Allas space needed -3. A short description of the data to be stored - -Note that the data stored in Allas -[consume billing units of the project](../../accounts/billing.md). - ## Example 1: Using Allas with a-commands ### A. Uploading data from Mahti to Allas diff --git a/docs/data/Allas/index.md b/docs/data/Allas/index.md index 4af8001372..274a7ff12d 100644 --- a/docs/data/Allas/index.md +++ b/docs/data/Allas/index.md @@ -5,18 +5,17 @@ This Allas object storage service user guide consists of following parts: ## Introduction * [Introduction](./introduction.md) An overview of the service. Recommended reading before starting to use Allas. - * [Common use cases](./using_allas/common_use_cases.md) Some examples. ## Accessing Allas * [Accessing Allas](./accessing_allas.md) A summary of the most commonly used Allas tools. Including: - * [Gaining Access](./accessing_allas.md#gaining-access) - * [Accessing Allas from a web browser](./accessing_allas.md#accessing-allas-from-the-web-browser) - * [Accessing Allas in the CSC computing environment and other linux-platforms](./accessing_allas.md#accessing-allas-in-the-csc-computing-environment-and-other-linux-platforms) - * [Accessing Allas with Windows or Mac](./accessing_allas.md#accessing-allas-with-windows-or-mac) - * [Other ways of accessing Allas](./accessing_allas.md#other-ways-of-accessing-allas) + * [Gaining Access](./introduction.md#gaining-access) + * [Accessing Allas from a web browser](./accessing_allas.md#web-browser-interfaces) + * [Accessing Allas with command-line tools](./accessing_allas.md#commandline-tools) + * [Accessing Allas with graphical tools](./accessing_allas.md#graphical-tools) + * [Accessing Allas with other tools: Python and R libraries etc](./accessing_allas.md#other-tools-python-and-r-libraries-etc) ### Understanding error messages @@ -25,9 +24,22 @@ This Allas object storage service user guide consists of following parts: ## Tutorials and other documents -* [Using Allas interactively in Puhti and Mahti](./allas-examples.md) +* [Using Allas interactively in Puhti and Mahti](./allas-hpc.md) * [Using Allas in batch jobs](./allas_batchjobs.md) * [Using Allas to host a data set for a research group](./allas_project_example.md) * [Using Allas to transport sensitive data](../sensitive-data/sequencing_center_tutorial.md) * [Tools for client side encryption for Allas](./allas_encryption.md) * [Using Allas from LUMI supercomputer](./allas_lumi.md) + +## Allas FAQ + +* [How to move data from Puhti to Allas and vice versa?](../../support/faq/how-to-move-data-between-puhti-and-allas.md) +* [Can I have a personal project or bucket in Allas?](../../support/faq/can-i-have-a-personal-project-or-bucket-in-allas.md) +* [Is it possible to move data in Allas to new project?](../../support/faq/is-it-possible-to-move-data-in-allas-to-new-project.md) +* [Is it possible to make data in Allas read-only?](../../support/faq/is-it-possible-to-make-data-in-allas-read-only.md) +* [What to do for data in Allas after project ends?](../../support/faq/what-to-do-for-data-in-allas-after-project-ends.md) +* [How to make a bucket public?](../../support/faq/how-to-make-a-bucket-public.md) +* [How to get Allas S3 credentials](../../support/faq/how-to-get-Allas-s3-credentials.md) +* [How to mount my Allas S3 bucket to a VM running in cPouta](../../support/faq/how-to-mount-os-bucket-for-cpouta.md) +* [How to backup a Postgres DB into Allas from Rahti?](../../support/faq/backup-postgres-allas.md) +* [What is the difference between SD Connect and Allas?](../../support/faq/sensitive-data-connect.md#what-is-the-difference-between-sd-connect-and-allas) \ No newline at end of file diff --git a/docs/data/Allas/introduction.md b/docs/data/Allas/introduction.md index 3fc416f665..7582cf9e5d 100644 --- a/docs/data/Allas/introduction.md +++ b/docs/data/Allas/introduction.md @@ -12,142 +12,140 @@ The stored objects can be of any data type, such as images or compressed data fi **Benefits** * The object storage can handle practically any static data. - * The data can be accessed from anywhere using the URL. + * The data can be accessed from anywhere. * The data can have different levels of access control. * The data can have lifecycle policy set. + * You can access Allas from any machine or server that is connected to internet. This can be a your laptop, supercomputer at CSC, virtual machine in cloud or enven your phone. **Limitations** * Specific tools are required to use the object storage. The object storage cannot be properly mounted for local disk-like usage. There are some tools that can do this, but they have their limitations. For example, _svfs_ can be used to mount _Swift_ as a file system, but it uses _FUSE_ which is slow. * It is unsuitable for files that change constantly during their lifetime (e.g. most SQL databases). * The data cannot be modified while it is in Allas. It must be downloaded to a server for processing, and the previous version replaced with a new one. - * In case of swift protocol, files larger than 5 GB are divided into smaller segments. Normally, this is done automatically during the upload. See [Files larger than 5 GB](./using_allas/common_use_cases.md#files-larger-than-5-gb). + * In case of swift protocol, files larger than 5 GB are divided into smaller segments. Normally, this is done automatically during the upload. See [Files larger than 5 GB](accessing_allas.md#files-larger-than-5-gb). + +See also the [common use cases](./using_allas/common_use_cases.md). + +## System characteristics + +In Allas, objects are stored in buckets. A bucket is a data object container. Buckets should not be confused with _dockers_ or other computing containers. A bucket functions similarly to a file system directory, except that there can only be one level, i.e. buckets cannot contain other buckets. + +![Allas projects and buckets](img/allas_projects_and_buckets.PNG) +**Figure** Data structure in Allas + +## Different ways to use Allas + +Allas can be used from CSC computing environment or any other laptop or server connected to Internet. There are many tools for working with Allas: + +![Allas access clients](img/allas-access-flavors.png) + +* Web browser interfaces +* Command-line tools +* Graphical locally installed tools +* Other tools: Python and R libraries etc + +[The accessing Allas page](accessing_allas.md) describes the different options in detail. + +## Gaining access + +To be able to use Allas, you need to have: + +* [CSC account](../../accounts/how-to-create-new-user-account.md) +* [CSC a computing project](../../accounts/how-to-create-new-project.md) for which [Allas service has been enabled](../../accounts/how-to-add-service-access-for-project.md) + ## Billing and quotas Allas usage is based on CSC projects. All project members have equal access rights to the storage area that has been granted for the project. In practice, this means that if one project member uploads data to Allas, all other project members can also read, edit and delete the data. Allas itself does not store any information about who has uploaded the data to Allas. -The default quota for a new project is 10 TB, but that can be increased if needed. Allas is the preferred storage site for any large datasets in the CSC environment, so you should not hesitate to request a larger quota for Allas, if you work with larger data sets. +**The default quota for a new project is 10 TB**, but that can be increased if needed. Allas is the preferred storage site for any large datasets in the CSC environment, so you should not hesitate to request a larger quota for Allas, if you work with larger data sets. To increase your Allas quota, please send a request to: `servicedesk@csc.fi` In the request, define what Allas project you are using, how large storage space is needed and what kind of data will be stored to Allas. +Note that the data stored in Allas +[consume billing units of the project](../../accounts/billing.md). +In Allas, billing is based on the amount of data stored. The rate is 1 BU/TiBh, i.e. 1 TB of data stored in Allas consumes 24 BU in a day and 8760 BU in a year. + +Unlike most other object storage providers, CSC does not charge for object storage network transfers or API calls. **The default quotas for projects:** +### Resource limits + +Allas has technical limits, that normally can not be increased: + | Resource | Limit | | :-------- |:------- | -| Storage amount | 10 TiB | | Buckets per project | 1 000 | | Objects per bucket | 500 000 | -The maximum for the "Objects per bucket" quota is 500 000. By exeeding the limit you might get your bucket in a situation where you can't make any operations. If you need more objects than this, please plan on spreading the objects across multiple buckets. Spreading data to multiple buckets will give a better performance whenever writing objects. +If you a lot of objects, please plan on spreading the objects across multiple buckets. Spreading data to multiple buckets will give a better performance whenever writing objects. -Storing data in Allas consumes _billing units_. In Allas, billing is based on the amount of data stored. The rate is 1 BU/TiBh, i.e. 1 TB of data stored in Allas consumes 24 BU in a day and 8760 BU in a year. +## Protocols -Unlike most other object storage providers, CSC does not charge for object storage network transfers or API calls. +The object storage service is provided over two different protocols, **Swift** and **S3**. From the user perspective, one of the main differences between S3 and Swift is authentication. +* The token-based **Swift authentication** used remains valid for **eight hours* at a time. +* The key-based **S3**, the connection can stay **permanently open**. +The permanent connection of S3 is practical in many ways, but it includes a security aspect: if the server where Allas is used is compromised, the object storage space will be compromised as well. Due to this security concern, Swift is the recommended protocol for multiple-user servers such as Mahti and Puhti. Thus, for example, the CSC-specific `a-commands`, as well as the `rclone` configuration in Puhti and Mahti, are by default based on Swift. However, in some cases, the permanent connections provided by the S3 protocol may be the most reasonable option, for example, in personal virtual machines running in cPouta. -## Different ways to use Allas +The Swift and S3 protocols are not mutually compatible when handling objects. For small objects that do not need to be split during the upload, the protocols can be used interchangeably, but split objects can be accessed only with the protocol that was used for uploading them. The size limit for splitting an object depends on the settings and protocol. The limit is typically between 500 MB and 5 GB. -You cannot mount Allas direcly to a computer. This means that in order to use Allas, you need software tools to access it. There are four main ways to access Allas: +Generic recommendations for selecting the protocol: -![Allas access clients](img/allas-access-flavors.png) + * If possible, use the _Swift_ protocol. It is better supported. + * In any case, choose only one of the protocols. Do not mix _S3_ and _Swift_. + +Note, that some [Allas clients](accessing_allas.md) support only one of these protocols. -1. In the CSC computing environment (Puhti and Mahti), there are ready-to-use tools provided by CSC to access Allas. These tools are mostly the same that can also be installed in any Linux environment, e.g. a virtual machine in cPouta or a local Linux server. -In the CSC computing environment, Allas should be used to store any data that needs to be preserved for longer than a few weeks. The supercomputer's own storage has a policy to delete idle data, so the data must be moved to Allas after computing. See [Computing disk environment](../../computing/disk.md) +## Naming buckets and objects -2. [Allas Web UI](https://allas.csc.fi) provides a web-based graphical interface for managing object storage in Allas. It allows users to create and manage buckets, upload and download objects (up to 5 GiB per file), and configure sharing permissions. Allas Web UI offers a convenient way to interact with Allas without requiring command-line tools, making it suitable for users who prefer a visual approach. More information can be found in the [Allas Web UI guide](./using_allas/allas-ui.md). +Each bucket has a name that must be unique across all Allas users. If another user has a bucket called `test`, another bucket called `test` cannot be created. All bucket names are public, so please do not include any confidential information in the bucket names. You may, for example, use your project ID, e.g. _2000620-raw-data_. It is not possible to rename a bucket. -3. WWW access to Allas is provided by the web interface of the cPouta cloud environment [https://pouta.csc.fi](https://pouta.csc.fi). No special software is required to access Allas with a browser, making this the by far simplest way to access Allas. On the other hand, the browser user interface has a number of limitations compared to other clients, the most notable of which are lower performance and uploading/downloading only a single file at a time. Instructions for accessing and using Allas with a browser: [OpenStack Horizon web interface](./using_allas/web_client.md) +Object URLs can be in the DNS format, e.g. _https://a3s.fi/bucketname/objectname_. Please use a valid DNS name (RFC 1035). We recommend not using upper case or non-ASCII (ä, ö etc.) characters. -4. To access Allas with command line commands, client software supporting the _Swift_ or _S3_ protocol is required. This is the most flexible way to access Allas, but it requires more effort than other access methods. Instructions to use a command line client: [Accessing Allas with Linux](./accessing_allas.md#accessing-allas-in-the-csc-computing-environment-and-other-linux-platforms). +For object names, you can use [pseodo folders](terms_and_concepts.md#pseudo-folder), which some Allas clients display as folders. -5. To access Allas with a GUI client, a suitable GUI client is required. The client needs to be capable to use the _Swift_ or _S3_ access protocol. Instructions to use a GUI client: [Accessing Allas with Windows and Mac](./accessing_allas.md#accessing-allas-with-windows-or-mac). +## File sizes and packaging -See also the [common use cases](./using_allas/common_use_cases.md). +File size considerations: +* It is better to store a few large objects than many small objects. +* Keeping your objects under 5 GB if often practical, because bigger objects are chunked at upload. +* Using over 100 GB objects may cause problems because of long upload/download times. -## Protocols - -The object storage service is provided over two different protocols, _Swift_ and _S3_. From the user perspective, one of the main differences between S3 and Swift is authentication. The token-based Swift authentication used in Allas remains valid for eight hours at a time, but in the key-based S3, the connection can stay permanently open. The permanent connection of S3 is practical in many ways, but it includes a security aspect: if the server where Allas is used is compromised, the object storage space will be compromised as well. +When moving your data to Allas, you can take few different strategies: -Due to this security concern, Swift is the recommended protocol for multiple-user servers such as Mahti and Puhti. Thus, for example, the CSC-specific _a-commands_, as well as the standard _rclone_ configuration in Puhti and Mahti, are based on Swift. However, in some cases, the permanent connections provided by the S3 protocol may be the most reasonable option, for example, in personal virtual machines running in cPouta. +* Create one package of all your files, for example .tar or .zip and move the package to Allas. This suits for use cases, when the amount of data is not too big (< 100Gb). Allas is used as storage of data and for active use the data is moved elsewhere, for example CSC computing services. In this scenario, it is difficult to access a single original file. From Allas clients `a-commands` has best support for this. +* Move your files as such to Allas, so that in Allas would be as many files than originally. This suits for use cases, when originally the files have reasonable size and there is not too many of them. This is reasonable also, if access to single files is important. Many of the Allas clients support this. +* A combination of these approaches, so that some subsets of files are packaged for Allas. If you have a lot of small files and the total amount of data is big, then it likely makes sense to package for example different folders to own files that are then stored to Allas. -The Swift and S3 protocols are not mutually compatible when handling objects. For small objects that do not need to be split during the upload, the protocols can be used interchangeably, but split objects can be accessed only with the protocol that was used for uploading them. The size limit for splitting an object depends on the settings and protocol. The limit is typically between 500 MB and 5 GB. -Generic recommendations for selecting the protocol: +## Access rights to data - * If possible, use the _Swift_ protocol. It is better supported. - * In any case, choose only one of the protocols. Do not mix _S3_ and _Swift_. - * It is better to store a few large objects than many small objects. - * Using over 100 GB objects may cause problems because of long upload/download times. - -## Clients - -Allas is accessed via a client software that takes care of moving data to and from Allas and managing data objects. There are several different kinds of client software for accessing the object storage servers. Allas can be used with any object storage client that is compatible with the Swift or S3 protocol. - -| Client | Notes | -| :-------- | :------- | -| [Allas Web UI](./using_allas/allas-ui.md) | Use via [https://allas.csc.fi](https://allas.csc.fi). | -| [SD Connect](../sensitive-data/sd_connect.md) | Sensitive data orinented [Web interface for Allas](https://sd-connect.csc.fi). | -| [a-commands](./using_allas/a_commands.md) | Provides easy-to-use tools for basic use. Requires Rclone, Swift and OpenStack. | -| [swift python-swiftclient](./using_allas/swift_client.md)| The versatile Swift client. | -| [s3cmd](./using_allas/s3_client.md) | The versatile S3 client (version 2.0.2 or later). | -| [Python with S3](./using_allas/python_boto3.md) | Programmatic access. | -| [Python with SWIFT](./using_allas/python_swift.md) | Programmatic access. | -| [rclone](./using_allas/rclone.md) | Versatile command line tool for both Swift and S3 protocols | -| libs3 | | -| python-openstackclient | | -| aws-cli | aws-cli and the boto3 Python library. | -| curl | Extremely simple to use with public objects and temporary URLs. | -| wget | The same as curl. | -| [web client](./using_allas/web_client.md) | Use via [https://pouta.csc.fi](https://pouta.csc.fi). Provides basic functions. | - -### Client operations - -A _web client_ is suitable for using the basic functions. *a-commands* offer easy-to-use functions for using Allas either via a personal computer or supercomputer. Power users might want to consider the clients _rclone_, _Swift_ and _s3cmd_. The table displays the core functions of the power clients concerning data management in Allas. - -| | Allas Web UI | a-commands | rclone | Swift | s3cmd | -| :----- | :-----: | :----: | :----: | :-----: | :----: | -| Usage | _Basic_ | _Basic_ | _Power_ |_Power_ | _Power_ | -| **Create buckets** | ✔ |✔ | ✔ | ✔ | ✔ | -| **Upload objects** | ✔ | ✔ | ✔ | ✔ | ✔ | -| **List** | | | | | | -|        objects | ✔ | ✔ | ✔ | ✔ | ✔ | -|        buckets | ✔ | ✔| ✔ | ✔ | ✔ | -| **Download** | | | | | -|        objects | ✔ |✔ | ✔ | ✔ | ✔ | -|        buckets | ✔ | |✔ | ✔ | ✔ | -| **Remove** | | | | | | -|        objects | ✔ | ✔ | ✔ | ✔ | ✔ | -|        buckets | ✔•• | ✔ | ✔ | ✔ | ✔•• | -| **Managing access rights** | | | | | -|        public/private | | ✔| | ✔ | ✔ | -|        read/write access
       to another project | ✔ | ✔ | | ✔| ✔ | -|        temp URLs | | | | ✔ | ✔ | -| **Set lifecycle policies** | | | | | ✔ | -| **Move objects** | | | | ✔ | ✔ | -| **Edit metadata** | | | | ✔ | ✔ | -| **Download whole project** | | | ✔ | ✔ | | -| **Remove whole project** | | | ✔ | ✔ | | - -
•• Only empty buckets
+It is possible to assign rights at bucket level or object level. In general there are 3 types of access policies: -## System characteristics +* Data is accessible only for CSC project members. Note, that also all project members have same writing rights, so any project member can (accidentally) delete any object. +* Data is public to anybody, accessible with URLs. +* Data is opened to another CSC project. -In Allas, objects are stored in buckets. A bucket is a data object container. Buckets should not be confused with _dockers_ or other computing containers. A bucket functions similarly to a file system directory, except that there can only be one level, i.e. buckets cannot contain other buckets. +More info in [Sharing data](using_allas/common_use_cases.md#sharing-data) use case. -![Allas projects and buckets](img/allas_projects_and_buckets.PNG) -**Figure** Data structure in Allas +## Backup -## Naming buckets +Allas data is spread across various servers, which protects against disk and server failures. But, this does not protect the data from e.g. accidental deletion. Please make regular backups of important data. -Each bucket has a name that must be unique across all Allas users. If another user has a bucket called "_test_", another bucket called "_test_" cannot be created. All bucket names are public, so please do not include any confidential information in the bucket names. You may, for example, use your project ID, e.g. _2000620-raw-data_. -Object URLs can be in the DNS format, e.g. _https://a3s.fi/bucketname/objectname_. Please use a valid DNS name (RFC 1035). We recommend not using upper case or non-ASCII (ä, ö etc.) characters. +## 7 steps to get started -It is not possible to rename a bucket. +1. [Get access](#gaining-access) to Allas as service. +2. Plan how to store your data to Allas: [naming](#naming-buckets-and-objects), [file packaging](#file-sizes-and-packaging), [access rights](#access-rights-to-data). +3. Decide, which [protocol](#protocols) and [client](accessing_allas.md) to use. +4. If moving data from/to your local machine, install the selected tool (not needed if using webinterfaces). +5. [Configure the connection](using_allas/allas-conf.md) to Allas. +6. Move data to/from Allas. +7. If you want to share the data publically or with another project, change access rights for your data. -The data is spread across various servers, which protects against disk and server failures. **Please note:** This does not protect the data from e.g. accidental deletion. Please make regular backups of important data. +For last two steps, see tool specific instructions. \ No newline at end of file diff --git a/docs/data/Allas/terms_and_concepts.md b/docs/data/Allas/terms_and_concepts.md index 065fba40ee..fc6623a6b2 100644 --- a/docs/data/Allas/terms_and_concepts.md +++ b/docs/data/Allas/terms_and_concepts.md @@ -24,17 +24,7 @@ You can display the checksum using the command `md5sum`. ### Client -The _client software_ is used to access an object storage service, such as Allas. There are different types of clients: - -* **Web browser-based access** via: - - [Allas Web UI](./using_allas/allas-ui.md) – a user-friendly graphical interface for managing buckets, uploading and downloading objects, and sharing data. - - [OpenStack Horizon web interface](./using_allas/web_client.md) – a basic web interface for managing small files (≤5 GiB) in Allas. - -* **Command-line clients** such as: - - [Swift](./using_allas/swift_client.md) and [s3cmd](./using_allas/s3_client.md) – for power users who need advanced control over object storage. - -* **Programmable interface (API)** for those integrating Allas with software applications. - +The _client software_ is used to access an object storage service, such as Allas. Allas can be accessed with several different [clients](accessing_allas.md). ### Metadata @@ -51,16 +41,14 @@ _Object storage_ refers to a computer data storage that manages data as objects ### OpenStack -The _OpenStack cloud management middleware_ can be used to access Allas. -The [OpenStack Horizon web interface](./using_allas/web_client.md) offers some basic functionalities for data management in Allas. +The [OpenStack](https://www.openstack.org/) cloud management middleware_ can be used to access Allas. -See [OpenStack](https://www.openstack.org/) for more information. ### Pseudo folder Buckets cannot contain other buckets. You can, however, use so-called _pseudo folders_. -If an object's name contains a forward slash `/`, it is interpreted as a folder separator. These are shown as folder listings when accessing the data via the web interface. These pseudo folders are automatically added if you upload whole folders using a command line client. +If an object's name contains a forward slash `/`, it is interpreted as a folder separator. These are shown as folder listings when accessing the data via come clients, for example the web interfaces. These pseudo folders are automatically added if you upload whole folders using a command line client. For example, if you add two objects ```bash diff --git a/docs/data/Allas/using_allas/a_commands.md b/docs/data/Allas/using_allas/a_commands.md index 228a5e4f40..e5d76971d4 100644 --- a/docs/data/Allas/using_allas/a_commands.md +++ b/docs/data/Allas/using_allas/a_commands.md @@ -1,6 +1,6 @@ # a-commands, easy and safe -The Allas object storage system can be used in multiple ways and for many purposes. In many cases, using Allas efficiently requires that the user know the features of both the object storage system and the software or protocol used to manage the data in Allas. +The Allas object storage system can be used in multiple ways and for many purposes. In many cases, using Allas efficiently requires that the user knows the features of both the object storage system and the software or protocol used to manage the data in Allas. For users who simply want to use Allas for storing data that is in the CSC computing environment, CSC provides a set of commands for managing and moving data between the CSC computing environment and Allas: @@ -22,7 +22,7 @@ For users who simply want to use Allas for storing data that is in the CSC compu In addition to the above commands, there are separate tools for other purposes: - * __allas_conf__ : Set up and open a connection to Allas + * [__allas_conf__](allas-conf.md) : Set up and open a connection to Allas * [__allas-backup__](./a_backup.md) : Create a backup copy of a local dataset in a backup repository in Allas. * __allas-mount__ : Mount a bucket in allas to be used as a read-only directory in the local environment. * __allas-health-check__ : Check the integrity of over 5 GB objects in Allas. @@ -38,32 +38,18 @@ a-put --help # Example: Saving data from scratch directory to Allas -## Opening a connection +## Configuring a connection in supercomputers In order to use these tools in Puhti and Mahti, first load a-commands: ```text module load allas ``` -Then open a connection to Allas: -```text -allas-conf -``` -The connection remains open for eight hours. You can rerun the _allas-conf_ command at any time -to extend the validity of the connection for eight more hours or to switch to another Allas -project. -By default, _allas-conf_ lists your projects that have access to Allas, but if you know the name of the project, you -can also give it as an argument: -```text -allas-conf project_201234 -``` -Note that the Allas project does not need to be the same as the project you are using in Puhti or Mahti. +Configure Allas connection with [`allas-conf`](allas-conf.md). -If you are running big, multistep processes (e.g. batch jobs), it may be that your data management pipeline takes more than eight hours. In those cases you can add option `-k` to the `allas-conf` command. ```text -allas-conf -k +allas-conf ``` -With this option on, the password is stored into environment variable OS_PASSWORD. A-commands recognize this environment variable and when executed, automatically refresh the current Allas connection. ## Copying data between Puhti scratch directory and Allas @@ -107,7 +93,7 @@ By default, this tool performs the following operations: define the project that will be used to store the data. 2. In the case of a directory, the content of the directory is collected as a single file -using the `tar` command. +using the `tar` command. If you have a lot of data, this might not be a good option, consider then using some other [Allas client](../accessing_allas.md) that does not package files. 3. The packed data is uploaded to Allas using the `rclone` command and the _Swift_ protocol. @@ -369,7 +355,7 @@ Options: - **-t** **--target_file** Define a file name for the object for the object to be downloaded. - **-l** **--original_location** Retrieve the data to the original location in the directory structure. - **--asis** Download the object without unpacking tar files and uncompressing zst compressed data. -- **--s3cmd** Use S3 protocol and s3cmd command for data retrieval in stead of Swift protocol and rclone. +- **--s3s3cmd** Use S3 protocol and s3cmd command for data retrieval in stead of Swift protocol and rclone. At the moment, _a-get_ can download only one object at a time. If you need to download large number of objects you need to use loops. For example to download all the objects in bucket _bucket_123_ , you could use commands: @@ -469,5 +455,3 @@ a-put --nc my_data.b ``` You can check most commonly used settings from this sample [.a_tools_conf](https://github.com/CSCfi/allas-cli-utils/edit/master/.a_tools_conf) file. Copy the sample file to your home directory and un-comment and define the variables you wish to use. - - diff --git a/docs/data/Allas/using_allas/allas-conf.md b/docs/data/Allas/using_allas/allas-conf.md new file mode 100644 index 0000000000..e08611edd1 --- /dev/null +++ b/docs/data/Allas/using_allas/allas-conf.md @@ -0,0 +1,130 @@ +# Allas connection configuration + +The easiest general option to set up Allas connection configuration is to use `allas-conf` tool: + +* Sets up connection configuration for many different tools +* Swift or S3 connection +* Is itself a command-line tool. +* Available on CSC supercomputers +* Can be installed to Linux and Mac, but not Windows. +* Suits well, if only one CSC project at a time in use for Allas. + +Alternatively, one can use [Puhti or Mahti web interface](../../../computing/webinterface/file-browser.md#accessing-allas-and-lumi-o): + +* Sets up connection configuration for web interface file section and `rclone`, but not other Allas clients. +* Swift or S3 connection +* Available in web-interface, so no installations needed for using it, but the CSC project must have Puhti or Mahti service enabled. +* Suits well, if one or several CSC projects in use for Allas. + + +## `allas-conf` availability + +Available on CSC supercomputers with **allas** module. Can be (installed)[allas-conf-installation] to Linux and Mac, but not Windows. + +```text +module load allas +``` + +## `allas-conf` configure connection +### SWIFT connection + +is valid for up to **eight hours** + +Configure Allas access using Swift protocol: +```text +allas-conf +``` +The `allas-conf` command prompts for your CSC password (University/Haka password will not work here). It lists your Allas projects and asks you to define a project (if not already defined as an argument). + +By default, `allas-conf` lists your projects that have access to Allas, but if you know the name of the project, you +can also give it as an argument: +```text +allas-conf project_201234 +``` + +`allas-conf` enables you to use only one Allas project at a time in one session. You can switch to another project by running `allas-conf` again. + +Note that the Allas project does not need to be the same as the project you are using in Puhti or Mahti. + +`allas-conf` generates Swift configuration files for : `a-tools, `rclone` and `swift`. + +The authentication information is stored in the `OS_AUTH_TOKEN` and `OS_STORAGE_URL` environment variables. However, you can refresh the authentication at any time by running `allas-conf` again. The environment variables are set only for the current login session, so you need to configure authentication individually for each shell with which you wish to access Allas. + +If you are running big, multistep processes (e.g. batch jobs), it may be that your data management pipeline takes more than eight hours. In those cases you can add option `-k` to the `allas-conf` command. +```text +allas-conf -k +``` +With this option on, the password is stored into environment variable OS_PASSWORD. A-commands recognize this environment variable and when executed, automatically refresh the current Allas connection. + +### S3 connection + +To enable S3 protocol, use option `-m S3` +```text +allas-conf -m S3 +``` + +The same authentication is used for all login sessions and it does not have an expiration time. + +`allas-conf` generates configuration files in S3 mode for: +* `rclone`: `.config/rclone/rclone.conf` +* `s3cmd`: `~/.s3cfg` +* `aws`: credentials and S3 region in `~/.aws/credentials` and S3 endpoint in `~/.aws/config` files. + +Additionally, Python 'boto3' and R 'aws.s3' libraries use 'aws' configuration files. + +This saves the + +If you use these keys in other services, your should make sure that the keys always remain private. Any person who has access to these two keys, can access and modify all the data that the project has in Allas. + +If needed, you can deactivate an S3 key pair with command: + +``` +allas-conf --s3remove +``` + +!!! Note + Remember to be careful and security-aware when configuring S3 connection to Allas. The S3 keys are stored in a readable format in your home directory and anyone who can read your keys can access Allas until the keys are expilicitly revoked from Allas. Removing the keys from your own computer is not enough to deactivate them. + + +## `allas-conf` installation + +`allas-conf` can be installed only to Linux or Mac. + +1. Download the `allas_conf` script to set up the connection to your Allas project: `wget https://raw.githubusercontent.com/CSCfi/allas-cli-utils/master/allas_conf` +2. Install Rclone for Swift connection OR [OpenStack client](https://pypi.org/project/python-openstackclient/) for S3 connection + +If running `allas_conf` locally, you should modify all of the commands above with: + +* Add `source` the beginning +* Use `--user` option to define your CSC username . + +For example: + +```bash +source allas_conf --user your-csc-username -p your-csc-project-name +source allas_conf --user your-csc-username -p your-csc-project-name -m S3 +source allas_conf --user your-csc-username -p csc-project-name --s3remove +``` + +## S3 connection details + +To use Allas with S3 on a machine where `allas-conf` is not available or with tools not directly supported, usually the following information is needed: + +* S3 credentials: access key and secret key +* S3 endpoint: `a3s.fi` or `https://a3s.fi` +* S3 region: sometimes no settings needed, sometimes leave empty (````) + +The easiest way to get the S3 credentials is by configuring an [S3 connection](#s3-connection) on a CSC supercomputer (or some other machine that can run `allas_conf` tool) and see the keys from the printout of the command. + +``` +module load allas +allas-conf -m S3 +``` + +Later the keys can be found for example with: `less ~/.aws/credentials` + +If you wish to access Allas from a personal laptop or some other server with `s3cmd` or `aws` command-line tools, you can also copy the setting files as such. [Use any file transfer tool](../../moving/index.md), for example `scp`. + +* `aws`: copy the `~/.aws`-folder to your home directory to `C:\Users\username\.aws` on Windows or `~/.aws/` on Mac and Linux. +* `s3cmd`: copy the `~/.s3cfg`-file to your home directory on Mac and Linux. + diff --git a/docs/data/Allas/using_allas/allas-ui.md b/docs/data/Allas/using_allas/allas-ui.md index 3a2b4d8196..7487ee0d8d 100644 --- a/docs/data/Allas/using_allas/allas-ui.md +++ b/docs/data/Allas/using_allas/allas-ui.md @@ -19,7 +19,7 @@ Allas Web UI provides a simple-to-use web user interface for CSC Object Storage A bucket is a storage container for your objects. Follow these steps to create one: 1. Click on **Create bucket**. -2. Enter a bucket name (names cannot be modified later. See the [checklist for naming a bucket](../introduction.md#naming-buckets)). +2. Enter a bucket name (names cannot be modified later. See the [checklist for naming a bucket](../introduction.md#naming-buckets-and-objects)). 3. (Optional) Add tags for better organization and search abilities. 4. Click **Save**. @@ -32,7 +32,7 @@ You can upload objects in two ways: ### Uploading from the main page: 1. Click the **Upload** button on the dashboard. -2. Enter a bucket name (names cannot be modified later. See the [checklist for naming a bucket](../introduction.md#naming-buckets)). +2. Enter a bucket name (names cannot be modified later. See the [checklist for naming a bucket](../introduction.md#naming-buckets-and-objects)). 3. Select / "Drag & Drop" objects and click **Upload**. 4. A new bucket containing the objects will be created. @@ -99,7 +99,7 @@ To copy a bucket: ## Downloading Objects and Buckets !!! warning "" - Downloading buckets or folders larger than **5 GiB** is currently not supported. For larger downloads, consider using the **[Command Line Tools](./rclone.md)** instead. + Downloading buckets or folders larger than **5 GiB** is currently not supported. For larger downloads, consider using the **[Command Line Tools](../accessing_allas.md#commandline-tools)** instead. 1. Click the **Download** button next to the Bucket / Object. diff --git a/docs/data/Allas/using_allas/common_use_cases.md b/docs/data/Allas/using_allas/common_use_cases.md index 3a35379e6a..111a9f775f 100644 --- a/docs/data/Allas/using_allas/common_use_cases.md +++ b/docs/data/Allas/using_allas/common_use_cases.md @@ -1,52 +1,8 @@ # Common use cases -## Processing data in CSC supercomputers - -The CSC supercomputers provide disk environments for working with large datasets. These storage areas are however not intended for storing data that is not actively used. For example in the _scratch_ area of Puhti the un-used files are automatically removed after **180 days** if your project's scratch quota is less than 5 TiB. If it is 5 TiB or more, un-used files are deleted after **90 days**. - -One of the main use cases of Allas is to store data while it is not actively used in the CSC supercomputers. When you start -working, you stage in the data from Allas. And when the data is no longer actively used, it can be staged out to Allas. - -In CSC supercomputers, connection to Allas can be established with commands: -```text -module load allas -allas-conf -``` -After that you can: - -**List the data buckets and objects in Allas:** For listing we recommend [a-list](./a_commands.md#a-list). -```text -a-list -``` -The command above lists available data buckets in Allas. To list data objects in a bucket give command: -```text -a-list bucket_name -``` -alternatively you can use [rclone](./rclone.md) commands: -```text -rclone lsd allas: -rclone ls allas:bucket_name -``` -**Copy data from Allas to a supercomputer (Puhti or Mahti) (stage in):** For downloading we recommend [a-get](./a_commands.md#a-get-retrieves-stored-data) -```text -a-get bucket/object_name -``` -or [rclone copy](./rclone.md): -```text -rclone copy allas:bucket/object_name ./ -``` - -**Copy data from a Supercomputer to Allas (stage out):** For uploading we recommend [a-put](./a_commands.md#a-put-uploads-data-to-allas) -```text -a-put filename -``` -or [rclone copy](./rclone.md): -```test -rclone copy file.dat allas:/bucket_name -``` - -!!! note - Both a-put/a-get and rclone use Swift protocol on Allas. It is important not to mix Swift and S3, as these protocols are not fully mutually compatible. +## Storage for CSC computing projects + +The CSC supercomputers provide disk environments for working with large datasets. These storage areas are however not intended for storing data that is not actively used. The inactive data should be moved to Allas. See [Allas HPC tutorial](../allas-hpc.md) for further information. ## Sharing data @@ -57,20 +13,21 @@ The data can be accessed and shared in a variety of ways: * **Private – default:** By default, if you do not specify anything else, the contents of buckets can only be accessed by authenticated members of your project. **Private**/**Public** settings can be managed with: * [swift client](./swift_client.md#giving-another-project-read-and-write-access-to-a-bucket) Use this for buckets created/used by `a-put/a-get` or `rclone`. - * [Web client](./web_client.md#view-objects-via-the-internet) + * [Pouta web UI](./web_client.md#view-objects-via-the-internet) * [S3 client](./s3_client.md#s3cmd-and-public-objects) -* **Access control lists:** Access control lists (ACLs) work on buckets, not objects. With ACLs, you can share your data in a limited manner to other projects. You can e.g. grant a collaboration project authenticated read access to your datasets. +* **Access control lists:** Access control lists (ACLs) work on buckets, not objects. With ACLs, you can share your data in a limited manner to other projects. You can e.g. grant a collaboration project authenticated read access to your datasets. See [example how to share data to another CSC project](../allas_project_example.md). * **Temporary signed links** can be created with [s3cmd](./s3_client.md#publishing-objects-temporarily-with-signed-urls) . This kind of links can be used in cases where the data needs to be accessed over the internet without credentials, but is not supposed to remain publicly accessible. * **Public:** You can also have ACLs granting public read access to data, which is useful e.g. for permanently sharing public scientific results or public datasets. -## Static web content +## Publishing static web content A common way to use the object storage is storing static web content, such as images, videos, audio, pdfs or other downloadable content, and adding links to it on a web page, which can run either inside Allas or somewhere else, [like this example](https://a3s.fi/my_fishbucket/my_fish). -Uploading data to Allas can be done with any of the following clients: [web client](./web_client.md#upload-an-object), [a-commands](./a_commands.md#a-put-uploads-data-to-allas),[rclone](./rclone.md#create-buckets-and-upload-objects), [Swift](./swift_client.md#create-buckets-and-upload-objects) or [S3](./s3_client.md#create-buckets-and-upload-objects). +To publish a some content, first move it with a [Allas client](../accessing_allas.md) to Allas and then [make the files public](#sharing-data). + ## Storing data for distributed use @@ -78,7 +35,7 @@ There are several cases where you need to access data in several locations. In t ## Accessing the same data via multiple CSC platforms -Since the data in the object storage is available anywhere, you can access the data via both the CSC clusters and cloud services. This makes the object storage a good place to store data as well as intermediate and final results in cases where the workflow requires the use of e.g. both Allas and Puhti. +Since the data in the object storage is available anywhere, you can access the data via both the CSC supercomputers and cloud services. This makes the object storage a good place to store data as well as intermediate and final results in cases where the workflow requires the use of e.g. both cPouta and Puhti. ## Collecting data from different sources @@ -95,40 +52,3 @@ The object storage is also often used as a location for storing backups. It is a Allas-backup is not a real backup service. It only copies the data to another bucket in Allas which can be easily removed or overwrited by any authenticated user. - -## Files larger than 5 GB - -Files larger than 5 GB are divided into smaller segments during upload. - -* *a-put* and *rclone* split large files automatically: [a-put](./a_commands.md#a-put-uploads-data-to-allas) - -* Using _Swift_, you can use the _Static Large Object_: [swift with large files](./swift_client.md#files-larger-than-5-gb) - -* _s3cmd_ splits large files automatically: [s3cmd put](./s3_client.md#create-buckets-and-upload-objects) - - -After upload, s3cmd connects these segments into one large object, but in case of swift based uploads (a-put, rclone , swift) the large files are also stored as several objects. This is done automatically to a bucket that is named by adding extension `_segments` to the original bucket name. For example, if you would use _a-put_ to upload a large file to bucket _123-dataset_ the actual data would be stored as several pieces into bucket _123-dataset_segments_. The target bucket _123_dataset_ would contain just a front object that contains information what segments make the stored file. Operations performed to the front object are automatically reflected to the segments. Normally users don't need to operate with the _segments_ buckets at all and objects inside these buckets should not be deleted or modified. - -## Viewing - -In CSC supercomputers you can check the number of objects and the amount of stored data in your current Allas project with command: -```text -a-info -``` - -If you are using the _s3cmd client_, check your project's object storage usage: -```bash -s3cmd du -H -``` - -If you use the _Swift client_: -```bash -swift stat -``` - -Display how much space a bucket has used: -```bash -swift stat $bucketname -``` - -Please contact [servicedesk@csc.fi](mailto:servicedesk@csc.fi) if you have questions. diff --git a/docs/data/Allas/using_allas/python_boto3.md b/docs/data/Allas/using_allas/python_boto3.md index 6c983087f2..4142af8d20 100644 --- a/docs/data/Allas/using_allas/python_boto3.md +++ b/docs/data/Allas/using_allas/python_boto3.md @@ -51,37 +51,18 @@ installed. If you wish to use the library in another Python environment, you can use `pip` to [add it on top of an existing module](../../../support/tutorials/python-usage-guide.md#installing-python-packages-to-existing-modules). -## Configuring S3 -To use Allas with S3 and boto3, some configurations need to be set up: -* Credentials: access key and secret key -* S3 endpoint -* S3 region +### Credentials for accessing one CSC project -### Configurations for accessing a single CSC project - -The easiest way to set up S3 configuration for `boto3` is by -[configuring an S3 connection on a CSC supercomputer](s3_client.md#configuring-s3-connection-in-supercomputers). - -```bash +```text module load allas -allas-conf --mode S3 -``` - -This saves the credentials and S3 region in `credentials` and S3 endpoint in `config` file in the default `~/.aws/` folder. - -If you wish to access Allas from a personal laptop or some other server, -copy the `~/.aws`-folder to your home directory on that computer: `C:\Users\username\.aws` on Windows or `~/.aws/` on Mac and Linux. -[Use any file transfer tool](../../moving/index.md), for example `scp`. - -```bash -# Copy the aws configuration files to your home directory -scp -r @.csc.fi:~/.aws $HOME +allas-conf -m S3 ``` +`boto3` is internally using `aws`-library, so if you want to [copy your credentials](allas-conf.md#s3-connection-details) outside the supercomputer, follow instructions for `aws`. ### Credentials for accessing multiple CSC projects -Using `allas-conf --mode s3cmd` is straightforward, +Using `allas-conf -m S3` is straightforward, but it overwrites the existing credentials file when run, making it somewhat tedious to work with multiple projects. Therefore, it is recommended to use the diff --git a/docs/data/Allas/using_allas/rclone.md b/docs/data/Allas/using_allas/rclone.md index e15ad77e52..a2d905b5d6 100644 --- a/docs/data/Allas/using_allas/rclone.md +++ b/docs/data/Allas/using_allas/rclone.md @@ -33,39 +33,43 @@ The most frequently used Rclone commands: A more extensive list can be found on the [Rclone manual pages](https://rclone.org/docs/) or by typing the command `rclone` in Puhti. -## Authentication +## Authentication on CSC supercomputers -The first step is to authenticate to a project in Allas. Rclone can use both Swift and S3 protocols but these connections will have different names in rclone commands. +Below, we describe how Rclone is used in CSC computing environment (Puhti and Mahti). You can use [Rclone also in your local computer](./rclone_local.md). -In this document we describe how Rclone is used in CSC computing environment (Puhti and Mahti). You can use rclone also in your local computer. Instructions of configuring locally installed Rclone are here - * [Local Rclone configuration for Allas](./rclone_local.md) +The first step is to configure connection to a project in Allas. Rclone can use both Swift and S3 protocols but these connections will have different names in rclone commands. -### Rclone with swift on CSC supercomputers + [`allas-conf`](allas-conf.md#allas-conf-configure-connection) for more info and additional options. + +### Rclone with swift The default protocol of Allas is Swift. In Puhti and Mahti Swift based Allas connection is activated with commands: ```text module load allas allas-conf ``` -The `allas-conf` command asks for your CSC password (University/Haka password will not work here). It lists -your projects in Allas and asks you to define the project that will be used. Then _allas-conf_ generates a Rclone configuration file for the Allas service and authenticates the connection to the selected project. In Rclone command this swift based connection is referred with remote name `allas:`. The authentication information is stored in the shell variables `OS_AUTH_TOKEN` and `OS_STORAGE_URL` that are valid for up to eight hours. However, you can refresh the authentication at any time by running _allas-conf_ again. The environment variables are available only for that login session, so if you login to Puhti in another session, you need to authenticate again to access Allas. +In Rclone commands, this Swift based connection is referred with remote name `allas:`. -### Rclone with S3 on CSC supercomputers +### Rclone with S3 -If you want to use Allas with the S3 protocol instead, run the `allas-conf` command with the `--mode S3` option. +If you want to use Allas with the S3 protocol instead, run: ```text module load allas allas-conf --mode S3 ``` This command opens permanent S3 based connection to Allas. Rclone can now refer to this connection with remote name `s3allas:`. In the examples below the swift based `allas:` remote definition is used, but if you have S3 connection defined, you could replace it -with `s3allas:`. Note that you can have both `allas:` and `s3allas:` functional in the same time and that they can still use different Allas projects. However, you should avoid mixing protocols. If an object is loaded using `allas:` do also all operations with `allas:`. +with `s3allas:`. + +Note, that you can have both `allas:` and `s3allas:` functional in the same time and that they can still use different Allas projects. However, you should avoid mixing protocols. If an object is loaded using `allas:` do also all operations with `allas:`. + + ## Create buckets and upload objects -The data in Allas is arranged into containers called buckets. You can consider them as root-level directories. All buckets in Allas must have unique names – you cannot create a bucket if some other project has already used that bucket name. It is a good rule of thumb to have something project- or user-specific in the bucket name, e.g. _2000620-raw-data_. See the [checklist](../introduction.md#naming-buckets) for how to name a bucket. +The data in Allas is arranged into containers called buckets. You can consider them as root-level directories. All buckets in Allas must have unique names – you cannot create a bucket if some other project has already used that bucket name. It is a good rule of thumb to have something project- or user-specific in the bucket name, e.g. _2000620-raw-data_. See the [checklist](../introduction.md#naming-buckets-and-objects) for how to name a bucket. In the case of _Rclone_, create a bucket: ```text @@ -164,5 +168,38 @@ rclone sync allas:2000620-raw-data/mydata mydata This command returns the uploaded data from Allas to the _mydata_ directory. Note however that if you have added new data to _mydata_ after synchronizing the directory with Allas, this data will be erased. +## Copying files directly between object storages + +Rclone can also be used to directly copy files from another object storage (e.g. [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html), [Google cloud](https://cloud.google.com/learn/what-is-object-storage), [CREODIAS](https://creodias.eu/cloud/cloudferro-cloud/storage-2/object-storage/),...) to Allas. For this both credentials need to be stored in a Rclone configuration file in the users home directory (`.config/rclone/rclone.conf`). An example is shown below: + +``` +[s3allas] +type = s3 +provider = Other +env_auth = false +access_key_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +endpoint = a3s.fi +acl = private + +[otherobjectstorage] +type = s3 +provider = Other +env_auth = false +access_key_id = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy +secret_access_key = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy +endpoint = yourotherendpoint.com +acl = private +``` +The configuration for Allas is added automatically when configuring Allas in s3 mode + +`source allas_conf --mode s3cmd` . + +After creating/updating this file, Rclone can be used to copy files + +`rclone copy otherobjectstorage:bucket-x/object-y s3allas:bucket-z/object-a` + +or list files from either Allas or the other object storage by using the respective name +`rclone lsf otherobjectstorage: `. diff --git a/docs/data/Allas/using_allas/rclone_local.md b/docs/data/Allas/using_allas/rclone_local.md index fe81075ec7..81fab9da19 100644 --- a/docs/data/Allas/using_allas/rclone_local.md +++ b/docs/data/Allas/using_allas/rclone_local.md @@ -12,49 +12,16 @@ line: _Terminal_ in Mac and Linux, _Command prompt_ or _Powershell_ in Windows. ## Configuring Allas connection in Mac and Linux (Swift and S3) -If you are using Rclone in a local **Linux** or **Mac machine**, you can download -the `allas_conf` script to set up the connection to your Allas project. +First [install `allas-conf` locally](allas-conf.md#allas-conf-installation) and configure your connection. ```bash -wget https://raw.githubusercontent.com/CSCfi/allas-cli-utils/master/allas_conf +# Swift source allas_conf -u your-csc-username -p your-csc-project-name -``` - -Note that you should use the `-u` option to define your CSC username and `-p` to -define the CSC project you want to use. For example: - -```bash -source allas_conf -u kkayttaj -p project_2001234 -``` - -The command above defines a connection that uses Swift protocol and is valid in the current -terminal session for 8 hours. The Rclone _remote_ site name is in this case _allas_. After -configuration you can for example list your buckets in Allas with command: - -```bash -rclone lsd allas: -``` - -If you want to use S3 protocol instead of Swift, add option `-m S3` to the configuration command. - -```bash +# S3 source allas_conf -u kkayttaj -p project_2001234 -m S3 ``` -Note that in order to run the command above you must have _OpenStack_ client installed in your local machine. After configuring S3 connection to Allas, you can use it through Rclone remote called _s3allas:_. For example: - -```bash -rclone lsd s3allas: -``` - -S3 connection is active until you explicitly delete it with command: - -```bash -source allas_conf -u csc-user-name -p csc-project-name --s3remove -``` - -!!! Note - Remember to be careful and security-aware when configuring S3 connection to Allas. The S3 keys are stored in a readable format in your home directory and anyone who can read your keys can access Allas until the keys are expilicitly revoked from Allas. Removing the keys from your own computer is not enough to deactivate them. +In Rclone commands, this Swift based connection is referred with remote name `allas:` and S3 connection `s3allas:`. ## Configuring Allas connection in Windows diff --git a/docs/data/Allas/using_allas/s3_client.md b/docs/data/Allas/using_allas/s3_client.md index 6bd1faa3a3..33e54ed002 100644 --- a/docs/data/Allas/using_allas/s3_client.md +++ b/docs/data/Allas/using_allas/s3_client.md @@ -6,6 +6,8 @@ the _S3_ protocol that differs from the _Swift_ protocol used in the [Rclone](./ From the user perspective, one of the main differences between S3 and Swift protocols is that Swift based connections remain valid for eight hours at a time, but with S3, the connection remains permanently open. The permanent connection is practical in many ways but it has a security aspect: if your CSC account is compromised, so is the object storage space. +Use version 2.0.2 or later. + The syntax of the `s3cmd` command: ```text s3cmd -options command parameters @@ -34,7 +36,7 @@ s3cmd -h ``` ## Getting started with s3cmd -If you use Allas on Puhti or Mahti, all required packages and software are already installed. In this case you can skip this chapter and proceed to the section [Configuring S3 connection in supercomputers](#configuring-s3-connection-in-supercomputers). +If you use Allas on Puhti or Mahti, all required packages and software are already installed. In this case you can skip this chapter and proceed to the section [Configuring S3 connection in supercomputers](#configuring-s3-connection). To configure a s3cmd connection, you need to have _OpenStack_ and _s3cmd_ installed in your environment. @@ -64,40 +66,22 @@ s3cmd Please refer to [http://s3tools.org/download](http://s3tools.org/download) and [http://s3tools.org/usage](http://s3tools.org/usage) for upstream documentation. -**Configuring S3 connection on local computer** - -Once you have _OpenStack_ and _s3cmd_ installed in your environment, you can download the [allas_conf](https://raw.githubusercontent.com/CSCfi/allas-cli-utils/master/allas_conf) -script to set up the S3 connection to your Allas project. -```text -wget https://raw.githubusercontent.com/CSCfi/allas-cli-utils/master/allas_conf -source allas_conf --mode S3 --user your-csc-username -``` -Note that you should use the `--user` option to define your CSC username. The configuration command first asks for your -CSC password and then for you to choose an Allas project. After that, the tool creates a key file for the S3 connection and stores it in the default location (_.s3cfg_ in home directory). +Additionally install [`allas-conf`](allas-conf.md#allas-conf-installation). -## Configuring S3 connection in supercomputers +## Configuring S3 connection To use _s3cmd_ in Puhti and Mahti, you must first configure the connection: ```text module load allas allas-conf --mode S3 -``` -The configuration process first asks for your CSC password. Then it lists your Allas projects and asks to select the project to be used. The configuration information is stored in the file _$HOME/.s3cfg_. This configuration only needs to be defined once. In the future, _s3cmd_ will automatically use the object storage connection described in the _.s3cfg_ file. If you wish to change the Allas project that _s3cmd_ uses, you need to run the configuration command again. - -You can use the S3 credentials, stored in the _.s3cfg_ file, in other services too. You can check the currently used _access key_ and _secret_key_ with command: -``` -grep key $HOME/.s3cfg - ``` -If you use these keys in other services, your should make sure that the keys always remain private. Any person who has access to these two keys, can access and modify all the data that the project has in Allas. - -In needed, you can deactivate an S3 key pair with command: - -``` -allas-conf --s3remove +On local computer: +```text +source allas_conf --mode S3 --user your-csc-username ``` +See [`allas-conf`](allas-conf.md) for more info and additional options. ## Create buckets and upload objects @@ -119,6 +103,7 @@ List all buckets in a project: s3cmd ls ``` +List all objects in a bucket:du List all objects in a bucket: ```text s3cmd ls s3://my_bucket @@ -134,6 +119,11 @@ Display information about an object: s3cmd info s3://my_bucket/my_file ``` +Check your project's object storage usage: +```text +s3cmd du -H +``` + ## Download objects and buckets Download an object: diff --git a/docs/data/Allas/using_allas/swift_client.md b/docs/data/Allas/using_allas/swift_client.md index 6df52652ab..b0d716efb1 100644 --- a/docs/data/Allas/using_allas/swift_client.md +++ b/docs/data/Allas/using_allas/swift_client.md @@ -10,7 +10,7 @@ Open a connection to Allas: allas-conf ``` -The _allas-conf_ command above asks for your CSC password (the same that you use to login to CSC servers). It lists your projects in Allas and asks you to define the project that will be used. _allas-conf_ generates and authenticates the connection to the selected project in Allas. The authentication information is stored in the shell variables *OS_AUTH_TOKEN* and *OS_STORAGE_URL* that are valid for up to eight hours. However, you can refresh the authentication at any time by running _allas-conf_ again. The environment variables are available only for that login session. If you open another session, you need to authenticate again to access Allas. +See [`allas-conf`](allas-conf.md) for more info and additional options. This chapter includes instructions for the following operations: diff --git a/docs/data/Allas/using_allas/web_client.md b/docs/data/Allas/using_allas/web_client.md index 59f641ff83..192389a72b 100644 --- a/docs/data/Allas/using_allas/web_client.md +++ b/docs/data/Allas/using_allas/web_client.md @@ -1,8 +1,6 @@ -# The web client – OpenStack Horizon Dashboard +# Pouta web interface -This chapter provides instructions for using Allas with the user-friendly _OpenStack Horizon Dashboard_. - -OpenStack documentation for managing buckets over the web interface: [https://docs.openstack.org/horizon/latest/user/manage-containers.html](https://docs.openstack.org/horizon/latest/user/manage-containers.html) +This chapter provides instructions for using Allas with Pouta web interface OpenStack dashboard. The OpenStack dashboard has a small subset of object storage functionalities. The available operations: @@ -15,6 +13,8 @@ The OpenStack dashboard has a small subset of object storage functionalities. Th | _Remove_ objects and buckets | | Make buckets _public_ or _private_ | +[OpenStack documentation for managing buckets over the web interface](https://docs.openstack.org/horizon/latest/user/manage-containers.html) + ### **Alternative Web Interface – Allas Web UI** For a more intuitive and feature-rich web interface, you can also use **[Allas Web UI](./allas-ui.md)**. @@ -30,7 +30,7 @@ Allas Web UI provides a graphical way to manage object storage, create and delet !["Creating a container"](img/allas_screenshot_create_container.png) **Figure** Creating a container -3\. Press the **+Container** button and name the bucket (see the [checklist for naming a bucket](../introduction.md#naming-buckets)). If you choose to make the bucket _Public_, the contents of the bucket can be [viewed via the internet](#view-objects-via-the-internet). +3\. Press the **+Container** button and name the bucket (see the [checklist for naming a bucket](../introduction.md#naming-buckets-and-objects)). If you choose to make the bucket _Public_, the contents of the bucket can be [viewed via the internet](#view-objects-via-the-internet). ## Upload an object diff --git a/docs/data/moving/copy_allas_ida.md b/docs/data/moving/copy_allas_ida.md index 4de1e77aad..829917e9d7 100644 --- a/docs/data/moving/copy_allas_ida.md +++ b/docs/data/moving/copy_allas_ida.md @@ -46,7 +46,7 @@ Download the data from Allas to that new directory. You should use the same protocol as was used to originally upload the data to Allas. If the data was uploaded with command-line tools, preferably also use the same command-line tool. More information about the Allas tools on Puhti is available at -[Accessing Allas in the CSC computing environment and other Linux platforms](../Allas/accessing_allas.md#accessing-allas-in-the-csc-computing-environment-and-other-linux-platforms). +[Accessing Allas in the CSC computing environment and other Linux platforms](../Allas/accessing_allas.md). In our example case the data was originally uploaded to Allas with a-commands, so the user uses `a-get` to download the data: @@ -217,7 +217,7 @@ a-put ``` More information about the Allas tools on Puhti is available at -[Accessing Allas in the CSC computing environment and other Linux platforms](../Allas/accessing_allas.md#accessing-allas-in-the-csc-computing-environment-and-other-linux-platforms). +[Accessing Allas in the CSC computing environment and other Linux platforms](../Allas/accessing_allas.md). Continuing our example, assuming the unzipped data to be uploaded to Allas is in a directory `experiment_data`, it can be uploaded with `a-put` as: diff --git a/docs/support/faq/how-to-get-Allas-s3-credentials.md b/docs/support/faq/how-to-get-Allas-s3-credentials.md index c506c7cecb..f3ac0b0e4f 100644 --- a/docs/support/faq/how-to-get-Allas-s3-credentials.md +++ b/docs/support/faq/how-to-get-Allas-s3-credentials.md @@ -1,6 +1,14 @@ -# How to get Allas S3 credentials +# How to get Allas S3 credentials? -It is possible to use the openstack API to get Allas S3 credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY). Each credential pair is linked to a single OpenStack project. +How to get Allas S3 credentials: access key and secret key? Each credential pair is linked to a single CSC project. + +## 'allas-conf' + +Use [`allas-conf`](../../data/Allas/using_allas/allas-conf.md#s3-connection-details) to find out your S3 credentials. +It is pre-installed to CSC supercomputers or can be installed to Linux and Mac. + + +## openstack API 1. Install [OpenStack command line tools](../../cloud/pouta/install-client.md) diff --git a/docs/support/faq/how-to-move-data-between-puhti-and-allas.md b/docs/support/faq/how-to-move-data-between-puhti-and-allas.md index 33295560be..fc6d64020b 100644 --- a/docs/support/faq/how-to-move-data-between-puhti-and-allas.md +++ b/docs/support/faq/how-to-move-data-between-puhti-and-allas.md @@ -1,88 +1,4 @@ -# How to move data from Puhti to Allas and vice versa? +# How to move data from Puhti/Mahti to Allas and vice versa? -There are several options for moving data between Puhti and Allas. This page -summarizes how to move data using _a-commands_, _Swift_, _Rclone_ and _S3cmd_ -clients. +See [Tutorial for using Allas in CSC supercomputers](../../data/Allas/allas-hpc.md) -All the required packages and software needed for the clients are already -installed on Puhti. To activate Allas and authenticate to a project you need to -run the commands: - -```bash -module load allas -allas-conf -``` - -## Move data with a-commands - -[a-commands](../../data/Allas/using_allas/a_commands.md) are easy-to-use tools -for basic usage of Allas. The main commands to use for moving data between -Puhti and Allas are: - -* [`a-put`](../../data/Allas/using_allas/a_commands.md#a-put-uploads-data-to-allas): - Move data from Puhti to Allas -* [`a-get`](../../data/Allas/using_allas/a_commands.md#a-get-retrieves-stored-data): - Move data from Allas to Puhti - -## Move data with Swift - -The Swift client provides the options `upload` and `download` for moving data: - -```bash -swift upload -``` - -```bash -swift download -``` - -For more information, see -[Swift client](../../data/Allas/using_allas/swift_client.md). - -## Move data with Rclone - -Rclone is another client using which you can move data between Puhti and -Allas. For example, you could create a bucket called `2000620-raw-data` in -Allas using the command: - -```bash -rclone mkdir allas:2000620-raw-data -``` - -Uploading a file called `file.dat` to that bucket can be done using the -`rclone copy` command: - -```bash -rclone copy file.dat allas:2000620-raw-data/ -``` - -Downloading the file back to Puhti is done with the same `rclone copy` command: - -```bash -rclone copy allas:2000620-raw-data/file.dat . -``` - -!!! info "Note" - Another destination directory can be specified in the `rclone copy` command - as well. If this directory does not exist, Rclone will create it for you. - - ```bash - rclone copy allas:2000620-raw-data/file.dat my-new-folder - ``` - -For more information, see -[Using Allas with Rclone from Puhti](../../data/Allas/using_allas/rclone.md). - -## Move data with S3cmd - -For moving data between Puhti and Allas, -[S3cmd](../../data/Allas/using_allas/s3_client.md) provides the functions: - -* [`s3cmd put`](../../data/Allas/using_allas/s3_client.md#create-buckets-and-upload-objects): - Move data from Puhti to Allas - -* [`s3cmd get`](../../data/Allas/using_allas/s3_client.md#download-objects-and-buckets): - Move data from Allas to Puhti - -For more information, see -[S3 client](../../data/Allas/using_allas/s3_client.md). diff --git a/docs/support/tutorials/clean-up-data.md b/docs/support/tutorials/clean-up-data.md index dc8998f54c..3dd5ba4db1 100644 --- a/docs/support/tutorials/clean-up-data.md +++ b/docs/support/tutorials/clean-up-data.md @@ -29,7 +29,7 @@ Please do the following tasks: medium sized data transfers, in particular when you have a large amount of small files. These tools make the usage of Allas safer, and can make your data management easier. For very large data transfers we recommend using [rclone](../../data/Allas/using_allas/rclone.md). - A tutorial for data transfer is available at [allas-examples](../../data/Allas/allas-examples.md). + A tutorial for data transfer is available at [Allas HPC tutorial](../../data/Allas/allas-hpc.md). * **Archive files** that should be available longer than the lifetime of compute projects. Options for this can be for example your organizations own storage systems, or [IDA safe storage for research data](https://www.fairdata.fi/en/). diff --git a/docs/support/tutorials/gis/gdal_cloud.md b/docs/support/tutorials/gis/gdal_cloud.md index 1a2db5a017..59e1ebc673 100644 --- a/docs/support/tutorials/gis/gdal_cloud.md +++ b/docs/support/tutorials/gis/gdal_cloud.md @@ -69,7 +69,7 @@ For example to set Allas end-point: #### S3 connection set up for Allas The easiest way to set up S3 configuration for `GDAL` is by -[configuring an S3 connection on a CSC supercomputer](../../../data/Allas/using_allas/python_boto3.md#configuring-s3). +[configuring an S3 connection on a CSC supercomputer](../../../data/Allas/using_allas/allas-conf.md#s3-connection). GDAL requires additionally that Allas endpoint is given as AWS_S3_ENDPOINT environment variable, see the commands above. diff --git a/docs/support/tutorials/index.md b/docs/support/tutorials/index.md index 63cdd8c6df..b130d751e2 100644 --- a/docs/support/tutorials/index.md +++ b/docs/support/tutorials/index.md @@ -36,7 +36,7 @@ ## Allas -* [Using Allas interactively in Puhti and Mahti](../../data/Allas/allas-examples.md) +* [Using Allas interactively in Puhti and Mahti](../../data/Allas/allas-hpc.md) * [Using Allas in batch jobs](../../data/Allas/allas_batchjobs.md) * [Using Allas to host a dataset for a research project](../../data/Allas/allas_project_example.md) * [Tools for client side encryption for Allas](../../data/Allas/allas_encryption.md) diff --git a/mkdocs.yml b/mkdocs.yml index 8388517d9d..1eb44d20d0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -277,7 +277,6 @@ nav: - Supercomputer storage: - computing/disk.md - Lustre filesystem: computing/lustre.md - - Allas object storage: computing/allas.md - Module environment: computing/modules.md - Running jobs: - computing/running/getting-started.md @@ -502,17 +501,19 @@ nav: - Common Error Messages: data/Allas/using_allas/error_messages.md - Allas object storage related terms and concepts: data/Allas/terms_and_concepts.md - Allas clients: - - Allas Web UI: data/Allas/using_allas/allas-ui.md + - Allas connection configuration: data/Allas/using_allas/allas-conf.md + - Allas web UI: data/Allas/using_allas/allas-ui.md + - Puhti/Mahti web UI: computing/webinterface/file-browser.md - a-command: data/Allas/using_allas/a_commands.md - a-backup: data/Allas/using_allas/a_backup.md - Cyberduck: data/Allas/using_allas/cyberduck.md - Python with S3: data/Allas/using_allas/python_boto3.md - - Python with SWIFT: data/Allas/using_allas/python_swift.md + - Python with Swift: data/Allas/using_allas/python_swift.md - Rclone from Computing environment: data/Allas/using_allas/rclone.md - Rclone form Workstation: data/Allas/using_allas/rclone_local.md - Swift: data/Allas/using_allas/swift_client.md - S3cmd: data/Allas/using_allas/s3_client.md - - Web GUI: data/Allas/using_allas/web_client.md + - Pouta web UI: data/Allas/using_allas/web_client.md # data/Allas/using_allas/directory_object_error.md is not linked in this structure - Kaivos database service: - data/kaivos/overview.md