Update Including the Toolbox in a pipeline authored by Grailet Jean-François's avatar Grailet Jean-François
--- ---
title: Including the Toolbox in a pipeline title: Including the Toolbox in a pipeline
--- ---
You can use the [Toolbox](https://gitlab.uliege.be/especes/mast/toolbox) to assess model outputs at the end of a CI/CD pipeline. [An example of this approach is available on the Nemo4.2.0-Bamhbi repository.](https://gitlab.uliege.be/especes/mast/nemo4.2.0-bamhbi/-/blob/main/test_cases/lr_cluster/scripts/lr_nic5_assess.sh) To use the Toolbox, you will need to use a suitable environment, featuring Python and the relevant packages, and to clone the [Toolbox repository](https://gitlab.uliege.be/especes/mast/toolbox) within the environment. You can use the [Toolbox](https://gitlab.uliege.be/especes/mast/toolbox) to assess model outputs at the end of a CI/CD pipeline. [An example of this approach is available on the Nemo4.2.0-Bamhbi repository.](https://gitlab.uliege.be/especes/mast/nemo4.2.0-bamhbi/-/blob/main/test_cases/lr_cluster/scripts/lr_nic5_assess.sh) To use the Toolbox, you will need to use a suitable environment, featuring Python and the relevant packages, and to clone the [Toolbox repository](https://gitlab.uliege.be/especes/mast/toolbox) within the environment.
## Setting up a CI/CD job using the Toolbox ## Setting up a CI/CD job using the Toolbox
...@@ -19,7 +20,7 @@ Once you have a suitable environment, the main technical hurdle in using the [To ...@@ -19,7 +20,7 @@ Once you have a suitable environment, the main technical hurdle in using the [To
However, since this repository is not public (for the time being), you need to use an _access token_ to be able to clone it with ``git clone``. An access token is a long alphanumeric sequence meant to act as a means of identification to the repository, which only select people have. For security reasons, an access token should never be stored _as is_ in the repository, as anyone having access to the project files (one way or another) may abuse it to browse a repository they should not have access to. However, since this repository is not public (for the time being), you need to use an _access token_ to be able to clone it with ``git clone``. An access token is a long alphanumeric sequence meant to act as a means of identification to the repository, which only select people have. For security reasons, an access token should never be stored _as is_ in the repository, as anyone having access to the project files (one way or another) may abuse it to browse a repository they should not have access to.
Hopefully, GitLab offers the possibility to store _secret files_, files that cannot be read while browsing the Web interface of GitLab, but which can be copied during a CI/CD job. Such files are listed in **Settings > CI/CD > Secure files** on a repository. To keep this tutorial simple, we will assume a ``Token_Toolbox.txt`` file is already listed in **Settings > CI/CD > Secure files** on a repository (it's the case for the [Nemo4.2.0-Bamhbi repository](https://gitlab.uliege.be/especes/mast/nemo4.2.0-bamhbi)). Go to the Appendix [Creating and securing an access token](pipeline-toolbox#appendix-creating-and-securing-an-access-token) if you need to create and/or secure an access token to clone the Toolbox from a repository where a token is not available yet. Hopefully, GitLab offers the possibility to store _secret files_, files that cannot be read while browsing the Web interface of GitLab, but which can be copied during a CI/CD job. Such files are listed in **Settings > CI/CD > Secure files** on a repository. To keep this tutorial simple, we will assume a ``Token_Toolbox.txt`` file is already listed in **Settings > CI/CD > Secure files** on a repository (as it's the case on the [Nemo4.2.0-Bamhbi repository](https://gitlab.uliege.be/especes/mast/nemo4.2.0-bamhbi)). Go to the Appendix [Creating and securing an access token](pipeline-toolbox#appendix-creating-and-securing-an-access-token) if you need to create and/or secure an access token to clone the Toolbox from a repository where a token is not available yet.
Once a secure file is part of a repository, such as ``Token_Toolbox.txt`` on the [Nemo4.2.0-Bamhbi repository](https://gitlab.uliege.be/especes/mast/nemo4.2.0-bamhbi), any CI/CD job for this repository may use it by copying it at the very start of the ``script:`` part of a job, using the instruction at the end of the next CI/CD job excerpt. Once a secure file is part of a repository, such as ``Token_Toolbox.txt`` on the [Nemo4.2.0-Bamhbi repository](https://gitlab.uliege.be/especes/mast/nemo4.2.0-bamhbi), any CI/CD job for this repository may use it by copying it at the very start of the ``script:`` part of a job, using the instruction at the end of the next CI/CD job excerpt.
...@@ -48,3 +49,26 @@ Very similar instructions can be found in [this file from the Nemo4.2.0-Bamhbi r ...@@ -48,3 +49,26 @@ Very similar instructions can be found in [this file from the Nemo4.2.0-Bamhbi r
> **Important remark:** as noted in the [GitLab documentation](https://docs.gitlab.com/ci/secure_files/#with-the-download-secure-files-tool), the secure files are no longer hidden when inside a CI/CD job. In other words, it's possible to print their content using commands within the job, such as ``cat`` (``cat Token_Toolbox.txt`` will print the access token in the standard output), whose outputs will later appear in the job log. **Make sure to never use such commands to keep the files secure, and to silence any command that may inadvertently display the token.** > **Important remark:** as noted in the [GitLab documentation](https://docs.gitlab.com/ci/secure_files/#with-the-download-secure-files-tool), the secure files are no longer hidden when inside a CI/CD job. In other words, it's possible to print their content using commands within the job, such as ``cat`` (``cat Token_Toolbox.txt`` will print the access token in the standard output), whose outputs will later appear in the job log. **Make sure to never use such commands to keep the files secure, and to silence any command that may inadvertently display the token.**
## Appendix: creating and securing an access token ## Appendix: creating and securing an access token
We now review how you can create and/or secure an access token to be able to clone any (private) repository you have access to within a CI/CD job. To keep things simple, we the repository to clone will be hereby named _target repository_, and the repository whose CI/CD jobs must clone such target repository will be referred to as _host repository_.
### Creating an access token (target repository)
In the target repository, go to **Settings > Access tokens**. You should see a table with a button **Add new token** at its top right. Click on it and fill the subsequent form.
* _Token name_ and _Token description_ are purely informative fields which will have no practical effect on how to use the token. You may fill them to advertise your colleagues about their purpose.
* _Expiration date_ gives, by default, today's date plus one month. You can tune the expiration date at your will, but you can also do not give one. In such a case, the token will expire next year from now.
* _Select a role_ enables you to define the role associated to the token, which may restrict (or not) the actions you can carry out with the token. The _Developer_ role is more than enough for a token to be used for cloning the target repository.
* Finally, in _Select a scope_, you may define more precisely what scopes of actions can be performed with the token. If you only need to be able to clone the target repository, it's enough to check ``read_repository``.
Once the form is filled, click on the blue button at the end of the form. You will then see the generated token on top of a green background, as follows. Use the buttons (eye and clipboard) on the right side to either show or copy the token, which you will never be able to see again after.
> <img src="uploads/be174f8df292be5e7cfe1157c9ba5b33/access_token.png" alt="access_token" width="500"/>
Copy the token and save it in a text file on your computer.
### Securing the token (host repository)
Now, on the host repository, go to **Settings > CI/CD > Secure files**, click on **Upload file** and select, on your computer, the file you created at the end of the previous section. That's it !
Your secure file should not be listed under **Settings > CI/CD > Secure files**. It goes without saying that you will have to use its name during your CI/CD job (cf. the first part of this tutorial) to be able to take advantage of its content, i.e., the access token.
\ No newline at end of file