Changes for page Using JupyterHub in HDC

Last modified by Dennis Segebarth on 2024/10/02 18:14

From 2.1 to 1.7

From version 1.7

edited by Susan Evans
on 2023/07/11 14:05

Change comment: (Autosaved)

To version 1.1

edited by Susan Evans
on 2023/07/11 13:35

Change comment: (Autosaved)

Raw
Rendered

Summary

Page properties (1 modified, 0 added, 0 removed)
Attachments (0 modified, 0 added, 5 removed)

Details

Page properties

Content

@@ -180,218 +180,8 @@
  Afterwards, the environment will be listed when you open the Launcher to open a new Jupyter Notebook:
--[[image:Project Workspace Jupyter view new Kernel 2023-07-11.png||height="25%" width="50%"]]
--and also from each opened Notebook, e.g., via **Kernel > Change Kernel…** :
--[[image:Project Workspace Jupyter Kernel change Kernel dropdown 2023-07-11.png||height="64%" width="50%"]]
--= Installing New Python Packages =
--
--We highly recommend the use of virtual environments when installing new packages (see //Creating a Python Virtual Environment and Registering a Kernel// above for more details). Consequently, we recommend installing new packages via commands in the JupyterHub terminal in the corresponding virtual environments, instead of installing packages from within Jupyter Notebooks.
--
--Depending on the IT policies, outbound traffic may need to go through a proxy. If so, users will be required to provide the proxy command line argument such as pip, curl, wget, etc.
--
--For example:
--
--{{code language="none"}}
--pip install my_package
--{{/code}}
--
--If you are using conda to manage python packages:
--
--{{code language="none"}}
--conda install my_package
--{{/code}}
--
--The above information is provided as examples only. Please refer to documentation provided by your IT department with respect to proxy configuration.
--
--= Using the Pilot Command Line Interface in a JupyterHub Terminal =
--
--The Pilot Command Line Interface (CLI) is deployed within JupyterHub as extension resource. Project members can use the Pilot Command Line Interface in a JupyterHub terminal to download Project data from the Core for further analysis, and upload the derivative outputs back to the Green Room or Core.
--
--The Home Directory is your default directory. When you download a copy of your Core files to JupyterHub, the files persists in the JupyterHub environment until deleted by you, so you can return to the session and continue your work at a later time without the need to retrieve the data from the Core again.
--
--The following sections focus on getting started with basic pilotcli commands in JupyterHub. For additional pilotcli commands and usage, see the article //Working with HDC Project Files in the Command Line Interface//.
--
--== Launching Pilot Command Line Interface ==
--
--1. Launch your Project and click the **JupyterHub** icon in in the workspace icon group.
--1. Click the **Terminal **launcher icon to open the Terminal.
--1. In the Jupyterhub Terminal, type {{code}}pilotcli{{/code}} to launch the latest version of the Pilot Command Line Interface.
--1. Use the {{code}}pilotcli --help{{/code}} at any time to show the welcome message again.
--
--{{code language="none"}}
--collaborator4@jupyter-collaborator4:~$ pilotcli
--Usage: pilotcli [OPTIONS] COMMAND [ARGS]...
--
--  What's new (Version 2.2.0):
--
--   1. CLI supports to perform multi-threading upload for file/folders
--
--   2. CLI supports to perform resumable upload for single file
--
--
--
--Options:
--  --help  Show this message and exit.
--
--Commands:
--  container_registry  Container Registry Actions.
--  dataset             Dataset Actions.
--  file                File Actions.
--  project             Project Actions.
--  use_config          Config Actions.
--  user                User Actions.
--{{/code}}
--
--== Logging into the Pilot Command Line Interface ==
--
--Users are required to login with platform credentials before performing any tasks through Pilot Command Line Interface.
--
--* Use the command {{code}}pilotcli user login{{/code}} to log into the Pilot Command Line Interface.
--
--{{code language="none"}}
--collaborator4@jupyter-collaborator4:~$ pilotcli user login
--Please, access https://iam.staging.pilot.indocresearch.com/realms/pilot/device?user_code=XXXX-XXXX to proceed
-- ▄▄▄▄▄▄▄  ▄ ▄▄     ▄ ▄▄▄▄    ▄ ▄▄▄▄▄▄▄
-- █ ▄▄▄ █ ▄ ▄███   ▀▀  █▀  ▀██▄ █ ▄▄▄ █
-- █ ▄ ▀ ▄ ▀▄ ▀▀ ▄█▀▄▀ ▀▀▄█▄▄▀ █████▄▄▀▄
-- ▄▄▄▄▄▄▄ ▀ ▀█▄ ▀▄ ██▀█ ▄▀▄▄  █ ▄ █▀▄▄▄
-- █ ▄▄▄ █ █▀█▄▀ █▀   █▀▀█ ▀▄█▄█▄▄▄█▀▄█
-- █ ███ █ █▀██▀▄ █▀▄▄▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
-- █▄▄▄▄▄█ ▄▀▄▄██▄▄▀▄  ▀▀▄  ▄▄▀▀▀▄ █▄▄▄█
-- Waiting validation finish...
--{{/code}}
--
--* (((
--You’ll be asked to validate your HDC user account using one of the provided methods.
--
--* Copy and paste the provided validation link into a new browser tab or
--* Scan the QR code with your mobile device.
--)))
--* Open the login window and enter your HDC username and password (i.e. your EBRAINS account credentials).
--* Grant access by clicking **Yes**.
--
--[[image:Pilotcli Jupyter user login Grant Access window v2.4.0 2023-05-25.png||height="46%" width="50%"]]
--
--[[image:Pilotcli Jupyter user login Device Login Successful v2.4.0 2023-05-25.png||height="16%" width="50%"]]
--
--* After successful confirmation, return to the terminal in your JupyterHub browser tab.
--
--{{code language="none"}}
--Welcome to the Command Line Tool!
--{{/code}}
--
--* You’re now ready to start using the Pilot Command Line Interface to work with your Project data in JupyterHub.
--
--== Zone Restrictions when using Pilot Command Line Interface in JupyterHub ==
--
--When using the Pilot Command Line Interface in JupyterHub and the following actions are possible on the derivative files generated in JupyterHub:
--
--|=(% colspan="1" rowspan="1" %)(((
--**File Operation**
--)))|=(% colspan="1" rowspan="1" %)(((
--**Permitted in the **
--**Green Room**
--)))|=(% colspan="1" rowspan="1" %)(((
--**Permitted in the **
--**Core**
--)))
--|(% colspan="1" rowspan="1" %)File upload
--(upload derivative output files from JupyterHub to the Green Room or Core storage)|(% colspan="1" rowspan="1" %)(((
--Yes
--)))|(% colspan="1" rowspan="1" %)(((
--Yes
--)))
--|(% colspan="1" rowspan="1" %)File download
--(download files from Green Room or Core into JupyterHub)|(% colspan="1" rowspan="1" %)(((
--**No**
--)))|(% colspan="1" rowspan="1" %)(((
--Yes
--)))
--
--== Downloading Project Data to JupyterHub using the Pilot Command Line Interface ==
--
--After logging into the Pilot Command Line Interface, you can download data from the Project Core into the JupyterHub environment to start your data analyses.
--
--File related commands are grouped in the {{code}}file{{/code}} category. To view the full list of commands in this category, type {{code}}pilotcli file --help{{/code}}. To download project data, use the file sync command. To view the full list of commands in this category, type {{code}}pilotcli file sync --help{{/code}}.
--
--
--{{code language="none"}}
--collaborator4@jupyter-collaborator4:~$ pilotcli file sync --help
--Usage: pilotcli file sync [OPTIONS] [PATHS]... OUTPUT_PATH
--
--  Download files/folders from a given Project/folder/file in core zone.
--
--Options:
--  -z, --zone TEXT  Target Zone (i.e., core/greenroom)
--  --zip            Download files as a zip.
--  -i, --geid       Enable downloading by geid.
--  --help           Show this message and exit.
--{{/code}}
--
--=== Example ===
--
--Downloading a file from the Core to your Home Directory:
--
--Reminder: Please follow Linux conventions for file management. If your filename contains spaces, wrap it in single or double quotes.
--
--* //Filename~:// “Chemical Tracking Data.csv”
--* //Source~:// Project “Indoc Test Project”, “Core” storage zone, folder “collaborator4” {{code}}indoctestproject/collaborator4/Chemical Tracking Data.csv -z core{{/code}}
--* //Destination: //user's Home directory in the Guacamole or JupyterHub VM {{code}}.{{/code}}
--* //Command group/option: //{{code}}file sync{{/code}}
--
--{{code language="none"}}
--collaborator4@jupyter-collaborator4:~$ pilotcli file sync indoctestproject/collaborator4/'Chemical Tracking Data.csv' . -z core
--start downloading...
--Downloading Chemical Tracking Data.csv |██████████████████████████████ 100% 00:00
--File has been downloaded successfully and saved to: ./Chemical Tracking Data.csv
--{{/code}}
--
--To confirm successful download, type {{code}}ls{{/code}} and verify the file "Chemical Tracking Data.csv" is stored in the Home folder.
--
--{{code language="none"}}
--collaborator4@jupyter-collaborator4:~$ ls
--'Chemical Tracking Data.csv'   pilotcli
--{{/code}}
--
--The file “Chemical Tracking Data.csv” can be viewed in the JupyterHub graphical user interface:
--
--[[image:Jupyter downloaded file in Home folder v2.4.11 2023-05-25 1850.png||height="15%" width="50%"]]
--
--
--== Uploading Project Data from JupyterHub using the Pilot Command Line Interface ==
--
--After analyzing Project data inside the JupyterHub, you can upload the generated outputs back into the Project via the Pilot Command Line Interface.
--
--=== Example ===
--
--* //Filename//: Chemical Tracking Data rev.csv
--* //Source~:// user's Home directory in JupyterHub {{code}}.{{/code}}
--* //Destination//: Project “Indoc Test Project”, folder “collaborator4”, “Core” storage zone,
--{{code}}indoctestproject/collaborator4{{/code}} {{code}}-z core{{/code}}
--* //Command group/option~:// {{code}}file upload{{code}}
--* //User message// (for upload back to the Core): “my workbench output, no additional sensitive data"
--* //Command~:// pilotcli file upload ./'Chemical Tracking Data rev.csv' -p indoctestproject/collaborator4 -z core -m "my workbench output, no additional sensitive data"
--
--When uploading data to the Core, you are reminded that you are bypassing the usual Green Room upload workflow. To confirm, type y at the prompt, or N to cancel.
--
--{{code language="none"}}
--collaborator4@jupyter-collaborator4:~$ pilotcli file upload ./'Chemical Tracking Data rev.csv' -p indoctestproject/collaborator4 -z core -m "my workbench output, no additional sensitive data"
--You are about to transfer data directly to the PILOT Core! In accordance with the PILOT Terms of Use, please confirm that you have made your best efforts to
--pseudonymize or anonymize the data and that you have the legal authority to transfer and make this data available for dissemination and use within the PILOT .If you
--need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to perform these actions.
--To cancel this transfer, enter [n/No]
--To confirm and proceed with the data transfer, enter [y/Yes]
-- [y/N]: y
--Starting upload of: ./Chemical Tracking Data rev.csv
--Pre-upload complete.
--Uploading Chemical Tracking Data rev.csv:  |██████████████████████████████ 100% 00:00
--Upload Time: 2.92s for 1 files
--All uploading jobs have finished.
--{{/code}}
--
--

Jupyter downloaded file in Home folder v2.4.11 2023-05-25 1850.png

Author

...	...	@@ -1,1 +1,0 @@
1		-XWiki.sgevans

Size

...	...	@@ -1,1 +1,0 @@
1		-35.0 KB

Content

Pilotcli Jupyter user login Device Login Successful v2.4.0 2023-05-25.png

Author

...	...	@@ -1,1 +1,0 @@
1		-XWiki.sgevans

Size

...	...	@@ -1,1 +1,0 @@
1		-10.3 KB

Content

Pilotcli Jupyter user login Grant Access window v2.4.0 2023-05-25.png

Author

...	...	@@ -1,1 +1,0 @@
1		-XWiki.sgevans

Size

...	...	@@ -1,1 +1,0 @@
1		-13.9 KB

Content

Project Workspace Jupyter Kernel change Kernel dropdown 2023-07-11.png

Author

...	...	@@ -1,1 +1,0 @@
1		-XWiki.sgevans

Size

...	...	@@ -1,1 +1,0 @@
1		-38.9 KB

Content

Project Workspace Jupyter view new Kernel 2023-07-11.png

Author

...	...	@@ -1,1 +1,0 @@
1		-XWiki.sgevans

Size

...	...	@@ -1,1 +1,0 @@
1		-490.0 KB

Content