Changes for page Using JupyterHub in HDC

Last modified by Dennis Segebarth on 2024/10/02 18:14

From version 1.1

edited by Susan Evans
on 2023/07/11 13:35

Change comment: (Autosaved)

To version 19.1

edited by Dennis Segebarth
on 2024/10/02 18:14

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (2 modified, 0 added, 0 removed)
Attachments (0 modified, 10 added, 3 removed)

Details

Page properties

Author

@@ -1,1 +1,1 @@
--XWiki.sgevans
++XWiki.dsegebarth

Content

@@ -1,5 +1,13 @@
++{{box cssClass="floatinginfobox" title="Table of Contents"}}
++{{toc depth="2"/}}
++{{/box}}
++
++
  JupyterHub is an open-source, multi-user version of Jupyter Notebook for performing analysis of Project files in the Core. More information can be found in the application documentation [[https:~~/~~/jupyter.org/>>https://jupyter.org/]].
++[[image:1723798240534-407.png||height="189" width="291"]]
++
++
  = How it Works =
  JupyterHub allows Project members to create or import Jupyter Notebooks into the Project Workspace environment, retrieve Project files from the Core, perform computational workflows on the data, and write the outputs back to the Core where they can be accessed by other Project members. JupyterHub spins up a new JupyterLab instance for each Project member.
@@ -17,9 +17,13 @@
  JupyterHub is configured at the time of Project Setup. If you launch JupyterHub and receive a notice that it hasn’t been deployed for your project, please contact your Platform Administrator.
++{{info}}
++If you access JupyterHub of the HealthDataCloud Test Project, please be aware that the resources are limited for each user to: 2 GB of persistent storage volume, 4 GB memory, and a single CPU. These limitations can easily be adjusted for new Projects.
++{{/info}}
++
  = Launching JupyterHub =
--[[image:HDC Project Workspace tool navigation Jupyterhub v1.0.0 2023-05-25.png||height="10%" width="30%"]]
++[[image:1723798257792-201.png||height="121" width="349"]]
 . Launch your Project and click the **JupyterHub icon** in the left menu bar.
 . Click **Sign in with Keycloak** to initiate your session. JupyterHub automatically authenticates with your existing username and password and launches your session - no additional sign-in is required.
@@ -34,7 +34,7 @@
 . In the Launcher, click the **Python 3 Notebook **icon, or click **File > New > Notebook**.
 . Create your Notebook.
--[[image:Project Workspace Jupyter Create Python Notebook v2.1.6 2023-02-07.png||height="22%" width="50%"]]
++[[image:1723798278604-114.png||height="376" width="865"]]
  = Launching the Terminal =
@@ -43,7 +43,7 @@
 . In the Launcher, click the **Terminal **icon, or click **File > New > Terminal**.
 . The terminal window opens.
--[[image:Project Workspace Jupyter Launch Terminal v2.1.6 2023-02-07.png||height="9%" width="50%"]]
++[[image:1723798293872-992.png||height="162" width="863"]]
  Ubuntu is used to host Jupyter Notebook. Use the command cat /etc/os-release to determine to current version of Ubuntu:
@@ -180,8 +180,231 @@
  Afterwards, the environment will be listed when you open the Launcher to open a new Jupyter Notebook:
++[[image:1723798325144-485.png||height="436" width="867"]]
++and also from each opened Notebook, e.g., via **Kernel > Change Kernel…** :
++[[image:1723798338447-469.png||height="317" width="247"]]
++= Installing New Python Packages =
++
++We highly recommend the use of virtual environments when installing new packages (see //Creating a Python Virtual Environment and Registering a Kernel// above for more details). Consequently, we recommend installing new packages via commands in the JupyterHub terminal in the corresponding virtual environments, instead of installing packages from within Jupyter Notebooks.
++
++Depending on the IT policies, outbound traffic may need to go through a proxy. If so, users will be required to provide the proxy command line argument such as pip, curl, wget, etc.
++
++For example:
++
++{{code language="none"}}
++pip install my_package
++{{/code}}
++
++If you are using conda to manage python packages:
++
++{{code language="none"}}
++conda install my_package
++{{/code}}
++
++The above information is provided as examples only. Please refer to documentation provided by your IT department with respect to proxy configuration.
++
++= Using the Pilot Command Line Interface in a JupyterHub Terminal =
++
++The Pilot Command Line Interface (CLI) is deployed within JupyterHub as extension resource. Project members can use the Pilot Command Line Interface in a JupyterHub terminal to download Project data from the Core for further analysis, and upload the derivative outputs back to the Green Room or Core.
++
++The Home Directory is your default directory. When you download a copy of your Core files to JupyterHub, the files persists in the JupyterHub environment until deleted by you, so you can return to the session and continue your work at a later time without the need to retrieve the data from the Core again.
++
++The following sections focus on getting started with basic pilotcli commands in JupyterHub. For additional pilotcli commands and usage, see the article //Working with HDC Project Files in the Command Line Interface//.
++
++== Launching Pilot Command Line Interface ==
++
++1. Launch your Project and click the **JupyterHub** icon in in the workspace icon group.
++1. Click the **Terminal **launcher icon to open the Terminal.
++1. In the Jupyterhub Terminal, type {{code}}pilotcli{{/code}} to launch the latest version of the Pilot Command Line Interface.
++1. Use the {{code}}pilotcli --help{{/code}} at any time to show the welcome message again.
++
++{{code language="none"}}
++collaborator4@jupyter-collaborator4:~$ pilotcli
++Usage: pilotcli [OPTIONS] COMMAND [ARGS]...
++
++  What's new (Version 2.2.0):
++
++   1. CLI supports to perform multi-threading upload for file/folders
++
++   2. CLI supports to perform resumable upload for single file
++
++
++
++Options:
++  --help  Show this message and exit.
++
++Commands:
++  container_registry  Container Registry Actions.
++  dataset             Dataset Actions.
++  file                File Actions.
++  project             Project Actions.
++  use_config          Config Actions.
++  user                User Actions.
++{{/code}}
++
++== Logging into the Pilot Command Line Interface ==
++
++Users are required to login with platform credentials before performing any tasks through Pilot Command Line Interface.
++
++* Use the command {{code}}pilotcli user login{{/code}} to log into the Pilot Command Line Interface.
++
++{{code language="none"}}
++collaborator4@jupyter-collaborator4:~$ pilotcli user login
++Please, access https://iam.staging.pilot.indocresearch.com/realms/pilot/device?user_code=XXXX-XXXX to proceed
++ ▄▄▄▄▄▄▄  ▄ ▄▄     ▄ ▄▄▄▄    ▄ ▄▄▄▄▄▄▄
++ █ ▄▄▄ █ ▄ ▄███   ▀▀  █▀  ▀██▄ █ ▄▄▄ █
++ █ ▄ ▀ ▄ ▀▄ ▀▀ ▄█▀▄▀ ▀▀▄█▄▄▀ █████▄▄▀▄
++ ▄▄▄▄▄▄▄ ▀ ▀█▄ ▀▄ ██▀█ ▄▀▄▄  █ ▄ █▀▄▄▄
++ █ ▄▄▄ █ █▀█▄▀ █▀   █▀▀█ ▀▄█▄█▄▄▄█▀▄█
++ █ ███ █ █▀██▀▄ █▀▄▄▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
++ █▄▄▄▄▄█ ▄▀▄▄██▄▄▀▄  ▀▀▄  ▄▄▀▀▀▄ █▄▄▄█
++ Waiting validation finish...
++{{/code}}
++
++* (((
++You’ll be asked to validate your HDC user account using one of the provided methods.
++
++* Copy and paste the provided validation link into a new browser tab or
++* Scan the QR code with your mobile device.
++)))
++* Open the login window and enter your HDC username and password (i.e. your EBRAINS account credentials).
++* Grant access by clicking **Yes**.
++
++[[image:1723798355215-434.png||height="352" width="379"]]
++
++[[image:1723798365454-527.png||height="123" width="376"]]
++
++* After successful confirmation, return to the terminal in your JupyterHub browser tab.
++
++{{code language="none"}}
++Welcome to the Command Line Tool!
++{{/code}}
++
++* You’re now ready to start using the Pilot Command Line Interface to work with your Project data in JupyterHub.
++
++== Zone Restrictions when using Pilot Command Line Interface in JupyterHub ==
++
++When using the Pilot Command Line Interface in JupyterHub and the following actions are possible on the derivative files generated in JupyterHub:
++
++|=(% colspan="1" rowspan="1" %)(((
++**File Operation**
++)))|=(% colspan="1" rowspan="1" %)(((
++**Permitted in the **
++**Green Room**
++)))|=(% colspan="1" rowspan="1" %)(((
++**Permitted in the **
++**Core**
++)))
++|(% colspan="1" rowspan="1" %)File upload
++(upload derivative output files from JupyterHub to the Green Room or Core storage)|(% colspan="1" rowspan="1" %)(((
++Yes
++)))|(% colspan="1" rowspan="1" %)(((
++Yes
++)))
++|(% colspan="1" rowspan="1" %)File download
++(download files from Green Room or Core into JupyterHub)|(% colspan="1" rowspan="1" %)(((
++**No**
++)))|(% colspan="1" rowspan="1" %)(((
++Yes
++)))
++
++== Downloading Project Data to JupyterHub using the Pilot Command Line Interface ==
++
++After logging into the Pilot Command Line Interface, you can download data from the Project Core into the JupyterHub environment to start your data analyses.
++
++File related commands are grouped in the {{code}}file{{/code}} category. To view the full list of commands in this category, type {{code}}pilotcli file --help{{/code}}. To download project data, use the file sync command. To view the full list of commands in this category, type {{code}}pilotcli file sync --help{{/code}}.
++
++
++{{code language="none"}}
++collaborator4@jupyter-collaborator4:~$ pilotcli file sync --help
++Usage: pilotcli file sync [OPTIONS] [PATHS]... OUTPUT_PATH
++
++  Download files/folders from a given Project/folder/file in core zone.
++
++Options:
++  -z, --zone TEXT  Target Zone (i.e., core/greenroom)
++  --zip            Download files as a zip.
++  -i, --geid       Enable downloading by geid.
++  --help           Show this message and exit.
++{{/code}}
++
++=== Example ===
++
++Downloading a file from the Core to your Home Directory:
++
++Reminder: Please follow Linux conventions for file management. If your filename contains spaces, wrap it in single or double quotes.
++
++* //Filename~:// “Chemical Tracking Data.csv”
++* //Source~:// Project “Indoc Test Project”, “Core” storage zone, folder “collaborator4” {{code}}indoctestproject/collaborator4/Chemical Tracking Data.csv -z core{{/code}}
++* //Destination: //user's Home directory in the Guacamole or JupyterHub VM {{code}}.{{/code}}
++* //Command group/option: //{{code}}file sync{{/code}}
++
++{{code language="none"}}
++collaborator4@jupyter-collaborator4:~$ pilotcli file sync indoctestproject/collaborator4/'Chemical Tracking Data.csv' . -z core
++start downloading...
++Downloading Chemical Tracking Data.csv |██████████████████████████████ 100% 00:00
++File has been downloaded successfully and saved to: ./Chemical Tracking Data.csv
++{{/code}}
++
++To confirm successful download, type {{code}}ls{{/code}} and verify the file "Chemical Tracking Data.csv" is stored in the Home folder.
++
++{{code language="none"}}
++collaborator4@jupyter-collaborator4:~$ ls
++'Chemical Tracking Data.csv'   pilotcli
++{{/code}}
++
++The file “Chemical Tracking Data.csv” can be viewed in the JupyterHub graphical user interface:
++
++[[image:1723798383409-873.png||height="267" width="874"]]
++
++
++== Uploading Project Data from JupyterHub using the Pilot Command Line Interface ==
++
++After analyzing Project data inside the JupyterHub, you can upload the generated outputs back into the Project via the Pilot Command Line Interface.
++
++=== Example ===
++
++* //Filename//: Chemical Tracking Data rev.csv
++* //Source~:// user's Home directory in JupyterHub {{code}}.{{/code}}
++* //Destination//: Project “Indoc Test Project”, folder “collaborator4”, “Core” storage zone,
++{{code}}indoctestproject/collaborator4{{/code}} {{code}}-z core{{/code}}
++* //Command group/option~:// {{code}}file upload{{/code}}
++* //User message// (for upload back to the Core): “my workbench output, no additional sensitive data"
++* //Command~:// {{code}}pilotcli file upload ./'Chemical Tracking Data rev.csv' -p{{/code}} {{code}}indoctestproject/collaborator4 -z core -m "my workbench output, no additional sensitive data"{{/code}}
++
++When uploading data to the Core, you are reminded that you are bypassing the usual Green Room upload workflow. To confirm, type {{code}}y{{/code}} at the prompt, or {{code}}N{{/code}} to cancel.
++
++{{code language="none"}}
++collaborator4@jupyter-collaborator4:~$ pilotcli file upload ./'Chemical Tracking Data rev.csv' -p indoctestproject/collaborator4 -z core -m "my workbench output, no additional sensitive data"
++You are about to transfer data directly to the PILOT Core! In accordance with the PILOT Terms of Use, please confirm that you have made your best efforts to
++pseudonymize or anonymize the data and that you have the legal authority to transfer and make this data available for dissemination and use within the PILOT .If you
++need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to perform these actions.
++To cancel this transfer, enter [n/No]
++To confirm and proceed with the data transfer, enter [y/Yes]
++ [y/N]: y
++Starting upload of: ./Chemical Tracking Data rev.csv
++Pre-upload complete.
++Uploading Chemical Tracking Data rev.csv:  |██████████████████████████████ 100% 00:00
++Upload Time: 2.92s for 1 files
++All uploading jobs have finished.
++{{/code}}
++
++After completing the upload, you can confirm the new file “Chemical Tracking Data rev.csv" exists in the correct directory using the pilotcli file list command and/or in the Portal File Explorer.
++
++{{code language="none"}}
++collaborator4@jupyter-collaborator4:~$ pilotcli file list indoctestproject/collaborator4 -z core
++Chemical Tracking Data rev.csv  Chemical Tracking Data.csv
++{{/code}}
++
++[[image:1723798397694-530.png||height="217" width="863"]]
++
++----
++
++Copyright © 2023-2024 [[Indoc Systems>>url:https://www.indocsystems.com]].
++
++HealthDataCloud is powered by Pilot technology, a product of [[Indoc Systems>>url:https://www.indocsystems.com]].

HDC Project Workspace tool navigation Jupyterhub v1.0.0 2023-05-25.png

Author

...	...	@@ -1,1 +1,0 @@
1		-XWiki.sgevans

Size

...	...	@@ -1,1 +1,0 @@
1		-40.6 KB

Content

Project Workspace Jupyter Create Python Notebook v2.1.6 2023-02-07.png

Author

...	...	@@ -1,1 +1,0 @@
1		-XWiki.sgevans

Size

...	...	@@ -1,1 +1,0 @@
1		-793.7 KB

Content

Project Workspace Jupyter Launch Terminal v2.1.6 2023-02-07.png

Author

...	...	@@ -1,1 +1,0 @@
1		-XWiki.sgevans

Size

...	...	@@ -1,1 +1,0 @@
1		-346.3 KB

Content

1723798240534-407.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+XWiki.dsegebarth

Size

...	...	@@ -1,0 +1,1 @@
	1	+18.0 KB

Content

1723798257792-201.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+XWiki.dsegebarth

Size

...	...	@@ -1,0 +1,1 @@
	1	+42.2 KB

Content

1723798278604-114.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+XWiki.dsegebarth

Size

...	...	@@ -1,0 +1,1 @@
	1	+216.4 KB

Content

1723798293872-992.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+XWiki.dsegebarth

Size

...	...	@@ -1,0 +1,1 @@
	1	+113.5 KB

Content

1723798325144-485.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+XWiki.dsegebarth

Size

...	...	@@ -1,0 +1,1 @@
	1	+423.2 KB

Content

1723798338447-469.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+XWiki.dsegebarth

Size

...	...	@@ -1,0 +1,1 @@
	1	+27.1 KB

Content

1723798355215-434.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+XWiki.dsegebarth

Size

...	...	@@ -1,0 +1,1 @@
	1	+14.1 KB

Content

1723798365454-527.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+XWiki.dsegebarth

Size

...	...	@@ -1,0 +1,1 @@
	1	+10.0 KB

Content

1723798383409-873.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+XWiki.dsegebarth

Size

...	...	@@ -1,0 +1,1 @@
	1	+30.1 KB

Content

1723798397694-530.png

Author

...	...	@@ -1,0 +1,1 @@
	1	+XWiki.dsegebarth

Size

...	...	@@ -1,0 +1,1 @@
	1	+166.8 KB

Content