Using JupyterHub in HDC

JupyterHub is an open-source, multi-user version of Jupyter Notebook for performing analysis of Project files in the Core. More information can be found in the application documentation [[https:~~/~~/jupyter.org/>>https://jupyter.org/]].

2

3

= How it Works =

4

5

JupyterHub allows Project members to create or import Jupyter Notebooks into the Project Workspace environment, retrieve Project files from the Core, perform computational workflows on the data, and write the outputs back to the Core where they can be accessed by other Project members. JupyterHub spins up a new JupyterLab instance for each Project member.

= Prerequisites =

* Project Collaborator role or higher.

10

* JupyterHub has been configured for the Project by the Platform Administrator. See //Getting Access to JupyterHub//.

= Data Stewardship =

Users are reminded to abide by the Platform Terms of Use and any Project-specific restrictions when using Workspace tools to access data and code.

15

16

= Getting Access to JupyterHub =

17

18

JupyterHub is configured at the time of Project Setup. If you launch JupyterHub and receive a notice that it hasn’t been deployed for your project, please contact your Platform Administrator.

19

20

= Launching JupyterHub =

21

22

[[image:HDC Project Workspace tool navigation Jupyterhub v1.0.0 2023-05-25.png||height="10%" width="30%"]]

23

24

1. Launch your Project and click the **JupyterHub icon** in the left menu bar.

25

1. Click **Sign in with Keycloak** to initiate your session. JupyterHub automatically authenticates with your existing username and password and launches your session - no additional sign-in is required.

26

1. You can chose to either start a **Minimal environment**, which comes with Python, or a **Datascience environment**, which also includes R and Julia in addition to Python.

27

1. From the JupyterHub home page (a JupyterLab interface) you can now perform various actions such as creating and working on Jupyter Notebooks, importing existing ones, and using the Pilot Command Line Interface in the terminal to retrieve, analyze, and re-upload Project Core data, and create. Moreover, you can also use the pre-deployed and configured package management software conda to download, install, and manage for instance Python packages as per individual demand (see the sections //Installing New Python Packages// and //Creating a Virtual Python Environment and Registering a Kernel// below for more details).

28

1. When finished using JupyterHub, click **Logout** to end your session.

29

30

= Creating a Notebook =

31

32

Users can create a new Jupyter Notebook with Python 3 inside JupyterHub, with dedicated and persistent storage under the users' Home Directory.

33

34

1. In the Launcher, click the **Python 3 Notebook **icon, or click **File > New > Notebook**.

35

1. Create your Notebook.

36

37

[[image:Project Workspace Jupyter Create Python Notebook v2.1.6 2023-02-07.png||height="22%" width="50%"]]

38

39

= Launching the Terminal =

40

41

JupyterHub provides browser-based terminal access for advanced users to run commands directly in the system shell. Importantly, this allows users to sync data between for instance the Projects Core and their JupyterHub home directory using pilotcli, or to download and manage Python packages.

42

43

1. In the Launcher, click the **Terminal **icon, or click **File > New > Terminal**.

44

1. The terminal window opens.

45

46

[[image:Project Workspace Jupyter Launch Terminal v2.1.6 2023-02-07.png||height="9%" width="50%"]]

47

48

Ubuntu is used to host Jupyter Notebook. Use the command cat /etc/os-release to determine to current version of Ubuntu:

49

50

51

uname@jupyter-uname:/etc$ cat os-release

52

NAME="Ubuntu"

53

VERSION="20.04.4 LTS (Focal Fossa)"

54

ID=ubuntu

55

ID_LIKE=debian

56

PRETTY_NAME="Ubuntu 20.04.4 LTS"

57

VERSION_ID="20.04"

58

HOME_URL="https://www.ubuntu.com/"

59

SUPPORT_URL="https://help.ubuntu.com/"

60

BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"

61

PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"

62

VERSION_CODENAME=focal

63

UBUNTU_CODENAME=focal

64

65

66

= Creating a Python Virtual Environment and Registering a Kernel =

67

68

The user has full flexibility to use different virtual environment and/or package management systems. Please find the examples of using conda or Pythons in-built venv options described below. Importantly, in either case, the user has to register the new environment as a kernel using ipykernel, to make is accessible via the Jupyter Notebooks (see //Registering the new Virtual Environment as Kernel// for more details).

== Using conda ==

The package management software conda by Anaconda has become one of the most popular package management systems, especially for Data and Life Sciences. Therefore, conda is already pre-deployed and configured in each user’s JupyterHub. Please find the full documentation of conda [[here>>url:https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html]], and the corresponding documentation of how to manage virtual environments using conda [[here>>url:https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html]]. The following steps provide a short example of how you can use conda to create a new virtual environment using the JupyterHub terminal within the Platform.

73

74

At first, you need to activate conda. Since it is already pre-deployed and configured for you, all you need to do is launch a terminal within JupyterHub (see //Launching the Terminal// above) and execute the command {{code}}source activate{{/code}}. This will activate conda and you can see the success of this by the indication of the currently activated conda environment at the beginning of the line, displayed in parentheses - usually “base”:

75

76

77

username@jupyter-username:~$ source activate

78

(base) username@jupyter-username:~$

79

80

81

To create a new environment, run the following commands in the terminal after activating conda:

82

83

84

(base) username@jupyter-username:~$ conda create --name your_env_name

85

86

87

Replace {{code}}your_env_name{{/code}} with your preferred name for the environment. When being prompted by conda to confirm the creation of the environment at the specified location (per default in the users home directory - please do not change this location, to ensure persistency of your created environment), proceed with the creation by typing “y”, or abort the process by typing “N”. Once confirmed, conda will complete the environment creation process and remind you to activate the environment:

88

89

90

(base) username@jupyter-username:~$ conda create --name sample_env

91

Collecting package metadata (current_repodata.json): done

92

Solving environment: done

## Package Plan ##

environment location: /home/username/.conda_envs/sample_env

Proceed ([y]/n)? y

Preparing transaction: done

101

Verifying transaction: done

102

Executing transaction: done

103

#

104

# To activate this environment, use

105

#

106

# $ conda activate sample_env

107

#

108

# To deactivate an active environment, use

#

# $ conda deactivate

(base) username@jupyter-username:~$

113

114

115

Please note, at the end of the environment creation process, you will still remain in the previously activate environment (“base”, in this example). Therefore, please remember to activate the novel environment before installing any packages by running the command {{code}}conda activate your_env_name{{/code}} and replace “your_env_name” with the corresponding name you chose (“sample_env” in this example):

116

117

118

(base) username@jupyter-username:~$ conda activate sample_env

119

(/home/username/.conda_envs/sample_env) username@jupyter-username:~$

120

121

122

You can now install the desired packages in this new conda environment, for instance using the {{code}}conda install{{/code}} command. For example, in order to install the latest version of Python, run:

123

124

125

(/home/username/.conda_envs/sample_env) username@jupyter-username:~$ conda install python

126

127

128

To see a list of all installed packages in the currently activated environment (indicated in parentheses at the beginning of the line, “base” in this case), run:

129

130

131

(base) username@jupyter-username:~$ conda list

132

133

134

To see a list of all existing conda environments, run:

135

136

137

(base) username@jupyter-username:~$ conda info --envs

138

139

140

Please find many more examples and the full documentation of how to manage conda environments [[here>>url:https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#]]. Importantly, please remember to follow the instructions in the //Registering the new Virtual Environment as Kernel// section below, to make the virtual environment accessible via the Jupyter Notebooks.

== Using venv ==

As an alternative to using conda, you can also use the Python native package venv. Please find the full documentation of venv [[here>>url:https://docs.python.org/3/library/venv.html#]], and a short example of how to create a new virtual environment using venv below:

145

146

147

username@jupyter-username:~$ python3 -m venv your_env_name

148

username@jupyter-username:~$ source your_env_name/bin/activate

149

150

151

== Registering the new Virtual Environment as Kernel ==

152

153

In order to make the newly created virtual environment accessible for the Jupyter Notebooks, you have to register it using ipykernel. Importantly, please make sure that the corresponding environment is currently active before running the following commands:

154

155

156

username@jupyter-username:~$ python -m ipykernel install --user --name=your_env_name

157

158

159

Please replace {{code}}your_env_name{{/code}} with the name of your newly created environment. Depending on which package and/or virtual environment management system you chose to use, you may have to install ipykernel in the newly created environment first. Remember to activate the newly created environment and then run one of the following commands to install ipykernel, depending on your package management system of choice:

160

161

162

(your_env_name) username@jupyter-username:~$ conda install -c anaconda ipykernel

or:

username@jupyter-username:~$ pip install ipykernel