Changes for page Using JupyterHub in HDC

Last modified by Dennis Segebarth on 2024/10/02 18:14
From version 1.1
edited by Susan Evans
on 2023/07/11 13:35
Change comment: (Autosaved)
To version 1.6
edited by Susan Evans
on 2023/07/11 14:00
Change comment: (Autosaved)

Summary

Details

Page properties
Content
... ... @@ -180,8 +180,181 @@
180 180  
181 181  Afterwards, the environment will be listed when you open the Launcher to open a new Jupyter Notebook:
182 182  
183 +[[image:Project Workspace Jupyter view new Kernel 2023-07-11.png||height="25%" width="50%"]]
183 183  
184 184  
186 +and also from each opened Notebook, e.g., via **Kernel > Change Kernel…** :
185 185  
188 +[[image:Project Workspace Jupyter Kernel change Kernel dropdown 2023-07-11.png||height="64%" width="50%"]]
186 186  
190 += Installing New Python Packages =
191 +
192 +We highly recommend the use of virtual environments when installing new packages (see //Creating a Python Virtual Environment and Registering a Kernel// above for more details). Consequently, we recommend installing new packages via commands in the JupyterHub terminal in the corresponding virtual environments, instead of installing packages from within Jupyter Notebooks.
193 +
194 +Depending on the IT policies, outbound traffic may need to go through a proxy. If so, users will be required to provide the proxy command line argument such as pip, curl, wget, etc.
195 +
196 +For example:
197 +
198 +{{code language="none"}}
199 +pip install my_package
200 +{{/code}}
201 +
202 +If you are using conda to manage python packages:
203 +
204 +{{code language="none"}}
205 +conda install my_package
206 +{{/code}}
207 +
208 +The above information is provided as examples only. Please refer to documentation provided by your IT department with respect to proxy configuration.
209 +
210 += Using the Pilot Command Line Interface in a JupyterHub Terminal =
211 +
212 +The Pilot Command Line Interface (CLI) is deployed within JupyterHub as extension resource. Project members can use the Pilot Command Line Interface in a JupyterHub terminal to download Project data from the Core for further analysis, and upload the derivative outputs back to the Green Room or Core.
213 +
214 +The Home Directory is your default directory. When you download a copy of your Core files to JupyterHub, the files persists in the JupyterHub environment until deleted by you, so you can return to the session and continue your work at a later time without the need to retrieve the data from the Core again.
215 +
216 +The following sections focus on getting started with basic pilotcli commands in JupyterHub. For additional pilotcli commands and usage, see the article //Working with HDC Project Files in the Command Line Interface//.
217 +
218 +== Launching Pilot Command Line Interface ==
219 +
220 +1. Launch your Project and click the **JupyterHub** icon in in the workspace icon group.
221 +1. Click the **Terminal **launcher icon to open the Terminal.
222 +1. In the Jupyterhub Terminal, type {{code}}pilotcli{{/code}} to launch the latest version of the Pilot Command Line Interface.
223 +1. Use the {{code}}pilotcli --help{{/code}} at any time to show the welcome message again.
224 +
225 +{{code language="none"}}
226 +collaborator4@jupyter-collaborator4:~$ pilotcli
227 +Usage: pilotcli [OPTIONS] COMMAND [ARGS]...
228 +
229 + What's new (Version 2.2.0):
230 +
231 + 1. CLI supports to perform multi-threading upload for file/folders
232 +
233 + 2. CLI supports to perform resumable upload for single file
234 +
235 +
236 +
237 +Options:
238 + --help Show this message and exit.
239 +
240 +Commands:
241 + container_registry Container Registry Actions.
242 + dataset Dataset Actions.
243 + file File Actions.
244 + project Project Actions.
245 + use_config Config Actions.
246 + user User Actions.
247 +{{/code}}
248 +
249 +== Logging into the Pilot Command Line Interface ==
250 +
251 +Users are required to login with platform credentials before performing any tasks through Pilot Command Line Interface.
252 +
253 +* Use the command {{code}}pilotcli user login{{/code}} to log into the Pilot Command Line Interface.
254 +
255 +{{code language="none"}}
256 +collaborator4@jupyter-collaborator4:~$ pilotcli user login
257 +Please, access https://iam.staging.pilot.indocresearch.com/realms/pilot/device?user_code=XXXX-XXXX to proceed
258 + ▄▄▄▄▄▄▄ ▄ ▄▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
259 + █ ▄▄▄ █ ▄ ▄███ ▀▀ █▀ ▀██▄ █ ▄▄▄ █
260 + █ ▄ ▀ ▄ ▀▄ ▀▀ ▄█▀▄▀ ▀▀▄█▄▄▀ █████▄▄▀▄
261 + ▄▄▄▄▄▄▄ ▀ ▀█▄ ▀▄ ██▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
262 + █ ▄▄▄ █ █▀█▄▀ █▀ █▀▀█ ▀▄█▄█▄▄▄█▀▄█
263 + █ ███ █ █▀██▀▄ █▀▄▄▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
264 + █▄▄▄▄▄█ ▄▀▄▄██▄▄▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
187 187  
266 + Waiting validation finish...
267 +{{/code}}
268 +
269 +* (((
270 +You’ll be asked to validate your HDC user account using one of the provided methods.
271 +
272 +* Copy and paste the provided validation link into a new browser tab or
273 +* Scan the QR code with your mobile device.
274 +)))
275 +* Open the login window and enter your HDC username and password (i.e. your EBRAINS account credentials).
276 +* Grant access by clicking **Yes**.
277 +
278 +[[image:Pilotcli Jupyter user login Grant Access window v2.4.0 2023-05-25.png||height="46%" width="50%"]]
279 +
280 +[[image:Pilotcli Jupyter user login Device Login Successful v2.4.0 2023-05-25.png||height="16%" width="50%"]]
281 +
282 +* After successful confirmation, return to the terminal in your JupyterHub browser tab.
283 +
284 +{{code language="none"}}
285 +Welcome to the Command Line Tool!
286 +{{/code}}
287 +
288 +* You’re now ready to start using the Pilot Command Line Interface to work with your Project data in JupyterHub.
289 +
290 +== Zone Restrictions when using Pilot Command Line Interface in JupyterHub ==
291 +
292 +When using the Pilot Command Line Interface in JupyterHub and the following actions are possible on the derivative files generated in JupyterHub:
293 +
294 +|=(% colspan="1" rowspan="1" %)(((
295 +**File Operation**
296 +)))|=(% colspan="1" rowspan="1" %)(((
297 +**Permitted in the **
298 +**Green Room**
299 +)))|=(% colspan="1" rowspan="1" %)(((
300 +**Permitted in the **
301 +**Core**
302 +)))
303 +|(% colspan="1" rowspan="1" %)File upload 
304 +(upload derivative output files from JupyterHub to the Green Room or Core storage)|(% colspan="1" rowspan="1" %)(((
305 +Yes
306 +)))|(% colspan="1" rowspan="1" %)(((
307 +Yes
308 +)))
309 +|(% colspan="1" rowspan="1" %)File download
310 +(download files from Green Room or Core into JupyterHub)|(% colspan="1" rowspan="1" %)(((
311 +**No**
312 +)))|(% colspan="1" rowspan="1" %)(((
313 +Yes
314 +)))
315 +
316 +== Downloading Project Data to JupyterHub using the Pilot Command Line Interface ==
317 +
318 +After logging into the Pilot Command Line Interface, you can download data from the Project Core into the JupyterHub environment to start your data analyses.
319 +
320 +File related commands are grouped in the {{code}}file{{/code}} category. To view the full list of commands in this category, type {{code}}pilotcli file --help{{/code}}. To download project data, use the file sync command. To view the full list of commands in this category, type {{code}}pilotcli file sync --help{{/code}}.
321 +
322 +
323 +{{code language="none"}}
324 +collaborator4@jupyter-collaborator4:~$ pilotcli file sync --help
325 +Usage: pilotcli file sync [OPTIONS] [PATHS]... OUTPUT_PATH
326 +
327 + Download files/folders from a given Project/folder/file in core zone.
328 +
329 +Options:
330 + -z, --zone TEXT Target Zone (i.e., core/greenroom)
331 + --zip Download files as a zip.
332 + -i, --geid Enable downloading by geid.
333 + --help Show this message and exit.
334 +{{/code}}
335 +
336 +=== Example ===
337 +
338 +Downloading a file from the Core to your Home Directory:
339 +
340 +Reminder: Please follow Linux conventions for file management. If your filename contains spaces, wrap it in single or double quotes.
341 +
342 +* //Filename~:// “Chemical Tracking Data.csv”
343 +* //Source~:// Project “Indoc Test Project”, “Core” storage zone, folder “collaborator4” indoctestproject/collaborator4/Chemical Tracking Data.csv -z core
344 +* //Destination: //user's Home directory in the Guacamole or JupyterHub VM .
345 +* //Command group/option: //file sync
346 +
347 +{{code language="none"}}
348 +collaborator4@jupyter-collaborator4:~$ pilotcli file sync indoctestproject/collaborator4/'Chemical Tracking Data.csv' . -z core
349 +start downloading...
350 +Downloading Chemical Tracking Data.csv |██████████████████████████████ 100% 00:00
351 +File has been downloaded successfully and saved to: ./Chemical Tracking Data.csv
352 +{{/code}}
353 +
354 +
355 +
356 +
357 +
358 +To confirm successful download, type ls and verify the file {{code}}Chemical Tracking Data.csv{{/code}} is stored in the Home folder.
359 +
360 +[[image:Jupyter downloaded file in Home folder v2.4.11 2023-05-25 1850.png||height="15%" width="50%"]]
Jupyter downloaded file in Home folder v2.4.11 2023-05-25 1850.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.sgevans
Size
... ... @@ -1,0 +1,1 @@
1 +35.0 KB
Content
Pilotcli Jupyter user login Device Login Successful v2.4.0 2023-05-25.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.sgevans
Size
... ... @@ -1,0 +1,1 @@
1 +10.3 KB
Content
Pilotcli Jupyter user login Grant Access window v2.4.0 2023-05-25.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.sgevans
Size
... ... @@ -1,0 +1,1 @@
1 +13.9 KB
Content
Project Workspace Jupyter Kernel change Kernel dropdown 2023-07-11.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.sgevans
Size
... ... @@ -1,0 +1,1 @@
1 +38.9 KB
Content
Project Workspace Jupyter view new Kernel 2023-07-11.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.sgevans
Size
... ... @@ -1,0 +1,1 @@
1 +490.0 KB
Content