Changes for page Using JupyterHub in HDC

Last modified by Dennis Segebarth on 2024/10/02 18:14
From version 2.1
edited by Susan Evans
on 2023/07/11 14:10
Change comment: There is no comment for this version
To version 1.2
edited by Susan Evans
on 2023/07/11 13:40
Change comment: (Autosaved)

Summary

Details

Page properties
Content
... ... @@ -1,8 +6,3 @@
1 -{{box cssClass="floatinginfobox" title="Table of Contents"}}
2 -{{toc depth="2"/}}
3 -{{/box}}
4 -
5 -
6 6  JupyterHub is an open-source, multi-user version of Jupyter Notebook for performing analysis of Project files in the Core. More information can be found in the application documentation [[https:~~/~~/jupyter.org/>>https://jupyter.org/]].
7 7  
8 8  = How it Works =
... ... @@ -227,190 +227,4 @@
227 227  1. In the Jupyterhub Terminal, type {{code}}pilotcli{{/code}} to launch the latest version of the Pilot Command Line Interface.
228 228  1. Use the {{code}}pilotcli --help{{/code}} at any time to show the welcome message again.
229 229  
230 -{{code language="none"}}
231 -collaborator4@jupyter-collaborator4:~$ pilotcli
232 -Usage: pilotcli [OPTIONS] COMMAND [ARGS]...
233 233  
234 - What's new (Version 2.2.0):
235 -
236 - 1. CLI supports to perform multi-threading upload for file/folders
237 -
238 - 2. CLI supports to perform resumable upload for single file
239 -
240 -
241 -
242 -Options:
243 - --help Show this message and exit.
244 -
245 -Commands:
246 - container_registry Container Registry Actions.
247 - dataset Dataset Actions.
248 - file File Actions.
249 - project Project Actions.
250 - use_config Config Actions.
251 - user User Actions.
252 -{{/code}}
253 -
254 -== Logging into the Pilot Command Line Interface ==
255 -
256 -Users are required to login with platform credentials before performing any tasks through Pilot Command Line Interface.
257 -
258 -* Use the command {{code}}pilotcli user login{{/code}} to log into the Pilot Command Line Interface.
259 -
260 -{{code language="none"}}
261 -collaborator4@jupyter-collaborator4:~$ pilotcli user login
262 -Please, access https://iam.staging.pilot.indocresearch.com/realms/pilot/device?user_code=XXXX-XXXX to proceed
263 - ▄▄▄▄▄▄▄ ▄ ▄▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
264 - █ ▄▄▄ █ ▄ ▄███ ▀▀ █▀ ▀██▄ █ ▄▄▄ █
265 - █ ▄ ▀ ▄ ▀▄ ▀▀ ▄█▀▄▀ ▀▀▄█▄▄▀ █████▄▄▀▄
266 - ▄▄▄▄▄▄▄ ▀ ▀█▄ ▀▄ ██▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
267 - █ ▄▄▄ █ █▀█▄▀ █▀ █▀▀█ ▀▄█▄█▄▄▄█▀▄█
268 - █ ███ █ █▀██▀▄ █▀▄▄▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
269 - █▄▄▄▄▄█ ▄▀▄▄██▄▄▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
270 -
271 - Waiting validation finish...
272 -{{/code}}
273 -
274 -* (((
275 -You’ll be asked to validate your HDC user account using one of the provided methods.
276 -
277 -* Copy and paste the provided validation link into a new browser tab or
278 -* Scan the QR code with your mobile device.
279 -)))
280 -* Open the login window and enter your HDC username and password (i.e. your EBRAINS account credentials).
281 -* Grant access by clicking **Yes**.
282 -
283 -[[image:Pilotcli Jupyter user login Grant Access window v2.4.0 2023-05-25.png||height="46%" width="50%"]]
284 -
285 -[[image:Pilotcli Jupyter user login Device Login Successful v2.4.0 2023-05-25.png||height="16%" width="50%"]]
286 -
287 -* After successful confirmation, return to the terminal in your JupyterHub browser tab.
288 -
289 -{{code language="none"}}
290 -Welcome to the Command Line Tool!
291 -{{/code}}
292 -
293 -* You’re now ready to start using the Pilot Command Line Interface to work with your Project data in JupyterHub.
294 -
295 -== Zone Restrictions when using Pilot Command Line Interface in JupyterHub ==
296 -
297 -When using the Pilot Command Line Interface in JupyterHub and the following actions are possible on the derivative files generated in JupyterHub:
298 -
299 -|=(% colspan="1" rowspan="1" %)(((
300 -**File Operation**
301 -)))|=(% colspan="1" rowspan="1" %)(((
302 -**Permitted in the **
303 -**Green Room**
304 -)))|=(% colspan="1" rowspan="1" %)(((
305 -**Permitted in the **
306 -**Core**
307 -)))
308 -|(% colspan="1" rowspan="1" %)File upload 
309 -(upload derivative output files from JupyterHub to the Green Room or Core storage)|(% colspan="1" rowspan="1" %)(((
310 -Yes
311 -)))|(% colspan="1" rowspan="1" %)(((
312 -Yes
313 -)))
314 -|(% colspan="1" rowspan="1" %)File download
315 -(download files from Green Room or Core into JupyterHub)|(% colspan="1" rowspan="1" %)(((
316 -**No**
317 -)))|(% colspan="1" rowspan="1" %)(((
318 -Yes
319 -)))
320 -
321 -== Downloading Project Data to JupyterHub using the Pilot Command Line Interface ==
322 -
323 -After logging into the Pilot Command Line Interface, you can download data from the Project Core into the JupyterHub environment to start your data analyses.
324 -
325 -File related commands are grouped in the {{code}}file{{/code}} category. To view the full list of commands in this category, type {{code}}pilotcli file --help{{/code}}. To download project data, use the file sync command. To view the full list of commands in this category, type {{code}}pilotcli file sync --help{{/code}}.
326 -
327 -
328 -{{code language="none"}}
329 -collaborator4@jupyter-collaborator4:~$ pilotcli file sync --help
330 -Usage: pilotcli file sync [OPTIONS] [PATHS]... OUTPUT_PATH
331 -
332 - Download files/folders from a given Project/folder/file in core zone.
333 -
334 -Options:
335 - -z, --zone TEXT Target Zone (i.e., core/greenroom)
336 - --zip Download files as a zip.
337 - -i, --geid Enable downloading by geid.
338 - --help Show this message and exit.
339 -{{/code}}
340 -
341 -=== Example ===
342 -
343 -Downloading a file from the Core to your Home Directory:
344 -
345 -Reminder: Please follow Linux conventions for file management. If your filename contains spaces, wrap it in single or double quotes.
346 -
347 -* //Filename~:// “Chemical Tracking Data.csv”
348 -* //Source~:// Project “Indoc Test Project”, “Core” storage zone, folder “collaborator4” {{code}}indoctestproject/collaborator4/Chemical Tracking Data.csv -z core{{/code}}
349 -* //Destination: //user's Home directory in the Guacamole or JupyterHub VM {{code}}.{{/code}}
350 -* //Command group/option: //{{code}}file sync{{/code}}
351 -
352 -{{code language="none"}}
353 -collaborator4@jupyter-collaborator4:~$ pilotcli file sync indoctestproject/collaborator4/'Chemical Tracking Data.csv' . -z core
354 -start downloading...
355 -Downloading Chemical Tracking Data.csv |██████████████████████████████ 100% 00:00
356 -File has been downloaded successfully and saved to: ./Chemical Tracking Data.csv
357 -{{/code}}
358 -
359 -To confirm successful download, type {{code}}ls{{/code}} and verify the file "Chemical Tracking Data.csv" is stored in the Home folder.
360 -
361 -{{code language="none"}}
362 -collaborator4@jupyter-collaborator4:~$ ls
363 -'Chemical Tracking Data.csv' pilotcli
364 -{{/code}}
365 -
366 -The file “Chemical Tracking Data.csv” can be viewed in the JupyterHub graphical user interface:
367 -
368 -[[image:Jupyter downloaded file in Home folder v2.4.11 2023-05-25 1850.png||height="15%" width="50%"]]
369 -
370 -
371 -== Uploading Project Data from JupyterHub using the Pilot Command Line Interface ==
372 -
373 -After analyzing Project data inside the JupyterHub, you can upload the generated outputs back into the Project via the Pilot Command Line Interface.
374 -
375 -=== Example ===
376 -
377 -* //Filename//: Chemical Tracking Data rev.csv
378 -* //Source~:// user's Home directory in JupyterHub {{code}}.{{/code}}
379 -* //Destination//: Project “Indoc Test Project”, folder “collaborator4”, “Core” storage zone,
380 -{{code}}indoctestproject/collaborator4{{/code}} {{code}}-z core{{/code}}
381 -* //Command group/option~:// {{code}}file upload{{/code}}
382 -* //User message// (for upload back to the Core): “my workbench output, no additional sensitive data"
383 -* //Command~:// {{code}}pilotcli file upload ./'Chemical Tracking Data rev.csv' -p{{/code}} {{code}}indoctestproject/collaborator4 -z core -m "my workbench output, no additional sensitive data"{{/code}}
384 -
385 -When uploading data to the Core, you are reminded that you are bypassing the usual Green Room upload workflow. To confirm, type {{code}}y{{/code}} at the prompt, or {{code}}N{{/code}} to cancel.
386 -
387 -{{code language="none"}}
388 -collaborator4@jupyter-collaborator4:~$ pilotcli file upload ./'Chemical Tracking Data rev.csv' -p indoctestproject/collaborator4 -z core -m "my workbench output, no additional sensitive data"
389 -You are about to transfer data directly to the PILOT Core! In accordance with the PILOT Terms of Use, please confirm that you have made your best efforts to
390 -pseudonymize or anonymize the data and that you have the legal authority to transfer and make this data available for dissemination and use within the PILOT .If you
391 -need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to perform these actions.
392 -To cancel this transfer, enter [n/No]
393 -To confirm and proceed with the data transfer, enter [y/Yes]
394 - [y/N]: y
395 -Starting upload of: ./Chemical Tracking Data rev.csv
396 -Pre-upload complete.
397 -Uploading Chemical Tracking Data rev.csv: |██████████████████████████████ 100% 00:00
398 -Upload Time: 2.92s for 1 files
399 -All uploading jobs have finished.
400 -{{/code}}
401 -
402 -After completing the upload, you can confirm the new file “Chemical Tracking Data rev.csv" exists in the correct directory using the pilotcli file list command and/or in the Portal File Explorer.
403 -
404 -{{code language="none"}}
405 -collaborator4@jupyter-collaborator4:~$ pilotcli file list indoctestproject/collaborator4 -z core
406 -Chemical Tracking Data rev.csv Chemical Tracking Data.csv
407 -{{/code}}
408 -
409 -[[image:Jupyterhub file upload back to core v2.4.11 2023-05-25 1926.png||height="13%" width="50%"]]
410 -
411 -----
412 -
413 -Copyright © 2023 [[Indoc Research>>url:https://www.indocresearch.org/]].
414 -
415 -HealthDataCloud is powered by Pilot technology, a product of [[Indoc Research>>url:https://www.indocresearch.org/]].
416 -
Jupyter downloaded file in Home folder v2.4.11 2023-05-25 1850.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -35.0 KB
Content
Jupyterhub file upload back to core v2.4.11 2023-05-25 1926.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -507.5 KB
Content
Pilotcli Jupyter user login Device Login Successful v2.4.0 2023-05-25.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -10.3 KB
Content
Pilotcli Jupyter user login Grant Access window v2.4.0 2023-05-25.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -13.9 KB
Content