Version 3.1 by Susan Evans on 2023/07/11 13:01
Show last authors
1 {{box cssClass="floatinginfobox" title="Table of Contents"}}
2 {{toc depth="2"/}}
3 {{/box}}
4
5 The Pilot Command Line Interface (pilotcli) is a binary executable program that provides a convenient way for users to perform file actions and platform-related tasks programmatically. The Pilot Command Line Interface can be downloaded and run on on your local Linux and Linux subsystems (as Windows user, see for instance the [[Windows Subsystem for Linux installation guide>>url:https://learn.microsoft.com/en-us/windows/wsl/install]]) to support large file uploads. For convenience, pilotcli is also pre-deployed and installed in the users JupyterHub accessible via worskpace tools.
6
7 = Data Stewardship =
8
9 Users are reminded to abide by the Platform Terms of Use and any Project-specific restrictions when using the Command Line Interface to access data and code.
10
11 = Using the Pilot Command Line Interface in JupyterHub =
12
13 To get started with the Pilot Command Line Interface in the HDC JupyterHub workspace tool, see the article //Using JupyterHub in HDC //and refer to the remaining sections in this article for a full description of the pilotcli commands.
14
15 = Downloading the Pilot Command Line Interface =
16
17 If you wish to use the Pilot Command Line Interface to upload large files from your local environment, it is necessary to download Pilot Command Line Interface tool on your Linux system. The recommended Linux version is Ubuntu 20.04.
18
19 * Download the pilotcli tool [[here>>url:https://object.hdc.humanbrainproject.eu/public-resources/pilotcli]].
20
21 After download, please grant execution permission to the binary file so that Pilot Command Line Interface can be used. To grant permission to the binary file, use the following command in the downloaded pilotcli directory.
22
23 {{code language="none"}}
24 $ chmod +x ./pilotcli
25 {{/code}}
26
27
28 If you have sudo permission, you can make the Pilot Command Line Interface more accessible in any directory by moving it to the /usr/local/bin directory:
29
30 {{code language="none"}}
31 $ mv ./pilotcli /usr/local/bin
32 {{/code}}
33
34 Without sudo permission, you can still use the Pilot Command Line Interface in a different directory by referring the Command Line Interface path. For example, use a command line Interface in the Downloads folder:
35
36 {{code language="none"}}
37 $ ~/Downloads/pilotcli
38 {{/code}}
39
40 = Pilot Command Line Interface Command Summary =
41
42 Commands are organized into main groups:
43
44 1. **User**// - //User related actions such as login and logout
45 1. **Project** //- //Project related actions, including list Project Names and Codes for all Projects accessible by the logged-in user.
46 1. **File **- Actions on files. User could use file action commands to list files/folders, upload files/folders, list/export attribute, and download files as well.
47 1. **Dataset **- Actions related to datasets.
48
49 = User Commands: pilotcli **user** =
50
51 == Help ==
52
53 {{code language="none"}}
54 $ pilotcli user --help
55 Usage: pilotcli user [OPTIONS] COMMAND [ARGS]...
56
57 Options:
58 --help Show this message and exit.
59
60 Commands:
61 login For user to login.
62 logout For user to logout.
63 {{/code}}
64
65 == Command: **login** ==
66
67 * **Options**: -u ~-~-username, -p ~-~-password
68 * **Usage**: Allows users to log into the platform through pilotcli. After login, the user must be actively using the Pilot Command Line Interface to remain logged in, or they will be logged out automatically after the session is idle for five minutes.
69 * **Prerequisites**: None.
70
71 {{code language="none"}}
72 $ pilotcli user login --help
73 Usage: pilotcli user login [OPTIONS]
74
75 For user to login.
76
77 Options:
78 --help Show this message and exit.
79 {{/code}}
80
81 === Example ===
82
83 User login:
84
85 1) Use the output device validation link or QR code to launch authentication validation:
86
87 {{code language="none"}}
88 $ pilotcli user login
89 Please, access https://iam.hdc.humanbrainproject.eu/realms/hdc/device?user_code=XXXX-XXXX to proceed
90 ▄▄▄▄▄▄▄ ▄▄ ▄ ▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
91 █ ▄▄▄ █ ▄ ▄▄▀█ █▀▀▀▄▀█▀ ▀██▄ █ ▄▄▄ █
92 █ ███ █ ██▄ ▀▀▄▀▄▄█▄ ▄ ▀▀▀▄ █ ███ █
93 █▄▄▄▄▄█ █▀▄▀█▀▄▀█▀▄ █ ▄ ▄ █▀█ █▄▄▄▄▄█
94 ▄▄▄▄▄▄▄ ▀█▀█▄▄█▄ █▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
95 █ ▄▄▄ █ █▄█▄██ ▀ ▀█▀▀█ ▀▄█▄█▄▄▄█▀▄█
96 █ ███ █ █▀█ ▄▀██▄█▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
97 █▄▄▄▄▄█ ▄▀▄ ██▄▀▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
98
99
100 Waiting validation finish...
101 {{/code}}
102
103 2) Grant access by clicking Yes:
104
105 [[image:Pilot CLI authentication.png||height="34%" width="40%"]]
106
107 3) Device Login Successful message will appear and your authentication browser window can be closed.
108
109 4) The output of the Command Line Tool will also be updated to notify you about the successful login:
110
111 {{code language="none"}}
112 $ pilotcli user login
113 Please, access https://iam.hdc.humanbrainproject.eu/realms/hdc/device?user_code=XXXX-XXXX to proceed
114 ▄▄▄▄▄▄▄ ▄▄ ▄ ▄ ▄ ▄▄▄▄ ▄ ▄▄▄▄▄▄▄
115 █ ▄▄▄ █ ▄ ▄▄▀█ █▀▀▀▄▀█▀ ▀██▄ █ ▄▄▄ █
116 █ ███ █ ██▄ ▀▀▄▀▄▄█▄ ▄ ▀▀▀▄ █ ███ █
117 █▄▄▄▄▄█ █▀▄▀█▀▄▀█▀▄ █ ▄ ▄ █▀█ █▄▄▄▄▄█
118 ▄▄▄▄▄▄▄ ▀█▀█▄▄█▄ █▀█ ▄▀▄▄ █ ▄ █▀▄▄▄
119 █ ▄▄▄ █ █▄█▄██ ▀ ▀█▀▀█ ▀▄█▄█▄▄▄█▀▄█
120 █ ███ █ █▀█ ▄▀██▄█▀▀█▄▀▀█▄▀█ ▀ ▀▄▀██
121 █▄▄▄▄▄█ ▄▀▄ ██▄▀▀▄ ▀▀▄ ▄▄▀▀▀▄ █▄▄▄█
122
123
124 Waiting validation finish...
125 Welcome to Command Line Tool!
126 {{/code}}
127
128 == Command: **logout** ==
129
130 * **Options**: -y ~-~-yes
131 * **Usage**: Allows users to log out of the platform through pilotcli.
132 * **Prerequisites**: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'.
133
134 {{code language="none"}}
135 $ pilotcli user logout --help
136 Usage: pilotcli user logout [OPTIONS]
137
138 For user to logout.
139
140 Options:
141 -y, --yes Input Y/yes to confirm you want to logout, otherwise input N/no
142 to remain logged in.
143
144 --help Show this message and exit.
145 {{/code}}
146
147 === Examples ===
148
149 User logout:
150
151 {{code language="none"}}
152 $ pilotcli user logout
153 Are you sure you want to logout? [y/N]: y
154 Logged out successfully. Bye!
155 {{/code}}
156
157 User logout canceled:
158
159 {{code language="none"}}
160 $ pilotcli user logout
161 Are you sure you want to logout? [y/N]: n
162 Logout canceled!
163 {{/code}}
164
165 User logout with option to skip additional confirmation:
166
167 {{code language="none"}}
168 $ pilotcli user logout -y
169 Logged out successfully. Bye!
170 {{/code}}
171
172 = Project Commands: pilotcli **project** =
173
174 == Help ==
175
176 {{code language="none"}}
177 $ pilotcli project --help
178 Usage: pilotcli project [OPTIONS] COMMAND [ARGS]...
179
180 Options:
181 --help Show this message and exit.
182
183 Commands:
184 list List accessible projects.
185 {{/code}}
186
187 == Command: **list** ==
188
189 * **Options**: None
190 * **Usage**: List all Projects to which the user has access
191 * **Prerequisite**: User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.'
192
193 {{code language="none"}}
194 $ pilotcli project list --help
195 Usage: pilotcli project list [OPTIONS]
196
197 List accessible projects.
198
199 Options:
200 --page INTEGER The page to be listed [default: 0]
201 --page-size INTEGER number of objects per page [default: 10]
202 --order [asc|desc] sorting order [default: desc]
203 --order-by [created_at|name|code]
204 sorting column [default: created_at]
205 -d, --detached whether run in detached mode
206 --help Show this message and exit.
207 {{/code}}
208
209 === Examples ===
210
211 Get a list of all Projects the user has access to:
212
213 {{code language="none"}}
214 $ pilotcli project list
215 Project Name Project Code
216 ---------------------------------------------------------------------------
217 Test Project 02-13 | test230213byshan
218 Test1_Audit | test1audit
219 Test20221109 | test20220109
220 Indoc Test Project | indoctestproject
221
222 Page: 0, Number of projects: 4
223 {{/code}}
224
225 By default, the table will only display the first ten elements. If the user has access to more than ten Projects, you will be prompted to use your arrow keys to select one of two options to either view the next ten items (//next page//), or to exit (//exit//). The selected option is indicated by two arrows and can be confirmed by hitting enter:
226
227 {{code language="none"}}
228 Page: 0, Number of projects: 10
229
230 What do you want? (Use arrow keys)
231 » next page
232 exit
233 {{/code}}
234
235 = File Commands: pilotcli **file** =
236
237 == Help ==
238
239 {{code language="none"}}
240 $ pilotcli file --help
241 Usage: pilotcli file [OPTIONS] COMMAND [ARGS]...
242
243 Options:
244 --help Show this message and exit.
245
246 Commands:
247 attribute-export Export attribute template from a given Project.
248 attribute-list List attribute templates of a given Project.
249 list List files and folders inside a given Project/folder.
250 resume Resume the upload process with given manifest file.
251 sync Download files/folders from a given Project/folder/file...
252 upload Upload files/folders to a given Project path.
253 {{/code}}
254
255 == Command: **attribute-list** ==
256
257 * **Options**: -p ~-~-project-code
258 * **Usage**: List all attributes from a specified Project.
259 * **Prerequisite**:
260
261 * User login is required, otherwise an error is returned: ‘The current login session is invalid. Please login to continue.’
262 * Requires the Project code, which can be retrieved via {{code}}pilotcli project list{{/code}}.
263
264 {{code language="none"}}
265 $ pilotcli file attribute-list --help
266 Usage: pilotcli file attribute-list [OPTIONS]
267
268 List attribute templates of a given Project.
269
270 Options:
271 -p, --project-code TEXT Project Code
272 --help Show this message and exit.
273 {{/code}}
274
275 === Examples ===
276
277 List attributes from a given Project by providing the Project Code at the prompt. In this example, “cli” is the corresponding Project Code:
278
279 {{code language="none"}}
280 $ pilotcli file attribute-list
281 ProjectCode: cli
282
283 Manifest1
284 --------------------------------------------------------------------------
285 | Attribute Name | Type | Value | Optional |
286 --------------------------------------------------------------------------
287 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
288 --------------------------------------------------------------------------
289 | attr2 | text | None | False |
290 --------------------------------------------------------------------------
291 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
292 --------------------------------------------------------------------------
293
294 Manifest2
295 --------------------------------------------------------------------------
296 | Attribute Name | Type | Value | Optional |
297 --------------------------------------------------------------------------
298 | attr1 | multiple_choice | a1,a2,a3,a4 | True |
299 --------------------------------------------------------------------------
300 | attr2 | text | None | False |
301 --------------------------------------------------------------------------
302 | attr3 | multiple_choice | t1,t2,t3 | True |
303 --------------------------------------------------------------------------
304 All Attributes fetched successfully.
305 {{/code}}
306
307 List attributes from given a Project and provide the Project Code “cli” as an option in the same line {{code}}-p cli{{/code}}:
308
309 {{code language="none"}}
310 $ pilotcli file attribute-list -p cli
311
312 Manifest1
313 --------------------------------------------------------------------------
314 | Attribute Name | Type | Value | Optional |
315 --------------------------------------------------------------------------
316 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
317 --------------------------------------------------------------------------
318 | attr2 | text | None | False |
319 --------------------------------------------------------------------------
320 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
321 --------------------------------------------------------------------------
322
323 Manifest2
324 --------------------------------------------------------------------------
325 | Attribute Name | Type | Value | Optional |
326 --------------------------------------------------------------------------
327 | attr1 | multiple_choice | a1,a2,a3,a4 | True |
328 --------------------------------------------------------------------------
329 | attr2 | text | None | False |
330 --------------------------------------------------------------------------
331 | attr3 | multiple_choice | t1,t2,t3 | True |
332 --------------------------------------------------------------------------
333 All Attributes fetched successfully.
334 {{/code}}
335
336 == Command: **attribute-export** ==
337
338 * **Options**: -p ~-~-project-code, -n ~-~-attribute-name
339 * **Usage**: Export an attribute template as JSON-file from a given Project
340 * (((
341 **Prerequisite**:
342
343 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
344 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
345 * Requires the Project code, which can be retrieved via pilotcli project list.
346 * Requires the Attribute template name. If the attribute template name does not exist or is spelled incorrectly, an error is returned: "Attribute 'sample-template' not found in Project. Please verify and try again."
347 * pilotcli must have the permission to write a file in the directory, otherwise an error is returned.
348 )))
349
350 {{code language="none"}}
351 $ pilotcli file attribute-export --help
352 Usage: pilotcli file attribute-export [OPTIONS]
353
354 Export attribute template from a given Project.
355
356 Options:
357 -p, --project-code TEXT Project Code
358 -n, --attribute-name TEXT Attribute Template Name
359 --help Show this message and exit.
360 {{/code}}
361
362 === Examples ===
363
364 Export an attribute template from a given Project. The user is prompted to provide the Project Code (“cli”) and the Attribute Template Name (“Manifest1”):
365
366 {{code language="none"}}
367 $ pilotcli file attribute-export
368 ProjectCode: cli
369 AttributeName: Manifest1
370
371 Manifest1
372 --------------------------------------------------------------------------
373 | Attribute Name | Type | Value | Optional |
374 --------------------------------------------------------------------------
375 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
376 --------------------------------------------------------------------------
377 | attr2 | text | None | False |
378 --------------------------------------------------------------------------
379 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
380 --------------------------------------------------------------------------
381 Template saved successfully: cli_Manifest1_template.json
382 Attribute definition saved successfully: cli_Manifest1_definition.json
383 {{/code}}
384
385 Export an attribute template from a given Project using options to provide both the Project Code {{code}}(-p cli){{/code}} and the Attribute Template Name {{code}}(-n Manifest1){{/code}}, “cli” and “Manifest1” in this case, respectively:
386
387 {{code language="none"}}
388 $ pilotcli file attribute-export -p cli -n Manifest1
389
390 Manifest1
391 --------------------------------------------------------------------------
392 | Attribute Name | Type | Value | Optional |
393 --------------------------------------------------------------------------
394 | attr1 | multiple_choice | a1,a2,a3,a4,a5 | False |
395 --------------------------------------------------------------------------
396 | attr2 | text | None | False |
397 --------------------------------------------------------------------------
398 | attr3 | multiple_choice | t1,t2,t3,t4,t5 | True |
399 --------------------------------------------------------------------------
400 Template saved successfully: cli_Manifest1_template.json
401 Attribute definition saved successfully: cli_Manifest1_definition.json
402 {{/code}}
403
404 == Command: **list** ==
405
406 * **Options**: -z ~-~-zone
407 * **Usage**: List files and folders in the specified directory within the Platform. Folders will be displayed in blue and files will be displayed in white (or black, depending on the background of your terminal). If no zone (i.e. Green Room or Core) is specified, Green Room will be used as default.
408 * (((
409 **Prerequisite**:
410
411 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
412 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
413 * Requires the Project code, which can be retrieved via pilotcli project list.
414 )))
415
416 {{code language="none"}}
417 $ pilotcli file list --help
418 Usage: pilotcli file list [OPTIONS] PATHS
419
420 List files and folders inside a given Project/folder.
421
422 Options:
423 -z, --zone TEXT Target Zone (i.e., core/greenroom) [default:
424 greenroom] [default: greenroom]
425
426 --page INTEGER The page to be listed [default: 0]
427 --page-size INTEGER number of objects per page [default: 10]
428 -d, --detached whether run in detached mode
429 --help Show this message and exit.
430 {{/code}}
431
432 === Examples ===
433
434 List folders in the directory “admin” in the Project with the Project Code “indoctestproject”. Since no zone was specified, the corresponding directory in the Green Room will be queried:
435
436 {{code language="none"}}
437 $ pilotcli file list indoctestproject/admin
438 test_folder1 test_folder2 file1 file2 file3
439 {{/code}}
440
441 To list the corresponding elements in the “indoctestproject/admin” directory in the Projects' Core, the zone needs to be specified {{code}}(-z core){{/code}}:
442
443 {{code language="none"}}
444 $ pilotcli file list indoctestproject/admin -z core
445 folder1 folder2 test_folder.zip file.txt sample.txt large-file.json
446 {{/code}}
447
448 == Command: **upload** ==
449
450 * **Options**: -p ~-~-project, -a ~-~-attribute, -t ~-~-tag, -z ~-~-zone, -m ~-~-upload-message, -s ~-~-source-file, ~-~-zip, -td ~-~-thread, -o ~-~-output path
451 * **Usage**: Upload files/folders to the given project, with optional tags and attributes, to the target folder.
452 * (((
453 **Prerequisite**:
454
455 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
456 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
457 * When uploading file to the Project Core, the user must have access to the Core (Project Contributors cannot upload files/folders to the Project Core).
458 )))
459
460 {{code language="none"}}
461 $ pilotcli file upload --help
462 Usage: pilotcli file upload [OPTIONS] [PATHS]...
463
464 Upload files/folders to a given Project path.
465
466 Options:
467 -p, --project-path TEXT Project folder path starting from Project code.
468 (i.e., indoctestproject/user/folder) [required]
469
470 -a, --attribute TEXT File Attribute Template used for annotating files
471 during upload.
472
473 -t, --tag TEXT Add a tag to the file. This option could be used
474 multiple times for adding multiple tags.
475
476 -z, --zone TEXT Target Zone (i.e., core/greenroom) [default:
477 greenroom] [default: greenroom]
478
479 -m, --upload-message TEXT The message used to comment on the purpose of
480 uploading your processed file [default: ]
481
482 -s, --source-file TEXT The Project path of the source file of your
483 processed files.
484
485 --zip Upload folder as a compressed zip file.
486 -td, --thread INTEGER The number of thread for upload a file [default:
487 1]
488
489 -o, --output-path TEXT The output path for the manifest file of
490 resumable upload [default: ./manifest.json]
491
492 --help Show this message and exit.
493 {{/code}}
494
495 === Examples ===
496
497 Upload a file (or folder) to a specified directory of a Project’s Green Room, in this case the file located at {{code}}./sample_folder/sample_file1{{/code}} will be uploaded to the folder “admin” in the Project with the Project Code “cli” {{code}}(-p cli/admin){{/code}}:
498
499 {{code language="none"}}
500 $ pilotcli file upload ./sample_folder/sample_file1 -p cli/admin
501 Starting upload of: ./sample_folder/sample_file1
502 Pre-upload complete.
503 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: e958247c-ce05-48b2-9038-78b774836256-1627498771 resumable_relative_path admin/
504
505 Upload job is finalizing, please wait...
506 Upload job complete.
507 All files uploaded successfully.
508 {{/code}}
509
510 In case the specified destination directory does not exist yet, you will be prompted to confirm its creation by entering “y”, or to abort the command by entering “N”:
511
512 {{code language="none"}}
513 Target folder does not exist. Would you like to create a new folder? [y/N]: y
514 {{/code}}
515
516 In order to upload a file (or folder) to a destination directory in a Project's Core, core must be specified as corresponding zone {{code}}(-z core){{/code}} and you will be required to provide a message to that should describe the files` content and why you are uploading it directly into the Core {{code}}(-m "sample message"){{/code}}. Moreover, you will be reminded about the Terms of Use of the Platform and additional considerations regarding the direct upload of data into the Core zone, and are prompted to confirm your intention to proceed with the upload by entering “y” or “Yes”:
517
518 {{code language="none"}}
519 $ pilotcli file upload ./sample_folder/sample_file2 -p cli/admin -z core -m "sample message"
520 You are about to transfer data directly to the Core! In accordance with the Terms of Use, please confirm that you
521 have made your best efforts to pseudonymize or anonymize the data and that you have the legal authority to transfer and make this
522 data available for dissemination and use within the Platform.To review the Terms of Use, please the Portal. If you
523 need to process the data to remove sensitive identifiers, please cancel this transfer and upload the data to the Green Room to
524 perform these actions.
525 To cancel this transfer, enter [n/No]
526 To confirm and proceed with the data transfer, enter [y/Yes]
527 [y/N]: y
528 Starting upload of: ./sample_folder/sample_file2
529 Pre-upload complete.
530 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: fa807a0f-ceb0-4fff-9c9e-563bdcbb309f-1627498851 resumable_relative_path admin/
531
532 Upload job is finalizing, please wait...
533 Upload job complete.
534 All files uploaded successfully.
535 {{/code}}
536
537 Upload a folder to Project Green Room
538
539 {{code language="none"}}
540 $ pilotcli file upload ./sample_folder/example_folder -p cli/admin
541 Starting upload of: ./sample_folder/example_folder
542 Current version does not support folder tagging, any selected tags will be ignored
543 Pre-upload complete.
544 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 4f0eee2e-70f4-4b03-9e72-040544ec7d3a-1627498907 resumable_relative_path admin/example_folder/
545
546 Starting finalization...
547 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 41ec70e2-0859-4aba-9009-06e700c7c79e-1627498907 resumable_relative_path admin/example_folder/
548
549 Starting finalization...
550 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 28454c51-ec9a-4dc5-ab3c-0f7e80e3050d-1627498907 resumable_relative_path admin/example_folder/
551
552 Upload job is finalizing, please wait...
553 Upload job complete.
554 All files uploaded successfully.
555 {{/code}}
556
557 Upload file to existing folder in Green Room
558
559 * Specify a target folder to upload files by inputting folder name after the Project code separated by the '/'.
560
561 {{code language="none"}}
562 $ pilotcli file upload ./sample_folder/sample_file5 -p cli/admin/example_folder
563 Starting upload of: ./sample_folder/sample_file5
564 Pre-upload complete.
565 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: 7ef58005-d408-4ad9-a741-fa938f58e67f-1627498977 resumable_relative_path admin/example_folder/
566
567 Upload job is finalizing, please wait...
568 Upload job complete.
569 All files uploaded successfully.
570 {{/code}}
571
572 Upload folder as zip to a specified directory of a Project's Green Room by adding the ~-~-zip option.
573
574 * The specified folder will then be compressed as a zip file and subsequently be uploaded to the specified destination. Once the upload is completed, the compressed zip file will be removed from the directory. Note: pilotcli must have the permission to create and delete zip files in the corresponding directory of your local machine:
575
576 {{code language="none"}}
577 $ pilotcli file upload ./sample_folder/example_folder -p cli/admin --zip
578 Starting upload of: ./sample_folder/example_folder
579 Started zipping files.
580 Pre-upload complete.
581 uploader: admin container_id: 18061 total_size: 3595 total_chunks: 1 uniq_identifier: 61a729ce-cdd6-4a87-82b5-3f8ea63e6c4a-1627499012 resumable_relative_path admin/
582
583 Upload job is finalizing, please wait...
584 Upload job complete.
585 All files uploaded successfully.
586 {{/code}}
587
588 Upload a file with tags and/or attributes.
589
590 * When entering the upload command, the user can add a maximum of ten tags to the file. For example, in order to add “sample_tag1” and “sample_tag2” to your file, simply add {{code}}-t sample_tag1{{/code}} {{code}}-t sample_tag2{{/code}} to the command (see example below using {{code}}tag1{{/code}} and {{code}}tag2{{/code}} as tags).
591 * In addition, you can also use attribute templates for a more enriched annotation of your files. Please note, that the corresponding attribute template file needs to be edited & saved prior to uploading. To get the blank template, see the **attribute-export** instructions above for more details. To attach a attribute JSON-file to your file upon uploading, use {{code}}-a cli_Manifest1_template.json{{/code}}, where “{{code}}cli_Manifest1_template.json{{/code}}" points to the location of the attributes JSON-file on your local machine.
592 * Upon uploading, pilotcli will also validate the content of the attached attribute JSON-file. Whenever a template is invalid, including missing non-optional fields, non-existing choice in multiple choice attribute, exceeding length text field and so on, pilotcli will raise error to user and user must update the attribute template and upload again:
593
594 {{code language="none"}}
595 $ pilotcli file upload ./sample_folder/sample_file6 -p cli/admin -t tag1 -t tag2 -a cli_Manifest1_template.json
596 File attribute validation passed: True
597 Starting upload of: ./sample_folder/sample_file6
598 Pre-upload complete.
599 uploader: admin container_id: 18061 total_size: 1048576 total_chunks: 1 uniq_identifier: bc47ec2a-1a4e-4f71-a37a-7ce6473e415a-1627499273 resumable_relative_path admin/
600
601 Upload job is finalizing, please wait...
602 Upload job complete.
603 Attribute attached
604 All files uploaded successfully.
605 {{/code}}
606
607 Upload a file with specified number of threads, using the {{code}}-td <thread_number>{{/code}} option.
608
609 * By default, the number of threads is 1. There will no UI difference when using threads. The maximal number of threads depends on your local machine and the recommend thread number is 3:
610
611 {{code language="none"}}
612 $ pilotcli file upload -p indoctestproject/admin/test20230306_3 /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img -td 3
613 Target folder does not exist. Would you like to create a new folder? [y/N]: y
614 Starting upload of: /home/color/indoc/script/large_folder_test_data/gentoo_root_100M.img
615 Pre-upload complete.
616 Uploading gentoo_root_100M.img , resumable_id: c68752372b42c85d, job_id: 56de3632-58df-445c-bcb9-bddfe6be5bf7: |██████████████████████████████ 100% 00:00
617 Upload Time: 79.77s for 1 files
618 All uploading jobs have finished.
619 {{/code}}
620
621 (10) upload with customized manifest name
622
623 When you upload a file using pilotcli, a JSON file with the details of your specific upload job will be created automatically and allows you to resume the upload at a later time, in case the process is aborted due to any reason. By default, this JSON file will be created as “manifest.json” in the current working directory (see //Command: **resume**// below for more details on how to resume the file upload). However, you can use the {{code}}--output-path{{/code}} option to specify the filepath and, thus, the filename of this JSON file:
624
625 {{code language="none"}}
626 $ pilotcli file upload -p <path> <file> --output-path test_manifest.json
627 {{/code}}
628
629 == Command: **resume** ==
630
631 * **Options**: -td ~-~-thread, -r ~-~-resumable-manifest
632 * **Usage**: Resume unfinished file uploading based on the manifest file.
633 * (((
634 **Prerequisite**:
635
636 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
637 * Project must be accessible to the user, otherwise an error is returned: "Project Code not found in list. Please verify and try again."
638 * The resumable manifest JSON file is required, which is generated upon executing the {{code}}pilotcli file upload{{/code}} command.
639 )))
640
641 === Example: ===
642
643 For instance upon disconnecting from the internet, or by accidentally closing the terminal, a currently running uploading process can be aborted. You can use the {{code}}pilotcli file resume{{/code}} command in order to resume the uploading process right where you left off. Simply point to the manifest JSON file that was created upon executing the original upload command. Please note that the source files still have to be in the same location, otherwise resuming the uploading process will fail.
644
645 {{code language="none"}}
646 $ pilotcli file resume -r ./manifest.json
647 Resume the upload for 10 files.
648 Resumable upload check complete.
649 Uploading aggregated.py: |██████████████████████████████ 100% 00:00
650 Uploading __init__.py: |██████████████████████████████ 100% 00:00
651 Uploading service_meta_class.py: |██████████████████████████████ 100% 00:00
652 Uploading upload_form.py: |██████████████████████████████ 100% 00:00
653 Uploading convert_type.py: |██████████████████████████████ 100% 00:00
654 Uploading singleton.py: |██████████████████████████████ 100% 00:00
655 Uploading __init__.py: |██████████████████████████████ 100% 00:00
656 Uploading base_library.zip: |██████████████████████████████ 100% 00:00 0% ?
657 Uploading EXE-00.toc: |██████████████████████████████ 100% 00:00
658 Uploading xref-pilotcli_test20230227_3.html: |██████████████████████████████ 100% 00:00
659 Upload Time: 11.13s for 10 files
660 {{/code}}
661
662 == Command: **sync** ==
663
664 * **Usage**: Download a file or a folder from a Project. If pilotcli is used outside of the Platform, e.g. the users local machine, users can only download from Project’s Core. When pilotcli is used within the Platform, e.g. from one of the workspace tools, users will also be able to download from the Green Room of a Project as well.
665 * (((
666 **Options**: -z ~-~-zone, -i ~-~-geid, ~-~-zip
667
668 * To look up a file’s geid, or unique file ID, open the Project File Explorer in the Portal and inspect the File Properties for the ID.
669 )))
670 * (((
671 **Prerequisite**:
672
673 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
674 * The specified file or folder must exist, otherwise an error is returned: “File does not exist or source file provided is invalid: folder_name/file_name.”
675 * If the user has no access to the specified file or folder on the Portal, an error is returned: “Permission denied. Please verify that your role in the Project has permission to perform this action.”
676 )))
677
678 {{code language="none"}}
679 $ pilotcli file sync --help
680 Usage: pilotcli file sync [OPTIONS] [PATHS]... OUTPUT_PATH
681
682 Download files/folders from a given Project/folder/file in core zone.
683
684 Options:
685 -z, --zone TEXT Target Zone (i.e., core/greenroom)
686 --zip Download files as a zip.
687 -i, --geid Enable downloading by geid.
688 --help Show this message and exit.
689 {{/code}}
690
691 === Examples ===
692
693 Download a file or a folder from the Core zone of a Project:
694
695 {{code language="none"}}
696 $ pilotcli file sync cli/admin/sample_file2 -z core ~/Downloads/
697 Preparing status: READY_FOR_DOWNLOADING
698 start downloading...
699 Downloading sample_file2 |██████████████████████████████ 100% 00:00
700 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2
701 {{/code}}
702
703 Download multiple files or folders as a compressed zip by simply providing all source files or folders and adding the {{code}}--zip{{/code}} option:
704
705 {{code language="none"}}
706 $ pilotcli file sync cli/admin/sample_file2 cli/admin/sample_file7 -z core ~/Downloads/ --zip
707 Preparing downloading
708 Preparing status: READY_FOR_DOWNLOADING
709 start downloading...
710 Downloading cli_1632340658.768384.zip |██████████████████████████████ 100% 00:00
711 File has been downloaded successfully and saved to: /home/uname/Downloads/cli_1632340658.768384.zip
712 {{/code}}
713
714 Download files or folders by their geid, instead of their location. To enable this, add {{code}}-i{{/code}} or {{code}}--geid{{/code}} to the command:
715
716 {{code language="none"}}
717 $ pilotcli file sync -i 6d89afd4-f86e-4df7-b6c5-01a9af4e828e-1627499957 7e8e2bac-97c6-4738-bc08-12139c9ba225-1627498854 ~/Downloads/
718 Preparing status: READY_FOR_DOWNLOADING
719 start downloading...
720 Downloading sample_file7 |██████████████████████████████ 100% 00:00
721 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
722 Preparing status: READY_FOR_DOWNLOADING
723 start downloading...
724 Downloading sample_file2 |██████████████████████████████ 100% 00:00
725 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2
726 {{/code}}
727
728 Download a file or folder from the Green Room of a Project. This will only be possible if pilotcli is used within the Platform, e.g. in one of the Workspace tools. It works the same way as downloading from Core, including all the additional options like zipping, by either adding {{code}}-z greenroom{{/code}} or by omitting the {{code}}-z{{/code}} option entirely, as “greenroom” is the default value:
729
730 {{code language="none"}}
731 $ pilotcli file sync cli/admin/sample_file2 cli/admin/sample_file7 ~/Downloads
732 Preparing status: READY_FOR_DOWNLOADING
733 /home/uname/Downloads/sample_file2 already exist, file will be saved as /home/uname/Downloads/sample_file2 (1)
734 start downloading...
735 Downloading sample_file2 (1) |██████████████████████████████ 100% 00:00
736 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file2 (1)
737 Preparing status: READY_FOR_DOWNLOADING
738 start downloading...
739 Downloading sample_file7 |██████████████████████████████ 100% 00:00
740 File has been downloaded successfully and saved to: /home/uname/Downloads/sample_file7
741 {{/code}}
742
743 = Dataset Commands: pilotcli **dataset** =
744
745 == Help ==
746
747 {{code language="none"}}
748 $ pilotcli dataset --help
749 Usage: pilotcli dataset [OPTIONS] COMMAND [ARGS]...
750
751 Options:
752 --help Show this message and exit.
753
754 Commands:
755 download Download a dataset or a particular version of a dataset.
756 list List datasets belonging to logged in user.
757 show-detail Show details of a dataset.
758 {{/code}}
759
760 == Command: **list** ==
761
762 * **Usage**: List all datasets for the logged in user
763 * (((
764 **Prerequisite**:
765
766 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
767 )))
768
769 {{code language="none"}}
770 $ pilotcli dataset list --help
771 Usage: pilotcli dataset list [OPTIONS]
772
773 List datasets belonging to logged in user.
774
775 Options:
776 --page INTEGER The page to be listed [default: 0]
777 --page-size INTEGER number of objects per page [default: 10]
778 -d, --detached whether run in detached mode
779 --help Show this message and exit.
780 {{/code}}
781
782 === Examples ===
783
784 List all datasets that are accessible for the user:
785
786 {{code language="none"}}
787 $ pilotcli dataset list
788 Dataset Title Dataset Code
789 ---------------------------------------------------------------------------
790 dataset2 aug24202102
791 dataset-01 aug242021
792 Page: 0, Number of datasets: 2
793 {{/code}}
794
795 == Command: **show-detail** ==
796
797 * **Usage**: Show the detailed information of a dataset.
798 * (((
799 **Prerequisite**:
800
801 * User login is required, otherwise an error is returned: "The current login session is invalid. Please login to continue."
802 )))
803
804 {{code language="none"}}
805 $ pilotcli dataset show-detail --help
806 Usage: pilotcli dataset show-detail [OPTIONS] CODE
807
808 Show details of a dataset.
809
810 Options:
811 --page INTEGER The page to be listed [default: 0]
812 --page-size INTEGER number of objects per page [default: 10]
813 -d, --detached whether run in detached mode
814 --help Show this message and exit.
815 {{/code}}
816
817 === Examples ===
818
819 Show the details of the specified Dataset by providing the unique Dataset Code - “aug242021” in this case:
820
821 {{code language="none"}}
822 $ pilotcli dataset show-detail aug242021
823 --------------------------------------------------------------------------------
824 | Title | dataset-01 |
825 --------------------------------------------------------------------------------
826 | Code | aug242021 |
827 --------------------------------------------------------------------------------
828 | Authors | uname, cli |
829 --------------------------------------------------------------------------------
830 | Type | GENERAL |
831 --------------------------------------------------------------------------------
832 | Modality | neuroimaging, microscopy, histological approach |
833 --------------------------------------------------------------------------------
834 | Collection_method | import, test, upload |
835 --------------------------------------------------------------------------------
836 | Tags | tag1, tag2, tag3 |
837 --------------------------------------------------------------------------------
838 | Versions | 1.0, 1.1, 1.2 |
839 --------------------------------------------------------------------------------
840 {{/code}}
841
842 == Command: **download** ==
843
844 * **Usage**: Download a particular version of a dataset.
845 * **Options**: -v ~-~-version
846 * (((
847 **Prerequisite**:
848
849 * User login is required, otherwise error is returned: "The current login session is invalid. Please login to continue."
850 )))
851
852 {{code language="none"}}
853 $ pilotcli dataset download --help
854 Usage: pilotcli dataset download [OPTIONS] [CODE]... OUTPUT_PATH
855
856 Download a dataset or a particular version of a dataset.
857
858 Options:
859 -v, --version TEXT Download a particular version of a dataset.
860 --help Show this message and exit.
861 {{/code}}
862
863
864 === Examples ===
865
866 Download a Dataset with Code “aug242021”:
867
868 {{code language="none"}}
869 $ pilotcli dataset download aug242021 ~/Downloads/
870 Pre downloading dataset
871 Preparing status: READY_FOR_DOWNLOADING
872 start downloading...
873 Downloading aug242021_1632342868.882398.zip |██████████████████████████████ 100% 00:00
874 File has been downloaded successfully and saved to: /home/uname/Downloads/aug242021_163234
875 2868.882398.zip
876 {{/code}}
877
878 Download the version 1.1 of a Dataset with Code “aug242021”:
879
880 {{code language="none"}}
881 $ pilotcli dataset download aug242021 ~/Downloads/ -v 1.1
882 Current dataset version: 1.1
883 Pre downloading dataset
884 start downloading...
885 Downloading aug242021_2021-08-24 11:10:00.042277.zip |██████████████████████████████ 100% 00:00
886 File has been downloaded successfully and saved to: /home/uname/Downloads/aug242021_2021-08-24 11:10:00.042277.zip
887 {{/code}}
888
889
890
891 ----
892
893 Copyright © 2023 [[Indoc Research>>url:https://www.indocresearch.org/]].
894
895 HealthDataCloud is powered by Pilot technology, a product of [[Indoc Research>>url:https://www.indocresearch.org/]].