Version 1.1 by Susan Evans on 2023/07/10 21:41
Show last authors
1 = Data Stewardship =
2
3 Users are reminded to abide by the Platform Terms of Use and any Project-specific restrictions when using the Portal to access data.
4
5 = Uploading Files =
6
7 All files uploaded to the platform land in the //Green Room//. In this isolated storage area, the only people who can view and download the files you have uploaded are you and the Project Administrator(s). Project Administrators can verify that the data are compliant with the Project’s data management plan before being copied to the //Core //where all other Project members (except Project Contributors, see //Managing HDC Projects > Project Roles//) can view and access the file.
8
9 [[image:Project File Upload v2.0.10 2023-01-17.png||height="8%" width="50%"]]
10
11 == Prerequisites ==
12
13 * Membership in the Project (any role) or Platform Administrator
14 * Stable internet connection
15
16 == Steps ==
17
18 1. (((
19 Navigate your Project’s File Explorer, click **Green Room**, and select a destination folder.
20
21 1. Unless otherwise specified, files will be uploaded into your top-level personal Green Room folder denoted by your username.
22 1. To upload the file to a sub-folder, click an existing sub-folder to open it or click **+ New Folder** to create a folder then click to open it.
23 1. If you’re a Project Administrator, you can select and upload to any folder in the Green Room.
24 )))
25 1. (((
26 Click **Upload**.
27
28 1. Check the Project field and destination folder name to confirm you’re in the correct Project and destination.
29 1. Click **Select, **choose **Select Files** or **Select Folder**, then choose files or folders from your local computer.
30 1. Review the file list. To remove a selected file, hover on the file name and click the **Trash** icon.
31 )))
32 1. (((
33 Add desired optional elements:
34
35 1. //Tags: //Enter one or more **Tags **(keywords) to attach to all the selected files. Note: adding tags during upload is not available for folders. To add Tags to a folder, upload the folder first, then use one of the alternate methods described in the section //Adding and Modifying File Tags.//
36 1. //Attributes//: If Attributes have been configured by the Project Administrator, a dropdown menu will be visible and you’ll be prompted to enter file attributes. See //Adding and Modifying File Attributes// for more information.
37 )))
38 1. Click **Upload** to initiate the file upload, or **Cancel** to exit without uploading.
39 1. The progress of file upload can be checked by clicking the the **File Status icon** from the Main Menu. See //Checking File Status// in this article for more information.
40 1. When the file or folder is uploaded it is stored in your Green Room folder denoted by your username.
41
42 == Considerations ==
43
44 * //Large file uploads: //The portal supports upload of files up to approximately 10 GB at a time. Upload performance may be impacted by your internet connection. If you need to upload larger files, consider using the Command Line Tool. If you need to upload an extremely large number of files (e.g., many tens of thousands) in one session, it is recommended to use an alternative upload method such as the Command Line Tool or a custom direct integration (contact Support for more information).
45 * //Compressed files~://** **The portal supports upload of individual files or entire folders, including compressed folders (e.g., .zip, .tar, .rar archives). Compressed files can be previewed using the File Preview tool (see //Previewing Files// in this article), and extracted or decompressed using one of the Workspace tools.
46 * //Moving and renaming files and folders~://** **Once a file or folder is uploaded it cannot be moved or renamed in the Green Room File Explorer. Instead, you can re-upload the file or folder with a different name or destination, or copy the file or folder to the Core and use the Workspace tools to organize the Project files.
47 * //Uploading duplicate files: //Duplicate files cannot be uploaded to the same destination folder. If you attempt to upload a file that already exists in the destination folder with the same filename, an error message is returned. Instead, you can either upload the file with a different filename, or create a new Green Room folder and upload the file to the new folder.
48
49 = Checking File Status =
50
51 You can check the progress and completion status (success or failure) of Project files uploaded, downloaded, and deleted during the current login session using the File Status panel. Click the **File Status icon** in the Main Menu to open the File Status panel then browse the activity tabs (**In Progress**, **Uploaded**, **Downloaded**, **Trash Bin**).
52
53 [[image:Project File Status v2.0.10 2023-01-16.png||height="20%" width="50%"]]
54
55 To view the complete history of all Project files uploaded, downloaded and deleted, not just activities in the current login session, use the File Stream Advanced Search.
56
57 [[image:Project Recent File Stream Advanced Search v2.2.6 2023-04-14.png||height="21%" width="50%"]]
58
59 1. Navigate to the Project Canvas and locate the Recent File Stream panel.
60 1. Click **Advanced Search** to open the File Stream Advanced Search.
61 1. Select your search criteria and click **Search**. Results are sorted by date with the most recent on top. Only successful file actions are captured - file uploads and downloads that were attempted but failed are not displayed in the history.
62
63 == Resuming Failed Uploads ==
64
65 If a file fails to upload from the Portal, a pop-up message alerts you and the Project **File Status** panel returns a “Failed” status in the **In Progress **activity tab. Try checking your internet connection and refreshing your login session, and consider the size of the file/folder being uploaded. If the file exceeds the recommended upload capacity of the Portal, consider using the Command Line Interface.
66
67 The Portal provides the ability to resume a file upload already in progress so you don’t have to restart the upload from the beginning.
68
69 [[image:Resume File Upload v2.2.6 2023-04-13 1815.png||height="19%" width="50%"]]
70
71 To resume a file upload after a failure:
72
73 1. Re-establish your upload environment by checking that your internet connection is working, refreshing the File Explorer page, and/or logging back into the Portal.
74 1. Open the **File Status panel** and check the **In Progress** tab for the failed file. Click **Re-upload file** and select the same file from your local source folder.
75 1. The upload resumes as shown by the progress bar and successful completion is indicated in the **File Status panel**.
76
77 = Viewing File Properties =
78
79 The File Properties panel displays detailed metadata about Project files to help find them and facilitate their appropriate use.
80
81 To view the File Properties:
82
83 1. Select a file in the File Explorer.
84 1. Click the **Action icon** (three dots) beside the file.
85 1. Select **Properties** to open the Properties panel.
86
87 Properties include:
88
89 * File metadata such as filename, user who uploaded the file, date and time uploaded, file size, and ID – an object identifier that uniquely identifies the file in the platform.
90 * User-added metadata such as Tags and Annotations
91 * System Tags, for example, copied-to-core to signify a file has been copied to the Core.
92
93 For a detailed description of other information available in the File Properties menu, see sections //Adding and Modifying File Tags//, //Adding and Modifying File Attributes,// and //Data Lineage Graph.//
94
95 = Previewing Files =
96
97 The File Previewer can be used to check the inner folder structure of uploaded file archives. Supported types include:
98
99 * Zip: .zip
100 * Tar: .tar, .tgz, .tbz, .txz, .tar.gz,.tar.br, .tar.bz2, .tar.xz, .tar.zst.
101 * 7zip: .7z
102 * RAR: .rar
103
104 [[image:Projects File Previewer v2.2.3 2023-05-03.png||height="29%" width="50%"]]
105
106 1. Locate the file in the File Explorer.
107 1. Click the **Action icon** (three dots) beside the file and select **Preview** (only visible for archive file listed above).
108 1. In the File Previewer, click to expand or collapse the folders and view the inner folders.
109 1. Click **OK **to close the File Previewer.
110
111 To unzip the file, use one of the available workspace tools such as a Jupyter notebook in **JupyterHub **(see //Using HDC Workspace Tools//).
112
113 = Adding and Modifying File Tags =
114
115 File Tags are keywords assigned to files and/or folders to help describe them and allow them to be found. Two types of tags can be associated with Project files:
116
117 * //System Tags//** **are applied automatically during a platform supported action. For example, the System Tag copied-to-core is attached to a file when it is copied from the Green Room to Core. System Tags cannot be changed by users.
118 * //Customized Tags// are applied by users during file upload, and can be added or modified after upload.
119
120 System and Customized Tags are visible in a file or folder’s General Properties. Additionally, you can search all Project Files for specific Customized Tags by using the Project Search tool (see //Projects Interface > Search //in the article //HDC Portal Navigation//).
121
122 Customized Tags can be added and/or modified using one of the following methods:
123
124 == Method 1: during File upload ==
125
126 //Not available for Folder upload.//
127
128 1. Begin a File upload (see //Uploading Files//)
129 1. In the Tags window, enter one or more Tags.
130 1. Complete the remaining fields and click **Submit**.
131
132 == Method 2: from the File Properties panel ==
133
134 Recommended for quick tagging of individual files or folders in the Green Room or Core.
135
136 1. Select a file or folder in the File Explorer, click the **Action icon **(three dots) beside the file, then select **Properties** to open the File Properties panel.
137 1. (((
138 In the General section, Click **Edit Tags **to add or delete Customized Tags.
139
140 1. NOTE: When using this method, tags attached to folders don’t propagate to the files inside. To add tags in bulk to the folder contents, use Method 3.
141 )))
142
143 == Method 3: from the Project File Explorer ==
144
145 Recommended for bulk tagging of files and folders.
146
147 [[image:Project Customized Tags v2.0.10 2023-01-17.png||height="8%" width="50%"]]
148
149 1. Navigate to the Project File Explorer (Green Room or Core).
150 1. Select one or more files and/or folders by clicking the checkbox beside each one, then click **Add/Remove Tags** in File Explorer menu.
151 1. (((
152 In the popup window, select one of the following options from the dropdown:
153
154 1. **+ Add Tags** to enter one or more Customized Tags.
155 1. **- Remove Tags** to delete an existing Customized Tag.
156 1. Enter the name(s) of the tags to be added or removed.
157 1. If you selected a folder, the tag(s) will be added or removed from all files inside the folder, but not to the actual folder(s). If you also wish to add or remove the tag on the folder as well, click the checkbox **Add tag to all the folders**.
158 )))
159 1. Click **Add Tags** to complete the action and close the menu.
160
161 == Considerations ==
162
163 * Tags can contain 1 - 32 characters consisting of lowercase letters, numbers and/or hyphens.
164 * When typing multiple tags in the entry line, be sure to hit Enter after each one to save it.
165 * A maximum of 10 tags are allowed per file or folder.
166 * In the Green Room, Customized Tags can only be added or edited by the Project Member who uploaded the file, or the Project Administrator.  In the Core, Collaborators and Project Administrators can edit Customized tags of files in any folder.
167 * Tags can also be attached to Projects and Datasets.
168
169 = Adding and Modifying File Attributes =
170
171 It may be important to annotate your uploaded Project files with descriptive metadata that conform with your own defined categories, formats, or values to make the files more readily searchable and machine readable. For example, when uploading a file containing test scores you wish to specify metadata on the research site that performed the test, or on a DICOM imaging file you wish to specify the MRI scanner manufacturer. File Attributes enable this enriched metadata capture on files.
172
173 Project Administrators create custom File Attributes for the Project using the File Attributes template (see //Managing File Attributes Templates //in the article //Managing HDC Projects//). If File Attributes have been configured for your Project, the attributes are available for all Project Members to annotate files using one of two methods:
174
175 == Method 1: during File upload ==
176
177 Use this method to add Project-specific metadata annotations to your files on upload.
178
179 [[image:Projects File Attributes v2.1.2 2023-01-18.png||height="58%" width="50%"]]
180
181 1. Begin a File upload (see //Uploading Files//)
182 1. Select a File Attribute template from the dropdown menu (only visible if the Project Administrator has configured File Attributes) and enter the requested information.  File Attributes can be either required or optional on upload depending on how the Project Administrator has configured them. 
183 Note: If multiple File Attribute templates are available, a file can only use one template.
184 1. Complete the remaining fields and click **Submit**.
185
186 == Method 2: from the Project File Explorer ==
187
188 [[image:Projects File Attributes from File Explorer v2.1.2 2023-01-18.png||height="8%" width="50%"]]
189
190 1. Navigate to the Project File Explorer Green Room or Core.
191 1. Select one or more files and/or folders by clicking the checkbox beside each one, then click **Add Attributes** in File Explorer menu.
192 1. Select an Attributes template, complete the information, and click Submit.
193
194 File Attributes added to a file or folder are visible in its Properties menu in the File Explorer. Additionally, you can search all Project Files for specific File Attributes by using the Project Search tool (see //Projects Interface > Search //in the article //HDC Portal Navigation//).
195
196 = Requesting File Copy from the Green Room to the Core =
197
198 The Core is the storage area where Project members can access each others’ Project files, create file collections, and use the workspace tools to open files and perform analyses.
199
200 Before a file is copied from the Green Room to the Core, it must be pseudonymized to the extent necessary to fulfil the Project objectives and agreements. Only Project Administrators can copy files from the Green Room to the Core (see //Managing HDC Projects > Files Copy from the Green Room//). However, Project members can mark files and/or folders in their Green Room folder and submit a request to the Project Administrator to copy the files to the Core.
201
202 To submit a request to copy data from the Green Room to the Core (requires Collaborator role),
203
204 [[image:Projects Request Copy to Core v2.1.2 2023-01-18.png||height="9%" width="50%"]]
205
206 1. Navigate to the Project File Explorer Green Room
207 1. Select one or more files and/or folders to be copied by clicking the checkbox beside each one. The Request to Core option becomes visible in the File Explorer menu.
208 1. Click **Request to Core**.
209 1. Select a destination folder in the Core.  Destination folders are denoted by Project member usernames (if unsure, select your own username folder).
210 1. If desired, enter any remarks for the Project Administrator in the Request Notes field.
211 1. Click **Confirm**.
212 1. The Project Administrator will receive an email notification with your request details and will need to log into the Portal to approve the request. After approval, your requested files will be visible in the Core destination folder.
213
214 To view your submitted and approved requests:
215
216 [[image:Projects Requests Left Menu v2.3.3 2023-04-26.png||height="27%" width="50%"]]
217
218 1. Click the Requests icon from the Project left menu bar.
219 1. Click **New** to view your new requests that have not been fulfilled, or **Completed **to view requests that the Project Administrator has fulfilled. Requests are identified by the username and the date/time stamp when the request was submitted.
220
221 = Viewing Data Lineage =
222
223 When you copy data from one location to another, execute a pipeline, or perform other actions on a file, the platform data lineage service captures the points in the file’s journey and displays them in the File Properties Data Lineage graph.
224
225 To access a file’s Data Lineage Graph,
226
227 1. Locate the file in the Green Room or Core File Explorer.
228 1. Click the **Action icon** (three dots) beside the file and open the Data Lineage Graph.
229 1. Hover over the graph icons to explore the points in the file’s lineage.
230 1. Click **See More** to explore the graph and use the dropdown menu to switch views between All Nodes, or just Upstream or Downstream. The lineage nodes (icons) are explained below.
231
232 == Upstream and Downstream Lineage ==
233
234 * //Nodes //represent the state of a file resting in storage.
235 * //Pipelines// describe actions that change the state of a file.
236 * //Upstream lineage// displays the points in a file's journey up to its current state. For example, a file newly uploaded to the Green Room has only one upstream point, the Green Room File node, representing the point when the file was introduced the the platform.
237 * //Downstream lineage// displays points in a file’s journey from a specific state forward. For example, when viewing the lineage of a file stored in the Green Room that was later copied to the Core, the downstream lineage displays the Green Room File node, the copy Pipeline, and the Core File node.
238
239 [[image:Project File Lineage v2.1.3 2023-01-19.png||height="25%" width="50%"]]
240
241 = Downloading Files =
242
243 ​Files can be downloaded individually or in bulk directly from the File Explorer.
244
245 [[image:Projects File Download v2.1.3 2023-01-19.png||height="9%" width="50%"]]
246
247 1. Before downloading a file, check your browser settings and ensure that pop-up windows are enabled.
248 1. (((
249 Navigate to the Project File Explorer Green Room or Core.
250
251 1. For a single file download, select a file, click the Action icon (three dots) beside the file, then select **Download**.
252 1. For bulk file download, select one or more files and/or folders by clicking the checkbox beside each one (or use the select all checkbox), then click **Download** in File Explorer menu//. //Bulk file downloads are packaged into a .zip file.
253 )))
254 1. Open the File Status panel to check the status of your downloads.
255 1. Open the Downloads folder on your local computer to find the downloaded file(s).
256
257 == Considerations ==
258
259 * Download permission is role-based - any Project member can download files that they uploaded in the Green Room, Collaborators can download files from any folder in the Core, and Project Administrators can download any file from any location.
260 * Download performance may be impacted by your internet connection.
261 * Ensure pop-ups are enabled in your browser.
262
263 = Deleting Files =
264
265 Project members can delete files that are no longer needed in the Green Room or Core. When you delete a file, it is moved to a Trash Bin where it can be permanently deleted later, or restored in case of accidental deletion.
266
267 [[image:Projects File Delete v2.1.3 2023-01-19.png||height="9%" width="50%"]]
268
269 1. Navigate to the Project File Explorer Green Room or Core.
270 1. Select one or more files and/or folders by clicking the checkbox beside each one (or use the select all checkbox), then click **Delete **in File Explorer menu//.//
271 1. A popup window prompts you to confirm the file deletion. Click **OK** or cancel to exit without deleting.
272 1. The file is moved to the Trash Bin.
273
274 Open the File Status panel to check the status of the single file deletion, or open the Trash Bin in the left menu of the File Explorer to view all files in your Trash Bin. Files can be permanently deleted by a Platform Administrator.
275
276 == Considerations ==
277
278 * Delete permission is role-based - any Project member can delete files that they uploaded in the Green Room, Collaborators can delete files from any folder in the Core, and Project Administrators can delete any file from any location.
279 * Deleted files are moved to a Trash Bin but are not permanently removed from the platform until requested.
280 * Files that have been copied to a Dataset will remain in the Dataset even if the original version has been deleted from the Core.
281
282 = Creating File Collections =
283
284 When working with Project files in the Core, it can be useful to organize them in temporary or permanent views to help find them more easily. A Collection gives Project members the flexibility to create their own virtual groups of Project files from any number of Core folder locations. Similar to a music playlist, a Collection establishes a reference to the file(s) without moving or copying the actual files themselves. Adding a Collection to your Favourites gives you quick access to it from your My Space page.
285
286 You can start by creating an empty Collection before adding any files to it, or create a Collection and add files at the same time.
287
288 == Create an empty Collection ==
289
290 [[image:Projects Core Create Collection v2.1.5 2023-01-20 1130.png||height="33%" width="50%"]]
291
292 1. Navigate to the Project’s Core File Explorer .
293 1. From the Left navigation menu, click **Create Collection**.
294 1. Enter the Collection name and click **Save**.
295 1. The new Collection will appear in the left File Explorer navigation menu.
296
297 == Add files to a Collection ==
298
299 [[image:Projects Core Add to Collection v2.1.3 2023-01-19.png||height="10%" width="50%"]]
300
301 1. Navigate to the Project’s Core File Explorer .
302 1. Select one or more files and/or folders by clicking the checkbox beside each one (or use the select all checkbox), then click **Add to Collection **in File Explorer menu//.//
303 1. (((
304 Choose one of following options:
305
306 1. Select an existing Collection and click **Add to Existing Collection**.
307 1. Click **+ Create New Collection**, enter the name for the Collection, and click Create New Collection. The Collection is created and the selected files added to it in one step. The new Collection will appear in the left File Explorer navigation menu.
308 )))
309
310 == Remove files from a Collection ==
311
312
313 1. Click the name of the Collection in the left File Explorer navigation menu to open it.
314 1. Click the checkbox beside the file(s) to be removed, then click **Remove from Collection** in the File Explorer menu.
315
316 Removing a file from a Collection does not remove the file from the Core folder where it is stored.
317
318 == Deleting or renaming a Collection ==
319
320 1. Click the name of the Collection in the left File Explorer navigation menu to open it.
321 1. Click either the Delete or Rename tool in the File Explorer menu.
322
323 Deleting a Collection does not remove the files from their Core storage folders.
324
325 == Considerations ==
326
327 * Collections can only be created with Project files from the Core.
328 * Collection names cannot exceed 20 characters.
329 * A file can be added to one or more Collection(s).
330 * Adding or removing files in a Collection does not change the file's storage location or data lineage.
331
332 = Adding Files and Collections to Favourites =
333
334 File and Collections from your Project can be “starred” to add them as Favourites and make them quickly accessible in the Favourites panel in the Dashboard
335
336 To add a Collection or File to your Favourites, locate the item in the Project File Explorer and click the **star icon**. To remove from Favourites, click the star again.
337
338
339