Changes for page Working with HDC Datasets

Last modified by Dennis Segebarth on 2024/10/02 18:25
From version 1.2
edited by Susan Evans
on 2023/07/11 14:43
Change comment: (Autosaved)
To version 23.1
edited by Dennis Segebarth
on 2024/10/02 18:25
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.sgevans
1 +XWiki.dsegebarth
Content
... ... @@ -2,7 +2,7 @@
2 2  {{toc depth="2"/}}
3 3  {{/box}}
4 4  
5 -Datasets extend Projects with comprehensive data management functionalities. Importantly, each Project can have multiple Datasets associated with it, but each Dataset can only be populated with data from a single Project.
5 +Datasets extend Projects with comprehensive data management functionalities. Importantly, each Project can have multiple Datasets associated with it, but each Dataset can only be populated with data from a single Project. Each Dataset can be viewed, accessed, and interacted with by the user who created it, as well as all users with the Project Administrator role in the Project the Dataset is associated with.
6 6  
7 7  In brief, Datasets are collections of related Project files, folders, and detailed metadata. The Dataset feature allows users to organize Project files, enrich with annotations, validate data structures, and publish controlled versions for sharing.
8 8  
... ... @@ -20,42 +20,39 @@
20 20  
21 21  = Creating a Dataset =
22 22  
23 -New Datasets are empty containers ready for you to add data files and/or folders from your Project Core. Your Dataset is private - only you can view and work with it until you download and share it with others.
23 +New Datasets are empty containers ready for you to add data files and/or folders from your Project Core. Each Dataset can be viewed, accessed, and interacted with by the user who created it, as well as all users with the Project Administrator role in the Project the Dataset is associated with.
24 24  
25 25  Datasets are created from the Main Menu. To create a new dataset,
26 26  
27 -[[image:Main Menu Datasets NonAdmin v2.26 2023-04-12.png||height="5%" width="50%"]]
27 +[[image:1723802288344-928.png||height="92" width="859"]]
28 28  
29 +
29 29  1. Click **Datasets** from the Main Menu to open the Datasets landing page. Datasets you previously created are displayed in this page in the “My Datasets” tab, and you can continue working on them as described in the following sections of this article, e.g. see //Adding Files to a Dataset//.
30 30  1. Click **+ Create New** in the upper right-hand corner. You will be prompted to provide some essential metadata information about your dataset in two sections (“Define Dataset” and “Description”, see the following list for more details):
31 31  Note: this information is stored in the Dataset’s Essential schema and is visible in the Dataset Home tab, and can be changed later (with the exception of Dataset Code) - see //Reviewing and Editing Metadata Annotations//. Mandatory fields are denoted by a red asterisk ( * ).
32 -11. Define Dataset~://Title*//: A short title for the Dataset (max. 100 characters). This entry can be changed later.
33 -1.
34 -1*. //Dataset Code*//: A distinct, immutable code defined by you to uniquely identify your Dataset. This entry cannot be changed later.
35 -1*. //Authors*//: One or more authors. Hit **Enter** after each entry. These entries can be changed later.
36 -1*. //Dataset Type//: Use the default **GENERAL** or, if your Dataset uses the Brain Imaging Data Structure (BIDS) standard and you wish to use the built-in BIDS validation tool, select **BIDS **from the dropdown menu.  This entry can be changed later.
37 -1.
33 +11. Define Dataset
34 +11*. //Title//*: A short title for the Dataset (max. 100 characters). This entry can be changed later.
35 +11*. //Dataset Code//*: A distinct, immutable code defined by you to uniquely identify your Dataset. This entry __cannot__ be changed later.
36 +11*. //Authors//*: One or more authors. Hit **Enter** after each entry. These entries can be changed later.
37 +11*. //Dataset Type//: Use the default **GENERAL** or, if your Dataset uses the Brain Imaging Data Structure (BIDS) standard and you wish to use the built-in BIDS validation tool, select **BIDS** from the dropdown menu. This entry can be changed later.
38 +11. Description:
39 +11*. //Dataset Description//*: A longer description of the dataset. Please remember to comply with the Platform Terms of Use, Privacy Policy, and do not enter sensitive personal information in this field.
40 +11*. //Modality//: Based on the Human Brain Project [[OpenMINDS>>http://github.com/HumanBrainProject/openMINDS]] standard.
41 +11*. //Collection Method//: Based on the Human Brain Project [[OpenMINDS>>http://github.com/HumanBrainProject/openMINDS]] standard.
42 +11*. //License//: License under which you want to share your Dataset (e.g., Creative Commons). A License is not required, yet highly recommended if you want to share your Dataset in compliance with the FAIR criteria.
43 +11*. //Tags//: Custom keywords you create for your Dataset. Tags are displayed on the Datasets landing page and information bar.
44 +1. When you provided all required information, click Create in the upper right-hand corner to complete the Dataset creation.
38 38  
39 -Description:
40 -
41 -1.
42 -1*. //Dataset Description*//:** **A longer description of the dataset. Please remember to comply with the Platform Terms of Use, Privacy Policy, and do not enter sensitive personal information in this field.
43 -1*. //Modality//: Based on the Human Brain Project [[OpenMINDS>>url:http://github.com/HumanBrainProject/openMINDS]] standard.
44 -1*. //Collection Method//: Based on the Human Brain Project [[OpenMINDS>>url:http://github.com/HumanBrainProject/openMINDS]] standard.
45 -1*. //License//: License under which you want to share your Dataset (e.g., Creative Commons). A License is not required, yet highly recommended if you want to share your Dataset in compliance with the FAIR criteria.
46 -1*. //Tags//: Custom keywords you create for your Dataset.  Tags are displayed on the Datasets landing page and information bar.
47 -)))
48 -1. When you provided all required information, click **Create** in the upper right-hand corner to complete the Dataset creation.
49 -
50 50  Your Dataset is now saved and is visible in the Datasets landing page under My Dataset. Next, you can go to a Project and begin copying files into the Dataset, as described below in the section //Adding Files to a Dataset.//
51 51  
48 +
52 52  = Adding Files to a Dataset =
53 53  
54 -As mentioned above, each Dataset is associated to a single Project. Therefore, files can be added to a Dataset only from the Core of the respective Project. This will create a copy of the file so the original file in the Core is unaffected by subsequent manipulations inside the Dataset. The Dataset must have already been created before files can be added (see //Creating a Dataset//).
51 +As mentioned above, each Dataset is associated to a single Project. Therefore, files can be added to a Dataset only from the Core of the respective Project. This will create a copy of the file so the original file in the Core is unaffected by subsequent manipulations inside the Dataset. The Dataset must have already been created before files can be added (see //Creating a Dataset//). Once this association between the Dataset and a Project has been established, the Dataset automatically becomes visible and accessible for all members of the Project that have the Project Administrator role.
55 55  
56 56  To add files to a Dataset,
57 57  
58 -[[image:Datasets Add to Datasets v2.1.5 2023-01-24.png||height="11%" width="50%"]]
55 +[[image:1723802315115-691.png||height="191" width="870"]]
59 59  
60 60  1. Navigate your Project’s File Explorer and click **Core**.
61 61  1. Select one or more files and/or folders by clicking the checkbox beside each one, then click **Add to Datasets** in the File Explorer menu.
... ... @@ -67,13 +67,13 @@
67 67  
68 68  = Organizing and Validating Datasets =
69 69  
70 -After copying Project files into a Dataset, you can perform a variety of tasks including moving, previewing, renaming or deleting files and folders, and validating the file structure against a supported external specification (currently, the [[Brain Imaging Data Structure (BIDS)>>url:https://bids.neuroimaging.io/specification.html]] standard is supported). You can also add metadata (see //Annotating Datasets with Metadata//).
67 +After copying Project files into a Dataset, you can perform a variety of tasks including moving, previewing, renaming or deleting files and folders, and validating the file structure against a supported external specification (currently, the [[Brain Imaging Data Structure (BIDS)>>url:https://bids.neuroimaging.io/specification.html]] standard is supported). You can also add metadata (see //Annotating Datasets with Metadata//) and share these with members of your Project by first creating a Space in the [[EBRAINS Knowledge Graph>>url:https://docs.kg.ebrains.eu/]], and then uploading the metadata to it (see //Interacting with the EBRAINS Knowledge Graph//).
71 71  
72 72  == Organizing Datasets ==
73 73  
74 74  Use the **Explorer** tab to organize and validate your Dataset.
75 75  
76 -[[image:Datasets Explorer v2.1.5 2023-01-24.png||height="10%" width="50%"]]
73 +[[image:1723802327251-439.png||height="182" width="897"]]
77 77  
78 78  1. Open the Dataset from the Datasets landing page, then select the **Explorer **tab.
79 79  1. Use **+ New Folder** or **Move to **to rearrange files and/or folders.
... ... @@ -92,13 +92,13 @@
92 92  
93 93  === BIDS Validation ===
94 94  
95 -[[image:BIDS_Logo.png||height="9%" width="20%"]]
92 +[[image:1723802339509-521.png||height="83" width="192"]]
96 96  
97 97  The **Validate BIDS** tool allows you to check your Dataset for conformation with the [[Brain Imaging Data Structure (BIDS)>>url:https://bids.neuroimaging.io/specification.html]] standard. The tool uses the open-source [[bids-validator 1.8.4 Python Package>>url:https://pypi.org/project/bids-validator/#quickstart]].
98 98  
99 99  Before running the Validate BIDS tool, ensure that the Dataset Type is set to BIDS (defined at Dataset creation or modified in the **Metadata **tab under Schemas > Essential).
100 100  
101 -[[image:Datasets Validate BIDS v2.1.5 2023-01-24.png||height="13%" width="50%"]]
98 +[[image:1723802351811-730.png||height="230" width="878"]]
102 102  
103 103  1. Open the Dataset from the Datasets landing page and select the **Explorer **tab.
104 104  1. Click **Validate BIDS**. If the Validate BIDS button isn’t visible, make sure your Dataset Type is set to BIDS (see above).
... ... @@ -128,17 +128,17 @@
128 128  
129 129  Use the **Metadata** tab to annotate your Dataset.
130 130  
131 -[[image:Datasets Metadata v2.1.5 2023-01-24.png||height="10%" width="50%"]]
128 +[[image:1723802365719-520.png||height="176" width="876"]]
132 132  
133 133  These steps describe how to annotate your dataset using the Default Schema. If you have created a Custom Schema (see the section titled //Creating a Custom Schema// in this article), you can follow the same steps described here, selecting your Custom Schema name instead of Default Schema in Step 2.
134 134  
135 135  1. Open the Dataset from the Datasets landing page and select the **Metadata **tab.
136 136  1. Under **Existing Schema**, select **Default Schemas**. The schemas that have already been completed in part or full are listed in the left panel, and their details are displayed in the right panel. This includes the Essential schema template with the information you entered at the time the Dataset was created.
137 -1. In the **Schemas **section on the right panel, click the **Select schema to complete **dropdown menu and select one of the Default schema templates or a Custom Schema created by you (see the section //Creating a Custom Schema //in this article).
134 +1. (((
135 +In the **Schemas **section on the right panel, click the **Select schema to complete **dropdown menu and select one of the Default schema templates or a Custom Schema created by you (see the section //Creating a Custom Schema //in this article).
138 138  
139 -[[image:Datasets Metadata Default Schema v2.1.5 2023-01-25.png||height="38%" width="50%"]]
137 +[[image:1723802376639-303.png||height="246" width="326"]]
140 140  
141 -1. (((
142 142  Default schema categories include the following:
143 143  
144 144  * //Contributors//: Information about the persons or organizations who contributed to the Dataset.
... ... @@ -160,14 +160,11 @@
160 160  
161 161  1. Navigate to **Existing Schema**// //(left panel of the Metadata tab) and select the **Default Schemas** category.
162 162  1. Click the schema name, then click the eye icon to open the Schemas view (right side). The existing entries are displayed in the **Schemas **panel at the right side.
163 -1. (((
164 -To edit the entries, click **Edit** and make the changes, and.
160 +1. To edit the entries, click **Edit** and make the changes, and
161 +1*. **X Reset** to return all entries to their original values.
162 +1*. **Cancel **to exit without saving changes.
163 +1*. **Update** to save the changes and return to the schema view.
165 165  
166 -1. **X Reset** to return all entries to their original values.
167 -1. **Cancel **to exit without saving changes.
168 -1. **Update** to save the changes and return to the schema view.
169 -)))
170 -
171 171  Any new changes are reflected in the Schemas view.
172 172  
173 173  == Creating a Custom Schema template ==
... ... @@ -176,7 +176,7 @@
176 176  
177 177  To create a Custom Schema,
178 178  
179 -[[image:Datasets Metadata Create Custom Schema v2.1.5 2023-01-25.png||height="24%" width="50%"]]
173 +[[image:1723802393964-978.png||height="237" width="485"]]
180 180  
181 181  1. Open the Dataset from the Datasets landing page and select the **Metadata **tab. Under **Existing Schema** (left panel), select **Default Schemas**.
182 182  1. In the **Schemas **panel (right side), click the **Select schema to complete **dropdown menu and select **+ Create Custom Schema** to open the Custom Schema Template creation window.
... ... @@ -205,18 +205,15 @@
205 205  
206 206  To manage a Custom Schema template that contains saved annotations,
207 207  
208 -[[image:Datasets Metadata Manage Custom Schema Template v2.1.5 2023-01-25.png||height="17%" width="50%"]]
202 +[[image:1723802410617-147.png||height="163" width="481"]]
209 209  
210 210  1. Navigate to **Existing Schema**// //(left panel of the **Metadata **tab) and select the **Default Schema** category.
211 211  1. Click the Custom schema name, then click the **eye icon** to open the Schemas view (right side). The existing entries are displayed.
212 212  1. Click **Manage Template**.
213 -1. (((
214 -Make your desired edits. The following changes are possible:
215 -
216 -1. Add new fields
217 -1. Delete existing fields
218 -1. Change an existing field from required to optional
219 -)))
207 +1. Make your desired edits. The following changes are possible:
208 +1*. Add new fields
209 +1*. Delete existing fields
210 +1*. Change an existing field from required to optional
220 220  1. Give the updated Schema template a new name, e.g., {name} version 2.
221 221  1. Click **+ Create New Template**.
222 222  
... ... @@ -224,7 +224,7 @@
224 224  
225 225  == Annotating with OpenMINDS ==
226 226  
227 -[[image:openMINDS_logo.png||height="10%" width="20%"]]
218 +[[image:1723802431881-142.png||height="111" width="222"]]
228 228  
229 229  The **open** **M**etadata **I**nitiative for **N**euroscience **D**ata **S**tructures ([[openMINDS>>url:https://github.com/HumanBrainProject/openMINDS]])// //is an open-source, community-driven research infrastructure initiative powered by EBRAINS and the Human Brain Project. The openMINDS schema gathers a set of metadata models that can be used for describing heterogeneous neuroscience data. The data can originate from human, animal or simulated studies, computational models, and software tools, as well as metadata or data models.
230 230  
... ... @@ -232,7 +232,7 @@
232 232  
233 233  Community tools are offered to help generate openMINDS JSON schemas, including a [[wizard>>url:https://wizard.kg-ppd.ebrains.eu/]] and a [[python package>>url:https://pypi.org/project/openMINDS/]]. For information or support, please contact the [[openMINDS team>>url:https://github.com/HumanBrainProject/openMINDS]].
234 234  
235 -[[image:Datasets Metadata openMINDS Schemas v2.1.5 2023-01-25.png||height="18%" width="50%"]]
226 +[[image:1723802444445-662.png||height="320" width="864"]]
236 236  
237 237  To upload an openMINDS JSON schema file,
238 238  
... ... @@ -253,7 +253,7 @@
253 253  
254 254  To view Activity, open the Dataset from the Datasets landing page and select the **Activity **tab. Use the date filters to narrow the range of activities displayed.
255 255  
256 -[[image:Datasets Activity v2.1.5 2023-01-24.png||height="10%" width="50%"]]
247 +[[image:1723802458591-860.png||height="174" width="863"]]
257 257  
258 258  
259 259  == Anytime Dataset Download ==
... ... @@ -268,7 +268,7 @@
268 268  
269 269  To release a Dataset version,
270 270  
271 -[[image:Datasets Release new version v2.1.5 2023-01-26.png||height="10%" width="50%"]]
262 +[[image:1723802471096-883.png||height="169" width="864"]]
272 272  
273 273  
274 274  1. Open the Dataset from the **Datasets **landing page and ensure you have made all the desired updates to the files, folder structure and metadata.
... ... @@ -282,14 +282,13 @@
282 282  
283 283  To view all released Dataset versions, click **Versions **in the Dataset information bar located next to the Dataset title. This opens the **Versions **sidebar where you can browse and download the Versions you have created.
284 284  
285 -[[image:Datasets Versions v2.1.5 2023-01-26.png||height="24%" width="50%"]]
276 +[[image:1723802481440-617.png||height="234" width="488"]]
286 286  
287 287  ----
288 288  
289 -Copyright © 2023 [[Indoc Research>>url:https://www.indocresearch.org/]].
280 +Copyright © 2023-2024 [[Indoc Systems>>url:https://www.indocsystems.com]].
290 290  
291 -HealthDataCloud is powered by Pilot technology, a product of [[Indoc Research>>url:https://www.indocresearch.org/]].
282 +HealthDataCloud is powered by Pilot technology, a product of [[Indoc Systems>>url:https://www.indocsystems.com]].
292 292  
293 293  
294 -
295 295  
BIDS_Logo.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -85.4 KB
Content
Datasets Activity v2.1.5 2023-01-24.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -323.8 KB
Content
Datasets Add to Datasets v2.1.5 2023-01-24.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -226.9 KB
Content
Datasets Explorer v2.1.5 2023-01-24.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -321.6 KB
Content
Datasets Metadata Create Custom Schema v2.1.5 2023-01-25.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -265.5 KB
Content
Datasets Metadata Default Schema v2.1.5 2023-01-25.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -398.9 KB
Content
Datasets Metadata Manage Custom Schema Template v2.1.5 2023-01-25.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -204.7 KB
Content
Datasets Metadata openMINDS Schemas v2.1.5 2023-01-25.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -727.6 KB
Content
Datasets Metadata v2.1.5 2023-01-24.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -324.1 KB
Content
Datasets Release new version v2.1.5 2023-01-26.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -325.5 KB
Content
Datasets Validate BIDS v2.1.5 2023-01-24.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -120.8 KB
Content
Datasets Versions v2.1.5 2023-01-26.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -374.2 KB
Content
Main Menu Datasets NonAdmin v2.26 2023-04-12.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -3.4 KB
Content
openMINDS_logo.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.sgevans
Size
... ... @@ -1,1 +1,0 @@
1 -50.3 KB
Content
1723802288344-928.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +4.1 KB
Content
1723802310399-642.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +4.1 KB
Content
1723802315115-691.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +101.2 KB
Content
1723802327251-439.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +115.1 KB
Content
1723802339509-521.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +121.0 KB
Content
1723802351811-730.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +93.2 KB
Content
1723802365719-520.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +112.8 KB
Content
1723802376639-303.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +415.8 KB
Content
1723802393964-978.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +244.9 KB
Content
1723802410617-147.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +210.6 KB
Content
1723802431881-142.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +49.0 KB
Content
1723802444445-662.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +291.0 KB
Content
1723802458591-860.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +108.8 KB
Content
1723802471096-883.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +115.2 KB
Content
1723802481440-617.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.dsegebarth
Size
... ... @@ -1,0 +1,1 @@
1 +319.9 KB
Content