Quickstart¶
Note
See the Installation instructions first if you have not yet installed Nipoppy.
Initializing a new dataset¶
An empty directory tree can be generated by running nipoppy init. Alternatively, a directory path can be specified:
$ nipoppy init --dataset <PATH_TO_DATASET_ROOT>
Warning
Initializing an existing directory will result in an error.
The newly created directory tree follows the Nipoppy specification. Other Nipoppy commands expect all these directories to exist – they will throw an error if that is not the case.
Tip
Each subdirectory contains a README.md file that briefly describes the purpose of the subdirectory and the type of data that it should contain.
Creating/modifying required files¶
Nipoppy requires two user-provided files in each dataset: a JSON configuration file and a tabular manifest file. Commands will result in errors if either of these files does not exist or are invalid.
Note
The nipoppy init command copies examples of these files to the expected paths within a new dataset, but you will most likely have to modify/overwrite them.
Customizing the global configuration file¶
The global configuration file at <DATASET_ROOT>/global_config.json contains the high-level description of a dataset (e.g. name, visit and session names) and configurations for running the available pipelines (e.g., pipeline version and runtime parameters).
The example global config file contains configurations for all BIDS conversion and image processing software that are supported out-of-the-box by Nipoppy. You should replace the placeholder strings/substrings (e.g. <DATASET_NAME>) by more appropriate values for your dataset. See dropdown below (lines that should be changed are highlighted).
You can also delete (or not) any configuration for a software/version that you do not plan to use.
The example global config file
Here is the default content of <DATASET_ROOT>/global_config.json:
1{
2 "DATASET_NAME": "<DATASET_NAME>",
3 "VISIT_IDS": [
4 "<VISIT_LABEL>",
5 "<OTHER_VISIT_LABEL>"
6 ],
7 "SESSION_IDS": [
8 "<SESSION_LABEL>",
9 "<OTHER_SESSION_LABEL>"
10 ],
11 "SUBSTITUTIONS": {
12 "[[NIPOPPY_DPATH_CONTAINERS]]": "[[NIPOPPY_DPATH_CONTAINERS]]",
13 "[[HEUDICONV_HEURISTIC_FILE]]": "<PATH_TO_HEURISTIC_FILE>",
14 "[[DCM2BIDS_CONFIG_FILE]]": "<PATH_TO_CONFIG_FILE>",
15 "[[FREESURFER_LICENSE_FILE]]": "<PATH_TO_FREESURFER_LICENSE_FILE>",
16 "[[TEMPLATEFLOW_HOME]]": "<PATH_TO_TEMPLATEFLOW_DIRECTORY>"
17 },
18 "DICOM_DIR_PARTICIPANT_FIRST": true,
19 "CONTAINER_CONFIG": {
20 "COMMAND": "apptainer",
21 "ARGS": [
22 "--cleanenv"
23 ]
24 },
25 "BIDS_PIPELINES": [
26 {
27 "NAME": "heudiconv",
28 "VERSION": "0.12.2",
29 "CONTAINER_INFO": {
30 "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/[[PIPELINE_NAME]]_[[PIPELINE_VERSION]].sif",
31 "URI": "docker://nipy/[[PIPELINE_NAME]]:[[PIPELINE_VERSION]]"
32 },
33 "STEPS": [
34 {
35 "NAME": "prepare",
36 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation-[[STEP_NAME]].json",
37 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor.json"
38 },
39 {
40 "NAME": "convert",
41 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation-[[STEP_NAME]].json",
42 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor.json",
43 "CONTAINER_CONFIG": {
44 "ARGS": [
45 "--bind",
46 "[[HEUDICONV_HEURISTIC_FILE]]"
47 ]
48 },
49 "UPDATE_STATUS": true
50 }
51 ]
52 },
53 {
54 "NAME": "dcm2bids",
55 "VERSION": "3.2.0",
56 "CONTAINER_INFO": {
57 "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/[[PIPELINE_NAME]]_[[PIPELINE_VERSION]].sif",
58 "URI": "docker://unfmontreal/[[PIPELINE_NAME]]:[[PIPELINE_VERSION]]"
59 },
60 "STEPS": [
61 {
62 "NAME": "prepare",
63 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation-[[STEP_NAME]].json",
64 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor-dcm2bids_helper.json",
65 "ANALYSIS_LEVEL": "group"
66 },
67 {
68 "NAME": "convert",
69 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation-[[STEP_NAME]].json",
70 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor-dcm2bids.json",
71 "CONTAINER_CONFIG": {
72 "ARGS": [
73 "--bind",
74 "[[DCM2BIDS_CONFIG_FILE]]"
75 ]
76 },
77 "UPDATE_STATUS": true
78 }
79 ]
80 },
81 {
82 "NAME": "bidscoin",
83 "VERSION": "4.3.2",
84 "STEPS": [
85 {
86 "NAME": "prepare",
87 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation-[[STEP_NAME]].json",
88 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor-bidsmapper.json",
89 "ANALYSIS_LEVEL": "group"
90 },
91 {
92 "NAME": "edit",
93 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation-[[STEP_NAME]].json",
94 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor-bidseditor.json",
95 "ANALYSIS_LEVEL": "group"
96 },
97 {
98 "NAME": "convert",
99 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation-[[STEP_NAME]].json",
100 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor-bidscoiner.json",
101 "ANALYSIS_LEVEL": "participant",
102 "UPDATE_STATUS": true
103 }
104 ]
105 }
106 ],
107 "PROC_PIPELINES": [
108 {
109 "NAME": "bids-validator",
110 "VERSION": "2.0.3",
111 "CONTAINER_INFO": {
112 "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/deno_2.2.3.sif",
113 "URI": "docker://denoland/deno:2.2.3"
114 },
115 "CONTAINER_CONFIG": {
116 "ARGS": [
117 "--bind",
118 "[[NIPOPPY_DPATH_SCRATCH]]/deno:/deno-dir"
119 ]
120 },
121 "STEPS": [
122 {
123 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation.json",
124 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor.json",
125 "ANALYSIS_LEVEL": "group",
126 "GENERATE_PYBIDS_DATABASE": false
127 }
128 ]
129 },
130 {
131 "NAME": "fmriprep",
132 "VERSION": "24.1.1",
133 "CONTAINER_INFO": {
134 "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/[[PIPELINE_NAME]]_[[PIPELINE_VERSION]].sif",
135 "URI": "docker://nipreps/[[PIPELINE_NAME]]:[[PIPELINE_VERSION]]"
136 },
137 "CONTAINER_CONFIG": {
138 "ENV_VARS": {
139 "TEMPLATEFLOW_HOME": "[[TEMPLATEFLOW_HOME]]"
140 },
141 "ARGS": [
142 "--bind",
143 "[[FREESURFER_LICENSE_FILE]]",
144 "--bind",
145 "[[TEMPLATEFLOW_HOME]]"
146 ]
147 },
148 "STEPS": [
149 {
150 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation.json",
151 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor.json",
152 "TRACKER_CONFIG_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/tracker_config.json"
153 }
154 ]
155 },
156 {
157 "NAME": "freesurfer",
158 "VERSION": "7.3.2",
159 "DESCRIPTION": "Freesurfer version associated with fMRIPrep version 23.0.0 and later",
160 "STEPS": [
161 {
162 "TRACKER_CONFIG_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/tracker_config.json"
163 }
164 ]
165 },
166 {
167 "NAME": "mriqc",
168 "VERSION": "23.1.0",
169 "CONTAINER_INFO": {
170 "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/[[PIPELINE_NAME]]_[[PIPELINE_VERSION]].sif",
171 "URI": "docker://nipreps/[[PIPELINE_NAME]]:[[PIPELINE_VERSION]]"
172 },
173 "CONTAINER_CONFIG": {
174 "ENV_VARS": {
175 "TEMPLATEFLOW_HOME": "[[TEMPLATEFLOW_HOME]]"
176 },
177 "ARGS": [
178 "--bind",
179 "[[TEMPLATEFLOW_HOME]]"
180 ]
181 },
182 "STEPS": [
183 {
184 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation.json",
185 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor.json",
186 "TRACKER_CONFIG_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/tracker_config.json"
187 }
188 ]
189 },
190 {
191 "NAME": "qsiprep",
192 "VERSION": "0.23.0",
193 "CONTAINER_INFO": {
194 "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/[[PIPELINE_NAME]]_[[PIPELINE_VERSION]].sif",
195 "URI": "docker://pennbbl/[[PIPELINE_NAME]]:[[PIPELINE_VERSION]]"
196 },
197 "CONTAINER_CONFIG": {
198 "ENV_VARS": {
199 "TEMPLATEFLOW_HOME": "[[TEMPLATEFLOW_HOME]]"
200 },
201 "ARGS": [
202 "--bind",
203 "[[FREESURFER_LICENSE_FILE]]",
204 "--bind",
205 "[[TEMPLATEFLOW_HOME]]"
206 ]
207 },
208 "STEPS": [
209 {
210 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation.json",
211 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor.json",
212 "TRACKER_CONFIG_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/tracker_config.json"
213 }
214 ]
215 },
216 {
217 "NAME": "rabies",
218 "VERSION": "0.5.1",
219 "CONTAINER_INFO": {
220 "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/rabies_0.5.1.sif",
221 "URI": "docker://ghcr.io/cobralab/rabies:0.5.1"
222 },
223 "CONTAINER_CONFIG": {
224 },
225 "STEPS": [
226 {
227 "NAME": "preprocess",
228 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation-[[STEP_NAME]].json",
229 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor-[[STEP_NAME]].json",
230 "ANALYSIS_LEVEL": "group",
231 "GENERATE_PYBIDS_DATABASE": false
232 },
233 {
234 "NAME": "confound-correction",
235 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation-[[STEP_NAME]].json",
236 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor-[[STEP_NAME]].json",
237 "ANALYSIS_LEVEL": "group",
238 "GENERATE_PYBIDS_DATABASE": false
239 },
240 {
241 "NAME": "analysis",
242 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation-[[STEP_NAME]].json",
243 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor-[[STEP_NAME]].json",
244 "GENERATE_PYBIDS_DATABASE": false,
245 "ANALYSIS_LEVEL": "group"
246 },
247 {
248 "NAME": "preprocess-tracking",
249 "TRACKER_CONFIG_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/[[STEP_NAME]]_tracker_config.json"
250 },
251 {
252 "NAME": "confound-correction-tracking",
253 "TRACKER_CONFIG_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/[[STEP_NAME]]_tracker_config.json"
254 },
255 {
256 "NAME": "analysis-tracking",
257 "TRACKER_CONFIG_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/[[STEP_NAME]]_tracker_config.json"
258 }
259
260 ]
261 }
262 ],
263 "EXTRACTION_PIPELINES": [
264 {
265 "NAME": "fs_stats",
266 "VERSION": "0.2.1",
267 "CONTAINER_INFO": {
268 "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/freesurfer_7.3.2.sif",
269 "URI": "docker://freesurfer/freesurfer:7.3.2"
270 },
271 "CONTAINER_CONFIG": {
272 "ENV_VARS": {
273 "FS_LICENSE": "[[FREESURFER_LICENSE_FILE]]"
274 },
275 "ARGS": [
276 "--bind",
277 "[[FREESURFER_LICENSE_FILE]]"
278 ]
279 },
280 "PROC_DEPENDENCIES": [
281 {
282 "NAME": "freesurfer",
283 "VERSION": "7.3.2"
284 }
285 ],
286 "STEPS": [
287 {
288 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation.json",
289 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor.json",
290 "ANALYSIS_LEVEL": "group"
291 }
292 ]
293 },
294 {
295 "NAME": "static_FC",
296 "VERSION": "0.1.0",
297 "PROC_DEPENDENCIES": [
298 {
299 "NAME": "fmriprep",
300 "VERSION": "23.1.3"
301 }
302 ],
303 "STEPS": [
304 {
305 "INVOCATION_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/invocation.json",
306 "DESCRIPTOR_FILE": "[[NIPOPPY_DPATH_PIPELINES]]/[[PIPELINE_NAME]]-[[PIPELINE_VERSION]]/descriptor.json"
307 }
308 ]
309 }
310 ],
311 "CUSTOM": {}
312}
Tip
See the Understanding the global configuration file for more information about each config field.
Generating the manifest file¶
The manifest file at <DATASET_ROOT>/manifest.tsv contains ground truth information about the participants and visits/sessions available for a dataset.
There must be only one row per unique participant/visit combination.
The example manifest looks like this:
participant_id |
visit_id |
session_id |
datatype |
|---|---|---|---|
01 |
BL |
BL |
[‘anat’] |
01 |
M06 |
||
01 |
M12 |
M12 |
[‘anat’] |
02 |
BL |
BL |
[‘anat’,’dwi’] |
02 |
M06 |
||
02 |
M12 |
M12 |
[‘anat’,’dwi’] |
Raw content of the example manifest file
1participant_id visit_id session_id datatype
201 BL BL ['anat']
301 M06
401 M12 M12 ['anat']
502 BL BL ['anat','dwi']
602 M06
702 M12 M12 ['anat','dwi']
It is extremely unlikely that this manifest file works for your dataset, so you will have to generate one yourself. We recommend writing a script, for the purpose of reproducibility and easy updates if the more data is added to the dataset.
Tip
See the schema reference for more information about each column.
Next steps¶
Note
The rest of this documentation is still work in progress. If the information you are looking for is not available in the user guide, refer to the commands associated with the data organization or processing step(s) you wish to perform.
Once the Nipoppy directory tree for a study is created, the next steps are typically to populate it with raw data, move and organize raw imaging data (typically DICOMs) into a regular structure, then convert the data to BIDS. However, depending on the type of raw data you have, your workflow might be a little different. See here for all the available documentation in the User guide.