HED Python tools

The HED (Hierarchical Event Descriptor) scripts and notebooks assume that the Python HedTools have been installed. The HedTools package is not yet available on PyPI, so you will need to install it directly from GitHub using:

 pip install git+https://github.com/hed-standard/hed-python/@master

There are several types of Jupyter notebooks and other HED support tools:

Jupyter notebooks for HED in BIDS

The following notebooks are specifically designed to support HED annotation for BIDS datasets.

Extract JSON template

The usual strategy for producing machine-actionable event annotation using HED in BIDS is to create a single events.json sidecar file in the BIDS dataset root directory. Ideally, this sidecar will contain all the annotations needed for users to understand and analyze the data.

See the BIDS annotation quickstart for additional information on this strategy and an online version of the tools. The Create a JSON template tutorial provides a step-by-step tutorial for using the online tool that creates a template based on the information in a single events.tsv file. For most datasets, this is sufficient. In contrast, the extract_json_template.ipynb Jupyter notebook bases the extracted template on the entire dataset.

To use this notebook, substitute the specifics of your BIDS dataset for the following variables:

Variables to set in the extract_json_template.ipynb Jupyter notebook.

Variable

Purpose

dataset_path

Full path to root directory of dataset.

exclude_dirs

List of directories to exclude when constructing the list of event files.

skip_columns

List of column names in the events.tsv files to skip in the template

value_columns

List of columns names in the events.tsv files that will be annotated
as a whole rather than by individual column value.

output_path

Full path of output file. If None, then output is printed.

The exclude_dirs should usually include the code, stimuli, derivatives, and sourcedata subdirectories. The onset, duration and sample columns are almost always skipped, since these are predefined in BIDS.

Columns designated as value columns are annotated with a single annotation that always includes a # placeholder. This placeholder marks the position in the annotation where each individual column value is substituted when the annotation is assembled.

All columns not designated as skip columns or value columns are considered to be categorical columns. Each individual value in a categorical column has its own This annotation that must include a # indicating where each value in the value column is substituted when the annotation is assembled. You should take care to correctly identify the skip columns and the value columns, as the individual values in a column such as onset are usually unique across the data, resulting in a huge number of annotations.

Find event combinations

The find_event_combinations.ipynb Jupyter notebook extracts a spreadsheet containing the unique combination of values in the specified key_columns. The setup requires the following variables for your dataset:

Variables to set in the find_event_combinations.ipynb Jupyter notebook.

Variable

Purpose

dataset_path

Full path to root directory of dataset.

exclude_dirs

List of directories to exclude when constructing file lists.

key_columns

List of column names in the events.tsv files to combine.

output_path

Output path for the spreadsheet template. If None, then print the result.

The result will be a tabular file (tab-separated file) whose columns are the key_columns in the order given. The values will be all unique combinations of the key_columns, sorted by columns from left to right.

This can be used to remap the columns in event files to use a new recoding. The resulting spreadsheet is also useful for deciding whether two columns contain redundant information.

Merge spreadsheet into sidecar

Often users find it easier to work with a spreadsheet rather than a JSON file when creating annotations for their dataset. For this reason, the HED tools offer the option of creating a 4-column spreadsheet from a BIDS JSON sidecar and using it to create annotations. The four columns are: column_name, column_value, description, and HED. See task-WorkingMemory_example_spreadsheet.tsv The sidecar_to_spreadsheet.ipynb

Variables to set in the extract_json_template.ipynb Jupyter notebook.

Variable

Purpose

spreadsheet_path

Full path to spreadsheet (4-column tsv file).

sidecar_path

Path to sidecar to merge into. If None, then just convert.

description_tag

(bool) If True, then the contents of the description column
is added to the annotation using the Description tag.

output_path

The output path of the merged sidecar. If None, then just print it.

Sidecar to spreadsheet

If you have a BIDS JSON event sidecar or a sidecar template, you may find it more convenient to view and edit the HED annotations in spreadsheet rather than working with the JSON file directly as explained in the Spreadsheet templates tutorial.

The sidecar_to_spreadsheet.ipynb notebook demonstrates how to extract the pertinent HED annotation to a 4-column spreadsheet (Pandas dataframe) corresponding to the HED content of a JSON sidecar. A spreadsheet representation is useful for quickly reviewing and editing HED annotations. You can easily merge the edited information back into the BIDS JSON events sidecar.

Here is an example of the spreadsheet that is produced by converting a JSON sidecar template to a spreadsheet template that is ready to edit. You should only change the values in the description and the HED columns.

Example 4-column spreadsheet template for HED annotation.

column_name

column_value

description

HED

event_type

setup_right_sym

Description for setup_right_sym

Label/setup_right_sym

event_type

show_face

Description for show_face

Label/show_face

event_type

left_press

Description for left_press

Label/left_press

event_type

show_circle

Description for show_circle

Label/show_circle

stim_file

n/a

Description for stim_file

Label/#

To use this notebook, you will need to provide the path to the JSON sidecar and a path to save the spreadsheet if you want to save it. If you don’t wish to save the spreadsheet, assign spreadsheet_filename to be None and the result is just printed.

Summarize events

Sometimes event files include unexpected or incorrect codes. It is a good idea to find out what is actually in the dataset event files and whether the information is consistent before starting the annotation process.

The summarize_events.ipynb finds the dataset event files and outputs the column names and number of events for each event file. You can visually inspect the output to make sure that the event file column names are consistent across the dataset. The script also summarizes the unique values that appear in different event file columns across the dataset.

To use this notebook, substitute the specifics of your BIDS dataset for the following variables:

Variables to set in the summarize_events.ipynb Jupyter notebook.

Variable

Purpose

dataset_path

Full path to root directory of dataset.

exclude_dirs

List of directories to exclude when constructing the list of event files.

skip_columns

List of column names in the events.tsv files to skip in the template

value_columns

List of columns names in the events.tsv files that will be annotated
as a whole rather than by individual column value.

output_path

Full path of output file. If None, then output is printed.

These same variables are required for the Extract JSON template operation.

For large datasets, be sure to skip columns such as onset and sample, since the summary produces counts of the number of times each unique value appears somewhere in dataset event files.

Validate BIDS dataset

Validating HED annotations as you develop them makes the annotation process easier and faster to debug. The HED validation guide discusses various HED validation issues and how to fix them.

The validate_bids_dataset.ipynb Jupyter notebook validates HED in a BIDS dataset using the validate method of BidsDataset. The method first gathers all the relevant JSON sidecars for each event file and validates the sidecars. It then validates the individual events.tsv files based on applicable sidecars.

Variables to set in the validate_bids_dataset.ipynb Jupyter notebook.

Variable

Purpose

dataset_path

Full path to root directory of dataset.

check_for_warnings

Boolean, which if True returns warnings as well as errors

The script requires you to set the check_for_warnings flag and the root path to your BIDS dataset.

Note: This validation pertains to event files and HED annotation only. It does not do a full BIDS validation.

Validate BIDS dataset with libraries

The validate_bids_dataset_with_libraries.ipynb Jupyter notebook validates HED in a BIDS dataset using the validate method of BidsDataset. The example uses three schemas and also illustrates how to manually override the schema specified in dataset_description.json with schemas from other places. This is very useful for testing new schemas that are underdevelopment.

Validate BIDS datasets

The validate_bids_datasets.ipynb is similar to the other validation notebooks, but it takes a list of datasets to validate as a convenience.

Jupyter notebooks for data curation

All data curation notebooks and other examples can now be found in the hed-examples repository.

Consistency of BIDS event files

Some neuroimaging modalities such as EEG, typically contain event information encoded in the data recording files, and the BIDS events.tsv files are generated post hoc.

In general, the following things should be checked before data is released:

  1. The BIDS events.tsv files have the same number of events as the data recording and that onset times of corresponding events agree.

  2. The associated information contained in the data recording and event files is consistent.

  3. The relevant metadata is present in both versions of the data.

The example data curation scripts discussed in this section assume that two versions of each BIDS event file are present: events.tsv and a corresponding events_temp.tsv file. The example datasets that are using for these tutorials assume that the recordings are in EEG.set format.

Calling HED tools

This section shows examples of useful processing functions provided in HedTools:

Getting a list of files

Many situations require the selection of files in a directory tree based on specified criteria. The get_file_list function allows you to pick out files with a specified filename prefix and filename suffix and specified extensions

The following example returns a list of full paths of the files whose names end in _events.tsv or _events.json that are not in any code or derivatives directories in the bids_root_path directory tree. The search starts in the directory root bids_root_path:

Get a list of specified files in a specified directory tree.

file_list = get_file_list(bids_root_path, extensions=[ ".json", ".tsv"], name_suffix="_events",
                          name_prefix="", exclude_dirs=[ "code", "derivatives"])

Dictionaries of filenames

The HED tools provide both generic and BIDS-specific classes for dictionaries of filenames.

Many of the HED data processing tools make extensive use of dictionaries specifying both data and format.

BIDS-specific dictionaries of files

Files in BIDS have unique names that indicate not only what the file represents, but also where that file is located within the BIDS dataset directory tree.

BIDS file names and keys

A BIDS file name consists of an underbar-separated list of entities, each specified as a name-value pair, followed by suffix indicating the data modality.

For example, the file name sub-001_ses-3_task-target_run-01_events.tsv has entities subject (sub), task (task), and run (run). The suffix is events indicating that the file contains events. The extension .tsv gives the data format.

Modality is not the same as data format, since some modalities allow multiple formats. For example, sub-001_ses-3_task-target_run-01_eeg.set and sub-001_ses-3_task-target_run-01_eeg.edf are both acceptable representations of EEG files, but the data is in different formats.

The BIDS file dictionaries represented by the class BidsFileDictionary and its extension BidsTabularDictionary use a set combination of entities as the file key.

For a file name sub-001_ses-3_task-target_run-01_events.tsv, the tuple (‘sub’, ‘task’) gives a key of sub-001_task-target, while the tuple (‘sub’, ‘ses’, ‘run’) gives a key of sub-001_ses-3_run-01. The use of dictionaries of file names with such keys makes it easier to associate related files in the BIDS naming structure.

Notice that specifying entities (‘sub’, ‘ses’, ‘run’) gives the key sub-001_ses-3_run-01 for all three files: sub-001_ses-3_task-target_run-01_events.tsv, sub-001_ses-3_task-target_run-01_eeg.set and sub-001_ses-3_task-target_run-01_eeg.edf. Thus, the expected usage is to create a dictionary of files of one modality.

Create a key-file dictionary for files ending in events.tsv in bids_root_path directory tree.

from hed.tools import FileDictionary
from hed.util import get_file_list

file_list = get_file_list(bids_root_path, extensions=[ ".set"], name_suffix="_eeg", 
                          exclude_dirs=[ "code", "derivatives"])
file_dict = BidsFileDictionary(file_list, entities=('sub', 'ses', 'run) )

In this example, the get_file_list filters the files of the appropriate type, while the BidsFileDictionary creates a dictionary with keys such as sub-001_ses-3_run-01 and values that are BidsFile objects. BidsFile can hold the file name of any BIDS file and keeps a parsed version of the file name.

A generic dictionary of filenames

Create a key-file dictionary for files ending in events.json in bids_root_path directory tree.

from hed.tools import FileDictionary
from hed.util import get_file_list

file_list = get_file_list(bids_root_path, extensions=[ ".json"], name_suffix="_events", 
                          exclude_dirs=[ "code", "derivatives"])
file_dict = FileDictionary(file_list, name_indices=name_indices)

Keys are calculated from the filename using a name_indices tuple, which indicates the positions of the name-value entity pairs in the BIDS file name to use.

The BIDS filename sub-001_ses-3_task-target_run-01_events.tsv has three name-value entity pairs (sub-001, ses-3, task-target, and run-01) separated by underbars.

The tuple (0, 2) gives a key of sub-001_task-target, while the tuple (0, 3) gives a key of sub-001_run-01. Neither of these choices uniquely identifies the file. The tuple (0, 1, 3) gives a unique key of sub-001_ses-3_run-01. The tuple (0, 1, 2, 3) also works giving sub-001_ses-3_task-target_run-01.

If you choose the name_indices incorrectly, the keys for the event files will not be unique, and the notebook will throw a HedFileError. If this happens, modify your name_indices key choice to include more entity pairs.

For example, to compare the events stored in a recording file and the events in the events.tsv file associated with that recording, we might dump the recording events in files with the same name, but ending in events_temp.tsv. The FileDictionary class allows us to create a keyed dictionary for each of these event files.

Logging processing steps

Often event data files require considerable processing to assure internal consistency and compliance with the BIDS specification. Once this processing is done and the files have been transformed, it can be difficult to understand the relationship between the transformed files and the original data.

The HedLogger allows you to document processing steps associated with the dataset by identifying key as illustrated in the following log file excerpt:

Example output from HED logger.

sub-001_run-01
	Reordered BIDS columns as ['onset', 'duration', 'sample', 'trial_type', 'response_time', 'stim_file', 'value', 'HED']
	Dropped BIDS skip columns ['trial_type', 'value', 'response_time', 'stim_file', 'HED']
	Reordered EEG columns as ['sample_offset', 'event_code', 'cond_code', 'type', 'latency', 'urevent', 'usertags']
	Dropped EEG skip columns ['urevent', 'usertags', 'type']
	Concatenated the BIDS and EEG event files for processing
	Dropped the sample_offset and latency columns
	Saved as _events_temp1.tsv
sub-002_run-01
	Reordered BIDS columns as ['onset', 'duration', 'sample', 'trial_type', 'response_time', 'stim_file', 'value', 'HED']
	Dropped BIDS skip columns ['trial_type', 'value', 'response_time', 'stim_file', 'HED']
	Reordered EEG columns as ['sample_offset', 'event_code', 'cond_code', 'type', 'latency', 'urevent', 'usertags']
	Dropped EEG skip columns ['urevent', 'usertags', 'type']
	Concatenated the BIDS and EEG event files for processing
	. . .

Each of the lines following a key represents a print message to the logger.

The most common use for a logger is to create a file dictionary using make_file_dict and then to log each processing step using the file’s key. This allows a processing step to be applied to all the relevant files in the dataset. After all the processing is complete, the print_log method outputs the logged messages by key, thus showing all the processing steps that have been applied to each file as shown in the previous example.

Using the HED logger.

from hed.tools import HedLogger
status = HedLogger()
status.add(key, f"Concatenated the BIDS and EEG event files")

# ... after processing is complete output or save the log
status.print_log()

The HedLogger is used throughout the processing notebooks in this repository.