Tamaslab conversion fixes

[sborde]

Dear Tom,

it's Sándor from the Tamás-lab. There are the minor fixes and modifications that Attila has mentioned to you. I've written a few sentences for each modification, I will copy them to the end of this message. Our main goal was to push all of our data through the ipfx, but I tried to avoid breaking the code with my changes. I have written new unit tests and all previous unit tests passed, so I hope I did everything right. :)

During these tests, I got an insight that the definition of the test pulse should be a bit more specific than finding the first two non-zero differences. This can cause some issues with the changes that I made in the get_stim_characteristic and get_test_epoch functions.

Sándor

Notes

Truncated test pulse issue in the get_stim_characteristic function

If the get_stim_characteristic function expects to have a test pulse at the beginning of the sweep, but it is truncated, the function skips the beginning of the real stimulus incorrectly. If a test pulse is expected, but the number of ups and downs is odd, skip the first deflection and act as if we hadn't a test pulse.
Note: the definition of the test pulse could be more robust than the first two non-zero difference.

Truncated test pulse issue in the get_test_epoch function

The ipfx.epochs.get_test_epoch function tries to find the start and the end index of the test epoch if present. However, if the test pulse is truncated, it throws an error. We have changed this code to print an error message instead and return None as in the case of a missing test pulse.
Note: here we should find out a better method for discriminating test pulses because it won't work if we have a truncated test pulse with the following stimulus.

Plotting missing spike features

In some cases, the rheobase spike of the long square pulse couldn't be fully described, and the fast- and slow trough Voltage values were missing. When the ipfx.plot_qc_figures.plot_single_ap_values function tried to plot the spike features, setting the x-axis's limit to NaN caused an error. To handle this, we only plot the successfully extracted (i.e., non-NaN) features. The same problem occurred at the creation of the ramp sweep's plot.

Determination of the deflection type

The peak deflection calculation is performed by calling the ipfx.subthresh_features.voltage_deflection function without specifying the deflection type. In this case, the function searches for minimum or maximum depending on the stimulus's polarity (i.e., maximum search at positive current, minimum search at negative current). However, we have recordings where the stimulus current is above the baseline, but its absolute value is negative. To handle this issue, we determine the type of extrema based on the relation of the stimulus and the baseline instead of its absolute value.
This problem appears at the input_resistance calculation, too: if the stimulus is above the baseline, but its absolute value is negative, the function searches for its minimum, which falls to the same point as the start. So instead of using the min deflection, we use the average voltage and current during the stimulus. Besides that, using the average is more robust than using a single value.

Ignoring nan at the comparison in find_widths

In the ipfx.spike_features.find_widths function, some of the spikes are excluded because those are clipped or have an invalid trough value. But later in the analysis (at the width_level refinement), there is a comparison where these nan values are included, which raises an error. It has been changed to work with indexes instead of logical vectors, so the error is avoided.

Avoiding over-indexing in check_thresholds_and_peaks

During the spike detection, in the ipfx.spike_detector.check_thresholds_and_peaks function, an error occurred if the last spike is too close to the end of the sweep. We have added a condition check before the indexing part to avoid over-indexing.

Handling the filtering of sweeps with a low sample rate

The time_series_utils.calculate_dvdt function is usually called with a filter parameter. If the sampling rate of the input sweep is less than expected (greater than double the filter frequency), the function raises a ValueError. However, in this case, the dv could be calculated from the unfiltered sweep (as we wouldn't have specified a filter value).

[tmchartrand]

This is great! Oddly enough github didn't notify me, but Gabor mentioned it this morning so I managed to find it. Thanks for putting these together and documenting clearly!

My only concern in looking over the changes are those for the negative stimulus and input resistance. Our convention is to always have the holding current subtracted off from that represented in the sweep, and I worry it may be confusing to release files with different conventions, even if the feature extractor is adapted to accommodate them. I'll raise this question with some others here, since we're also trying to compile all converted files in our file storage soon, and this might be a fix we'd want to apply in the conversion process.

Also, the change of shifting to an average v in the input resistance I-V curve is a fairly main one that would slightly shift the feature for all cells, so I hesitate to include that without a strong reason. That sort of change should be caught by the integration test (tests/test_mies_nwb_pipeline_output.py), but I'm guessing that test either didn't get run or was already failing on my branch?