Problem sorting NP1e data with kilosort

[RobertoDF]

Hi,

I just sorted my first ONIX rec using kilosort. This animal has been recorded lin the past using another system and showed a good number of units. However it looks like this after sorting on the ONIX:
https://github.com/user-attachments/assets/63af81d6-f5ed-46ae-907c-4f8b9013195e

Weirdly many units seems upside down. The problem is not resolved if I invert the traces though!

I used this script here:

import os
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import spikeinterface.extractors as se
import spikeinterface.widgets as sw
import probeinterface
import probeinterface.plotting
from pathlib import Path
from probeinterface import  read_probeinterface

ap_gain = 1000 
lfp_gain = 50 
bit_depth = 10
suffix = 6 # Change to match file names' suffix
sampling_frequency = 3e4
num_channels = 384 # Decrease channels to expedite plotting and inspect fewer traces

path_recording_folder = Path(r"X:\data\74\20250128")

probeinterface_filename = 'np1-config.json'

probes = read_probeinterface(Path(path_recording_folder, probeinterface_filename))

ap_scalar = 1.2e6 / (2 ** bit_depth) / ap_gain
ap_offset = (2 ** (bit_depth - 1)) * ap_scalar
ap_recording = se.read_binary(os.path.join(path_recording_folder,  f"np1-spike_{suffix}.raw"), 
                         sampling_frequency, 
                         np.uint16, 
                         num_channels, 
                         gain_to_uV=ap_scalar, 
                         offset_to_uV=-ap_offset) 

ap_recording = ap_recording.set_probegroup(probe)

binary_file_path = os.path.join(path_recording_folder,  f"np1-spike_{suffix}.raw")

probe_filename = f"{path_recording_folder}/spike_interface_output/probe{probe_n}/sorter_output/probe.prb"

pg =  ap_recording.get_probegroup()
write_prb(probe_filename, pg)
print(f"probe {probe_n} completed \n")

probe = load_probe(probe_filename)
settings = {'filename': binary_file_path,
            "n_chan_bin": probe["n_chan"], "fs": ap_recording.get_sampling_frequency()}

result_dir = path_recording_folder / "spike_interface_output" / f"probe0" / "sorter_output"
os.makedirs(result_dir, exist_ok=True)

run_kilosort(settings=settings, probe=probe, data_dtype=sub_rec.get_dtype(),
             device=torch.device("cuda"), results_dir=result_dir, clear_cache=True)

[RobertoDF]

I just realized the binary i am feeding to kilosort is not scaled, trying now with a scaled one

[RobertoDF]

I used

recording_in_uV = spre.scale_to_uV(recording=sub_rec)
write_binary_recording(recording_in_uV, file_paths=binary_file_path, dtype=sub_rec.get_dtype(), **job_kwargs)
        

To write a scaled binary to feed to kilosort but result is still wrong

SpikeInterface/spikeinterface#3053

[RobertoDF]

the scale_to_uV func was silently changeing the dtype therefore messing up the binary. After flipping the sign the sorting seems much better.

[cjsha]

This isn't related to your question, but you might want to consider using some preprocessing tools from spikeinterface (beyond scale_to_uV which you alluded to) before trying to sort spikes. I didn't see anything like that when cursorily reading your script. Preprocessing might help with the offset you're experiencing, though you should first try reducing the offset as much as possible by investigating your recording parameters as described in this comment.

[RobertoDF]

Sure, this was just an initial test. I am quite happy with the result after flipping already, many good units are sorted! I will experiment with some more preprocessing though

[cjsha]

Do you wanna share your data so I can try running it through kilosort myself at some point?

Also, feel free to close if your issue is resolved. I made another issue #202 that is specifies the problems with the loading scripts that you brought up in these issues in the past week.

[RobertoDF]

I will try next week to send you a small binary!thanks!