Pipeline v1.0.0

[alejoe91]

TODOs:

• Add main parameters.json to for all processes
• Integrate spikeinterface input type: Flexible input with spikeinterface option aind-ephys-job-dispatch#15

[alejoe91]

Fixes #38 #41 #43 #45

@vncntprvst It would be great if you could test this PR. There are a lot of updates and fixes :)

[vncntprvst]

Wonderful. We'll try it today with @kaid0212. I'll keep you posted.

[vncntprvst]

Just wanted to check: this error below means we now need to add a session.json file next to the subject.json and data_description.json for this NWB export to work?

  Exporting session: ecephys_session
  Found 4 JSON job files
  Number of NWB files to write: 1
  Number of streams to write for each file: 1
  Processing experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1
  	Loading experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1 from 4 JSON files
  		experiment1_Record Node 101#Neuropix-PXI-100.ProbeA_recording1_group0

Command error:
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  INFO:    Environment variable SINGULARITYENV_KACHERY_CLOUD_CLIENT_ID is set, but APPTAINERENV_KACHERY_CLOUD_CLIENT_ID is preferred
  INFO:    Environment variable SINGULARITYENV_KACHERY_CLOUD_PRIVATE_KEY is set, but APPTAINERENV_KACHERY_CLOUD_PRIVATE_KEY is preferred
  Cloning into 'capsule-repo'...
  + python -u run_capsule.py --params '{"stub":false,"stub_seconds":10,"write_lfp":true,"write_raw":false,"lfp_temporal_factor":2,"lfp_spatial_factor":4,"lfp_highpass_freq_min":0.1,"surface_channel_agar_probes_indices":"","lfp":{"filter":{"freq_min":0.1,"freq_max":500},"sampling_rate":2500}}'
  capsule/code/utils.py:64: UserWarning: Session file not found in ../data/ecephys_session
    warnings.warn(f"Session file not found in {session_folder}")
  ./run: line 5: 1586010 Killed                  python -u run_capsule.py "$@"

[vncntprvst]

For context, it's at this step:

executor >  slurm (7)
[04/6e848b] process > job_dispatch (job-dispatch)   [100%] 1 of 1 ✔
[-        ] process > preprocessing (preprocessing) -
[-        ] process > spikesort_kilosort4           -
[-        ] process > postprocessing                -
[-        ] process > curation                      -
[-        ] process > visualization                 -
[-        ] process > results_collector             -
[-        ] process > quality_control               -
[-        ] process > quality_control_collector     -
[88/6d7163] process > nwb_subject (nwb-subject)     [100%] 1 of 1 ✔
[77/1bd9b6] process > nwb_ecephys (nwb-ecephys)     [100%] 1 of 1, failed: 1 ✘
[-        ] process > nwb_units                     -
Error executing process > 'nwb_ecephys (nwb-ecephys)'

Caused by:
  Process `nwb_ecephys (nwb-ecephys)` terminated with an error exit status (137)

Command executed:

  #!/usr/bin/env bash
  set -e
  
  mkdir -p capsule
  mkdir -p capsule/data
  mkdir -p capsule/results
  mkdir -p capsule/scratch
  
  if [[ local == "slurm" ]]; then
      echo "[nwb-ecephys] allocated task time: null"
  fi
  
  echo "[nwb-ecephys] cloning git repo..."
  git clone "https://github.com/AllenNeuralDynamics/aind-ecephys-nwb.git" capsule-repo
  git -C capsule-repo -c core.fileMode=false checkout c5555443f8c9f73115147516ca539b07b2aee383 --quiet
  mv capsule-repo/code capsule/code
  rm -rf capsule-repo
  
  echo "[nwb-ecephys] running capsule..."
  cd capsule/code
  chmod +x run
  ./run --params '{"stub":false,"stub_seconds":10,"write_lfp":true,"write_raw":false,"lfp_temporal_factor":2,"lfp_spatial_factor":4,"lfp_highpass_freq_min":0.1,"surface_channel_agar_probes_indices":"","lfp":{"filter":{"freq_min":0.1,"freq_max":500},"sampling_rate":2500}}'
  
  echo "[nwb-ecephys] completed!"

Command exit status:
  137

Command output:
  [nwb-ecephys] cloning git repo...
  [nwb-ecephys] running capsule...
  
  
  NWB EXPORT ECEPHYS
  Running NWB conversion with the following parameters:
  Stub test: False
  Stub seconds: 10.0
  Write LFP: True
  Write RAW: False
  Temporal subsampling factor: 2
  Spatial subsampling factor: 4
  Highpass filter frequency: 0.1
  Surface channel indices for agar probes: 
  NWB backend: zarr