Implement first version of QA-turned-QC code for ROI mis-locating

[alexpiet]

No description provided.

[Ahad-Allen]

@hagikent to coordinate with @alexpiet about running the code for this

[alexpiet]

Create QC jsons, and add to metadata folder, and then they will show up in the QC portal

[hagikent]

Draft done (a reproducible capsule can be found here). The ideas is to visualize RawData (binary movie) + Experimenter Defined ROIs to check if ROIs are properly covering each fibre (+ empty space for floor signal recording).

UI during experiments

QA/QC view for a corresponding session: (clearly, ROIs in the Red channel are off)

Several discussion points:
1.Do we want to run on DAQ PC? or as a part of QC capsule?
2.Do we want to delete RawData once we confirm ROIs are ok ( = online extracted TCs in CSVs are valid)? This can save ~20GB per session. A while ago @dyf suggested deleting them. For deletion purpose, running on DAQ PC before watchdog-uploading would sound more logical.

[dyf]

Agree, I would create a quality_control.json using code run on the DAQ PC.

If you would prefer to defer this to the upload process, we can add a "job type" to the Data Transfer Service that computes this on the HPC before upload.

[alexpiet]

Makes sense to run on the acquisition computer. @hagikent Can this check run before the session runs so we can address the issue before we collect data? Or is the idea to detect issues that happen during the session?

[hagikent]

@dyf
This routine is super light-weight and no advantage of using clusters.
We should rather do it locally so that we can exclude rawdata from watchdog uploading (this part I need help from @alexpiet )

@alexpiet

to detect issues that happen during the session?
rather this.

---

Ok, then I will ditch CO and make a local .py version of this including generating QCs corresponding to this QA.

[hagikent]

Practically, it's impossible to do QA while having human in the loop before uploading. Thus, we agreed to make this just a part of QC pipeline. I will work on integrating the draft to the fip-raw-qc capsule and also set up a capsule to perform signal re-extraction (new ToDo item here: https://github.com/AllenNeuralDynamics/aind-physio-arch/issues/516).

[dougollerenshaw]

@hagikent are you still working on this? Any status update you can provide?

[hagikent]

Yes! When I have time to work on pipeline stuff, this would be the 1st thing I would do! Likely next week.