Skip to content

Tutorial: Reticle‐to‐Probe Transform from Triangulated Correspondence Points

Jump to bottom
Chris Chronopoulos edited this page Aug 12, 2023 · 6 revisions

This page is up-to-date as of Parallax version 0.28.2

This tutorial walks you through a procedure which is a major raison dêtre for Parallax: establishing a transform from world coordinates (be they headframe, CCF, etc.) to probe coordinates, for an arbitrarily placed probe, without having to manually register correspondence points through probe contact.

For this tutorial you'll need:

  • A calibration checkerboard
  • A targeting reticle, or some target with precisely defined fiducial markers
  • One probe module (Neuropixels probe mounted on a New Scale stage)
  • Two camera modules (FLIR Blackfly cameras with long working distance optics)

Instructions

  1. Set up your rig a. Mount the targeting reticle at the center of your rig. On the Modular Insertion System (MIS), this will be the headframe mount at the center of the table. b. Mount your cameras so they are pointing at the reticle. They should point from different angles - 90 degrees is ideal for triangulation accuracy, but smaller angles are also acceptable. c. Make sure the probe can be positioned so that it reaches the reticle and can navigate its extent d. Arrange a way hold the calibration checkerboard in both cameras' fields of view simultaneously, while allowing for articulation of the checkerboard through various angles. We suggest two optical posts joined with a right-angle connector.

PXL_20230812_044219569

  1. Start up Parallax.

  2. Right-click each of the two Viewports in the Main Window, and select Parallax > Cameras > (camera serial number) for each camera. Which camera streams to which Viewport is not important, but it must be consistent throughout this procedure.

  3. Focus your cameras so that they have as much of the targeting reticle in focus as possible.

  4. Place the checkerboard within view of both cameras. It should be entirely visible, and mostly in focus, in both views. Ideally it should span a fraction (~1/3) of the camera field of view, so that we can move it around to collect corner points.

checkerboard

Don't adjust the camera focus for the checkerboard - you want the cameras focus on the reticle which, you did in step 4.

  1. Click Tools > Checkerboard Tool (stereo) to launch the Stereo Checkerboard Tool. Again, right-click on the Viewports and set the cameras (in the same order as in step 3). In each Viewport, you should see a rainbow-colored figure highlighting the corner points of the checkerboard:

checkerboard_rainbow

  1. Right-click each Viewport and select Parallax > Filters > Checkerboard (smooth). This switches to the more accurate (but slower) checkerboard detection algorithm for data collection.

  2. Click "Grab Corners" in the Stereo Checkerboard Tool to save the corner set. You should see the pose count in the "Save Corners" button increase to 1.

one_pose

  1. Move the checkerboard to a different position and orientation, ensuring that it is still in view and in focus in both cameras. If the corners cannot be detected, you will know because the rainbow highlight overlay will disappear. Once the corner detection overlay has settled (there is a 5-frame latency due to the smoothing algorithm) click "Grab corners" to grab another corner set.

newpose

  1. Repeat Step 9 until you've collected at least 25 corner sets from different orientations and sub-regions of the fields of view.

  2. Click "Save Corners" to save the corner set. A timestamped filename is suggested, but feel free to call it what you like.

  3. Close the Checkerboard Tool and return to the Main Window. Go to Tools > Calibration from Stereo Corners. This will launch a new window:

calfromstereo

  1. In the new window, click "Load" and select the corners file you saved in Step 11. Enter a name for the calibration you wish to generate (or leave the suggested, timestamped name). Leave the "provide intrinsics" box un-checked. Click "Generate Calibration". Close the "Calibration from Stereo Corners" tool. In the Main Window, you should now see your new calibration in the Calibration Panel:

newcal

  1. Remove the checkerboard from view and return to the targeting reticle.

NOTE: In this tutorial, I will re-use the checkerboard as my targeting reticle - the procedure is the same for both, but where I am using checkerboard corners as fiducial points, you will be using ticks on the reticle)

  1. From the Main Window, go to Tools > Rigid Body Transform Tool; this will open a new window:

rbt

  1. In the new window, enter the name of your global coordinate system ("reticle" or "headframe" or similar) in the "From Coordinate System" field. Enter "camera" (or similar) in the "To Coordinate System" field. In "Transform Name", pick a name for the new transform you are about to create - perhaps "reticle2camera" or similar. This is how your transform will later be identified in the Transform Panel.

  2. Make sure your calibration from Step 11 is selected in the Calibration Panel. In the Main Window Viewports, select a fiducial point in both views by left-clicking (you may wish to zoom in quite a bit - use the mousewheel for this). Click "Triangulate" in the Calibration Panel (or hotkey T on the keyboard). You should see the triangulated value of the fiducial point reported in the message log:

triangulated

  1. In the Rigid Body Tool again, enter the reticle coordinates of this fiducial point in the "From" coordinate fields. Then enter the triangulated coordinates in the "To" coordinate fields. Click the "right arrow" icon to register this data to your point set; it will disappear from the left panel and appear in the list on the right.

HINT: you can drag-and-drop the triangulated coordinates from the Calibration Panel into the Rigid Body Transform tool.

  1. Repeat step 18 for several fiducial points (at least 5).

  2. Once you have collected you correspondence points for the reticle-to-camera transform, it's a good idea to save them - in case you want to add to them later, or use them in other Analyses. Click "Save" and enter a filename; your correspondence points will be saved in CSV format.

  3. Now, click "Generate Transform" in the Rigid Body Transform tool. You should see your transform appear in the Transform Panel on the Main Window:

newtransform

Click "Save" to save the transform to disk.

  1. Now that you have one transform computed, let's do it again for the second. Get a new transform ready by closing and re-opening the Rigid Body Transform Tool (or you can clear the fields one-by-one). Enter new metadata:

a. "From Coordinate System" : "camera" b. "To Coordinate System" - "probe", or "46798" (the stage serial number) if you have many probes you are calibating c. "Transform Name" - "camera2probe" or similar

  1. Now, place the probe into view from both cameras. If you're on the Modular Insertion System, it's a good idea to raise the Elevator slightly before doing this - that way you'll have plenty of room to move the probe clear of the reticle. Use the Stage Control Panel to roughly center the probe in both fields of view - make sure it is in good focus, too.

probe_centered

  1. We will use the probe calibration routine to maneuver the probe through the correspondence points for the next transform. You can maneuver it manually, but this is more convenient. From the Calibration Panel, click "Start"; this will launch the dialog for the calibration routine:

cal_dialog

Fill in the calibration routine parameters:

a. select the stage's serial from the dropdown b. use a resolution of 2 for 8 correspondence points (2^3 = 8) c. enter the spatial extent that the routine will move the probe through (6000 microns is good for our working distance) d. click "Set current position as origin" to grab the current stage position e. give a name like "dummy" to avoid confusion with your actual calibrations f. click "Start Calibration Routine"

  1. The routine will move the probe to its first correspondence point, and then prompt you in the message log "Highlight correspondence points and press C to continue". Zoom in on the Viewports and left-click the probe tip in both views:

probe_tips

Don't press C yet; first you want to triangulate that image point. Click Triangulate (or press T) - you should see the triangulated value reported in the message log.

  1. Enter the triangulated coordinates into the "From" fields of the Rigid Body Transform Tool (or, drag and drop from the Calibration Panel). Enter the stage's motor coordinates into the "To" fields (or, drag and drop from the Stage Control Panel). Click the right-arrow to register this point.

  2. Repeat Steps 25 and 26 for all 8 correspondence points.

  3. Click "Generate Transform"

  4. Now, go to the "From Composition" tab of the Rigid Body tool. Select your first transform "reticle2camera", and click Add. Then select your second transform "camera2probe" and click Add. Give the composited transform a name (perhaps "reticle2probe") and click Generate. The newly transform will appear in the Transform Panel. Click "Save" there to save it to disk.

  5. That's it! You've generated a coordinate transform from reticle coordinates to probe coordinates. To test it, make sure it is selected in the Transform Panel, and click Apply. This will open a new window that you can use to transform between the two systems.

apply_transform

On the left, enter the coordinates of a new fiducial point in the reticle system, and click "map forward" - the transformed coordinates (in the probe system) will appear on the left. You can manually drive the probe tip to this fiducial point and confirm that the motor coordinates agree. Once you are confident in the accuracy of the transform, you can simply drag-and-drop coordinates from the Apply Transform widget to the Targeting Dialog, and the probe will navigate to your reticle points automatically.

Clone this wiki locally