Description

PS7

Weight: 5% of total grade
Note: You have two weeks to complete this problem set.
PS7 Stereo Vision

Stereo vision is used to reconstruct 3D information from multiple images. In its simplest form of two-camera parallel stereo vision, the depth for each pixel can be calculated based on the disparity between two images, one from the left camera and the other from the right camera (see Figure 1).
Figure 2 shows an example of two images taken by parallel stereo cameras and their disparity. The disparity map shows objects closer to the cameras with lighter gray colors.
In this problem set, you will learn how to reconstruct 3D shapes from parallel stereo images and visualize the result as a colored 3D pointcloud file in the PLY format.

Figure 1: Parallel Stereo Vision – the depth value for each pixel is calculated based on the disparity.

Figure 2: Images taken by parallel stereo cameras and their disparity grayscale image.

Figure 3: Three sets of parallel stereo images and their 3D reconstruction results

Write a program that:
(1) takes as input a pair of parallel stereo images, left and right, e.g., “baby-left.png” and “baby-right.png,”
(2) generates a disparity grayscale image using the left camera position – name the image file “babydisparity.png,” and
(3) generates a colored 3D pointcloud file in the PLY format – name the PLY file “baby.ply.”
To render a 3D view of your PLY file, use CloudCompare (https://www.danielgm.net/cc/), an open-source pointcloud and mesh processing software. Appendix A shows an example of a colored PLY file.
As we discussed in class, you will need to know camera parameters to convert a disparity grayscale pixel value to a depth value. In this problem set, however, do not worry about the true depth values and use the reconstructed 3D shape of the bowling ball to find an appropriate mapping from the disparity grayscale and the depth – find the scaling ratio that makes the reconstructed ball surface look like a sphere. Once you find the ratio, apply it to the other two pairs of images.
Take a pair of parallel stereo images using your mobile phone camera or a digital camera and apply your program to create its disparity grayscale image and PLY file. Name the files: “andrewid-left.png,” “andrewidright.png,” “andrwid-disparity.png,” and “andrewid.ply.” (Replace “andrewid” with your own Andrew ID.)
Hint: use the block matching (BM) and/or the semi-global block matching (SGBM) in OpenCV for finding the disparity.

Submission
To prepare for the submission of your work on Gradescope, create:
(1) a folder called “ps7-1,” that contains the following files:
• source code file(s)
• two parallel stereo images that you took for testing your code o andrewid-left.png, andrewid-right.png
• three disparity map files created by your program: o baby-disparity.png, plant-disparity.png, ball-disparity.png, and andrewid-disparity.png
• three PLY files created by your program: o baby.ply, plant.ply, ball.ply, and andrewid.ply
• “readme.txt” file that includes:
o Operating system o IDE you used to write and run your code o The number of hours you spent to finish this problem
(2) a PDF file that contains the printouts and screenshots of all the files in the ps7-1 folder. Include the explanation of what modifications you made to detect all five types of mechanical parts. (Include, if any, the mathematical derivation and/or description of your method in the PDF file. Handwritten notes should be scanned and included in the PDF file.)

Submit your work on Gradescope

Submit two files on Gradescope – replace “andrewid” with your own Andrew ID:
(1) andrewid-ps7-files.zip – this ZIP file should contain all the files requested in PS7-1.
Please make sure that files are organized in an easy to understand sub-folder structure.
(2) andrewid-ps7-report.pdf – this PDF file serves as the report of your work, and it should contain the printouts and screenshots of all the files in the “ps7-1” folder. (Include, if any, the mathematical derivation and/or description of your method in the PDF file. Handwritten notes should be scanned and included in the PDF file.)
Please organize pages with section titles and captions to make the report easy to read.

Appendix A
ply format ascii 1.0 comment – Made for CVE
comment – This file represents a cube’s corner vertices element vertex 8 property float32 x property float32 y property float32 z property uint8 red property uint8 green property uint8 blue end_header
0 0 0 255 255 255
0 0 1 0 0 255
0 1 1 0 255 0
0 1 0 0 255 255 1 0 0 255 0 0
1 0 1 255 0 255
1 1 1 255 255 0
1 1 0 128 128 128