Entry 19: Color Image Segmentation

Images contain many information that can be retrieved through image processing.  Some of the information can be used to discriminate certain regions (regions of interest - ROI) as the important regions and put other regions not containing the specified region to a null value forcing other points to be a part of the background.

In this activity, we perform the method of color image segmentation to identify in the entire image the regions with the same information with the reference patch.  For this to be done, normalization of the color channels was performed given by the equation below.

 

We studied the two types of color image segmentation techniques, the parametric and the non-parametric color segmentation.  The parametric segmentation method utilizes the Gaussian distribution wherein the probability of the image value depends on the reference patch.  The mean of each color channel was used to be the abscissa with the highest probability.  The Gaussian distribution used in the parametric technique is stated below,

 

the spread and mean parameters of the distribution comes from the standard deviation and the mean value of the reference patch, respectively.

We used the technique of color image segmentation to the image below (My Toy Car :D) shown in Fig. 1,

Figure 1:  Image to be segmented

It is important to realize that we utilizes the Normalized Chromaticity Space in representing the location of the values to be calculated from the patch.  Also, this chromaticity space will help us visualize quantitatively what colors are present in the image.  A sample normalized chromaticity graph is shown below in Fig. 2,

Figure 2:  Sample Normalized Chromaticity Space

To continue with the parametric image segmentation, we chose a patch in the ROI to represent the entire region.  The patch chosen appear to have a large color component of yellow Fig. 3.

Figure 3:  Reference color patch for color image segmentation.

After doing the algorithm of taking the Gaussian probabilities and multiplying the results with each other, we get a parametrically segmented version of Fig. 1 as shown in Fig. 4.

Figure 4:  Segmented image through parametric color image segmentation.

Our next task is to segment the original image using the non-parametric approach.  In this technique, it requires the knowledge of the histogram of the normalized chromaticity space for the entire image and the patch representing the ROI.

For the image to be segmented, we calculated the 2D normalized chromaticity space histogram representing the entire image.  The computed histogram is shown in Fig. 4 along with the normalized chromaticity space representing the entire color spectrum.

Figure 4:  Left) The normalized chromaticity histogram representing the image, Right) the normalized chromaticty space fot the entire color spectrum.

We also computed the normalized chromaticity histogram for the patch chosen to represent the ROI.  In Fig. 5, we superimposed the computed normalized chromaticity histogram of the ROI to the normalized chromaticity space representing the entire color spectrum to properly visualize the color component representing the ROI.

Figure 5:  Superimposed normalized chromaticity histogram of the chosen representative region for the ROI with the normalized chromaticity space of the entire color spectrum.

It can be observed in Fig. 5 that the representative color of the ROI is near the yellow-orange region of the normalized chromaticity space of the entire spectrum.

We performed histogram backprojection which will yield the non-paramtrically segmented image.  The resulting non-parametrically segmented image is shown in Fig. 6.

Figure 6:  Resulting non-parametrically segmented image.

It is so handy to the techniques performed above to identify regions of interest.  Many applications of these methods can be made such as tracing the trajectory of certain material.

In this activity I want to give myself a 10/10 because of my efforts and enthusiasm in doing the activity.  I learned much in this activity which is  the most important thing for me.

Source:

    Activity Sheet for Activity 13 - Dr. Maricor Soriano

Entry 18: Color Camera Processing

Before this I took this course I was so naive about some things and included in those things is the concept of white balance.

I hear the term before but I never even bother to take some time to Google any information about it.  But after performing this activity, I truly appreciated the white balancing algorithms as they transform images taken with different light sources to maintain their color signatures.

In this activity, we explored two common method being utilized in white balancing namely, White Patch Algorithm and the Gray World Algorithm.

We first take images of a single scene with a white material in it using different white balancing modes in a digital camera. 

Given these images, we sampled an image with the least appropriate white balance setting for the lighting condition during the image was taken. From this data, we can start implementing the two algorithms mentioned above.

First, we implemented the White Patch Algorithm.  A requirement of the White Patch Algorithm is that a white patch must be present in the image to be a reference or the normalizing value.  Thus, we chose from the image a pixel with white value and inputted it to the algorithm.  Shown in Fig. 1 is the image captured under a sunlight lighting with a white balance mode of "Incandescent" set in the digital camera.

Figure 1:  Image with inappropriate white balance setting (Incandescent) used in the lighting condition (Sunlight).

The White Patch Algorithm simply normalizes the entire image with the chosen white patch from the image itself.  After the normalization, we formed the image array and reconstructed the image.  The result is shown in Fig. 2 - White Patch Algorithm applied to the original image (Fig. 1).

Figure 2:  Resulting image after the White Patch Algorithm has been applied to the original image.

It is quite noticeable that a significant change in the image can be observed.  The white bond paper lost the "bluish stain" after the algorithm was applied and it appears to be very pure white.  The other colors can be seen to have improved and look more like in the real world.

The next algorithm that will be performed is the Gray World Algorithm.  In this method, the mean of each color channel is the variable used to normalize each of the values in the respective channels.  We performed this technique to the original image and the result is shown in Fig. 3,

Figure 3:  Resulting image after the Gray World algorithm was applied to the original image.

In Fig. 3, the resulting image appear to be have a higher intensity but the white bond paper looks so pure white.  On the other hand, the other colored objects seem to be flooded by the white contrast.

The next aspect we looked at is the accuracy and efficiency of the two algorithms.  To do this, we took an image of collection of object sharing the same shade of hue with no white in the background.  The image used is shown in Fig. 4,

Figure 4:  Image with inappropriate white balance (Incandescent) with objects sharing the same shade of hue under a fluorescent lighting.

We performed both algorithms in this image and the results are shown in Fig. 5,

Figure 5:  Left) White Patch Algorithm, Right) Gray World Algorithm applied to the original image.

We have observed that the White Patch Algorithm is better used when the colors in the spectrum is not well represented in the image.  Even though the image appeared grainy after the application of the White Patch Algorithm it still appeared more crisp relative to the result after the gray World Algorithm has been applied to the images.

The possible reason why the White Patch Algorithm is suited in the case when only a small portion of the color spectrum is represented in the image is that for the Gray World Algorithm, the mean of the entire color channel is being used in the normalization, thus, since other colors are not represented the result of the normalization will probably not give the true white balance for the image.

In the succeeding figure, the results of  the implementation of both algorithms are shown for various white balance setting used in capturing the scene.  In the first column of the figures below show the images after the White Patch Algorithm has been implemented to the original images which are those in the middle column, while the last column corresponds to the result after the Gray World Algorithm has been applied.

Figure 6:  Left) White Patch Algorithm applied to the original image; Middle) Original image under "Cloudy" white balance mode; Right) Gray World Algorithm applied to the original image.

Figure 7:  Left) White Patch Algorithm applied to the original image; Middle) Original image under "Fluorescent 1  " white balance mode; Right) Gray World Algorithm applied to the original image.

Figure 8:  Left) White Patch Algorithm applied to the original image; Middle) Original image under "Fluorescent 2" white balance mode; Right) Gray World Algorithm applied to the original image.

Figure 9:  Left) White Patch Algorithm applied to the original image; Middle) Original image under "Fluorescent 3" white balance mode; Right) Gray World Algorithm applied to the original image.

Figure 10:  Left) White Patch Algorithm applied to the original image; Middle) Original image under "Incandescent" white balance mode; Right) Gray World Algorithm applied to the original image.

In this activity I would give myself a 10/10 for the completeness of the result despite of late posting.

Source:
     Activity Sheet for Activity 12  - Dr. Maricor Soriano

Entry 17: Playing Notes by Image Processing

When I was in elementary, I started to learn the basics of Music Theory.  I learned to read notes and other musical symbols.  I practiced how to play the piano.  I was really happy when I was able to hear the music coming from the piano.

Another happy music moment was when I started to study playing the violin.  I learned to play the violin in tutorials available in the Internet.  By then, I started saving money to buy my own violin.

This time around, again I felt a happy music moment when I knew that this activity will be about playing musical pieces through image processing. 

The goal of this activity was to "read" notes and "play" them using all of the methods learned in the previous activities.

To perform this, we first chose a simple musical piece - Twinkle Twinkle Little Star.  Next, to further simplify the process, we cropped the staffs from the musical piece (Fig. 1).

Figure 1:  Sample cropped staff from the music sheet.

With the staff in hand, we performed custom methods to classify the notes in the staff.  The critical information in this piece of image are the pitch of each note and the length of each note playing time.  Rests are not present in this piece so our task was a lot easier.

We cleaned the staff image and introduced morphological operations (successive erosion and closing).  The goal of this task is to characterize the notes on the piece and get their individual pitch and playing time information.

Figure 2:  Sample cleaned staff using erosion and closing.

We took the coordinates of the center of mass of each blob and compared it against the "known" coordinates of the staff's ledger lines.  This information will give us the pitch of the note.  Also, the area of the blobs representing the notes were taken to determine the type of note (i.e. quarter or half note).  Notice that half notes have a smaller area than the quarter note, thus, we can differentiate one from the other.

After the gathering of all the informations in the sheet, we inputed them in the function wavwrite() which converts an array of sinusoids with frequencies corresponding to each musical notes to a .wav music file.

After all of the procedures, finally the piece was already converted into a musical format and we were able to play it in our computer!  I used PiTiVi Video Editor in Ubuntu to render the video.

In this activity I would give myself a grade of 10/10 because of my efforts in finishing this activity.  Also, I have found an easy way of reading the piece without using imcorrcoef().

Source:
    Activity Sheet for Activity 11 - Dr. Maricor Soriano