Entry 17: Playing Notes by Image Processing

When I was in elementary, I started to learn the basics of Music Theory.  I learned to read notes and other musical symbols.  I practiced how to play the piano.  I was really happy when I was able to hear the music coming from the piano.

Another happy music moment was when I started to study playing the violin.  I learned to play the violin in tutorials available in the Internet.  By then, I started saving money to buy my own violin.

This time around, again I felt a happy music moment when I knew that this activity will be about playing musical pieces through image processing. 

The goal of this activity was to "read" notes and "play" them using all of the methods learned in the previous activities.

To perform this, we first chose a simple musical piece - Twinkle Twinkle Little Star.  Next, to further simplify the process, we cropped the staffs from the musical piece (Fig. 1).

Figure 1:  Sample cropped staff from the music sheet.

With the staff in hand, we performed custom methods to classify the notes in the staff.  The critical information in this piece of image are the pitch of each note and the length of each note playing time.  Rests are not present in this piece so our task was a lot easier.

We cleaned the staff image and introduced morphological operations (successive erosion and closing).  The goal of this task is to characterize the notes on the piece and get their individual pitch and playing time information.

Figure 2:  Sample cleaned staff using erosion and closing.

We took the coordinates of the center of mass of each blob and compared it against the "known" coordinates of the staff's ledger lines.  This information will give us the pitch of the note.  Also, the area of the blobs representing the notes were taken to determine the type of note (i.e. quarter or half note).  Notice that half notes have a smaller area than the quarter note, thus, we can differentiate one from the other.

After the gathering of all the informations in the sheet, we inputed them in the function wavwrite() which converts an array of sinusoids with frequencies corresponding to each musical notes to a .wav music file.

After all of the procedures, finally the piece was already converted into a musical format and we were able to play it in our computer!  I used PiTiVi Video Editor in Ubuntu to render the video.

In this activity I would give myself a grade of 10/10 because of my efforts in finishing this activity.  Also, I have found an easy way of reading the piece without using imcorrcoef().

Source:
    Activity Sheet for Activity 11 - Dr. Maricor Soriano