OpenCV Mouth Detection using Haar Cascades

In this OpenCV with Python tutorial, we’re going to discuss object detection with Haar Cascades. We’ll do mouth detection to start. In order to do object recognition/detection with cascade files, you first need cascade files. For the extremely popular tasks, these already exist. Detecting things like faces, cars, smiles, eyes, and license plates for example are all pretty prevalent.
First, I will show you how to use these cascade files, then I will show you how to embark on creating your very own cascades, so that you can detect any object you want, which is pretty darn cool!
You can use Google to find various Haar Cascades of things you may want to detect. You shouldn’t have too much trouble finding the aforementioned types. We will use a Face cascade , Mouth cascade , Nose Cascade and Eye cascade. You can find a few more at the root directory of Haar cascades. Note the license for using/distributing these Haar Cascades.
Let’s begin our code. I am assuming you have downloaded the haarcascade_eye.xml, haarcascade_frontalface_default.xml, haarcascade_mcs_mouth.xml and haarcascade_mcs_nose.xmlfrom the links above, and have these files in your project’s directory.

Understanding the code

Let’s dive straight into the code

import sys
import cv2
# Get user supplied values
imagePath = sys.argv[1]
cascPath = sys.argv[2]
# Create the haar cascade
mouthCascade = cv2.CascadeClassifier(cascPath)
# Read the image
image = cv2.imread(imagePath)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
# Draw a rectangle around the faces
roi_gray = gray[y:y+h, x:x+w]
roi_color = image[y:y+h, x:x+w]
mouth = mouthCascade.detectMultiScale(roi_gray)
for (ex,ey,ew,eh) in mouth:
    cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)
cv2.imshow("Mouth found" ,image)
cv2.waitKey(0) 

Let’s break down the actual code.

# Get user supplied values
imagePath = sys.argv[1]
cascPath = sys.argv[2]

You first pass in the image and cascade names as command-line arguments. We’ll use the Abba image as well as the default cascade for detecting mouth provided by OpenCV.

# Create the haar cascade
mouthCascade = cv2.CascadeClassifier(cascPath)

Now we create the cascade and initialize it with our mouth cascade. This loads the mouth cascade into memory so it’s ready for use. Remember, the cascade is just an XML file that contains the data to detect mouth.

# Read the image
image = cv2.imread(imagePath)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

Here we read the image and convert it to grayscale. Many operations in OpenCv are done in grayscale.

# Detect faces in the image
mouth = mouthCascade.detectMultiScale(roi_gray)

The function returns a list of rectangles where it believes it found mouth. Next, we will loop over where it thinks it found something.

for (ex,ey,ew,eh) in mouth:
    cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)

This function returns 4 values: the ex and ey location of the rectangle, and the rectangle’s width and height (ew , eh).
We use these values to draw a rectangle using the built-in rectangle() function.

cv2.imshow("Mouth found" ,image)
cv2.waitKey(0)

Checking the results

Let’s test against the Abba photo:

$ python mouth_detect.py lena.png haarcascade_mcs_mouth.xml