Project Overview

programming Computer Graphics

Goal – What are we going to do?

The importance of statistical learning methods in computer graphics research has increased tremendously over the last decade due to the rise of powerful computational resources and the massive accumulation of image data on the internet. As we have seen in the class, recapturing classic problems in computer graphics in terms of statistical learning has brought us novel insights. For example, Automatic Photo Pop Up uses decision trees to classify geometric cues in an image, which allows this system to reconstruct 3D image based on a single picture instead of multiple images. ShadowDraw hugely relies on Content Based Image Reitreival, which utilizes a number of statistical algorithms such as nearest neighbor matching. The main idea behind the paper "HelpingHand: Example Based Stylization" was non-parametric modeling, which is used to selectively copy portions of a library, generating new data that maintains a similar statistical distribution with the given image data.

In "Sketch Classification and Classification-driven Analysis using Fisher Vectors," they use Fisher Vectors to characterize a sample image from a distribution by its deviation from the generative model. In this project, we will focus on Bag-of-Visual-Words---another method for image classification that is one of the most widely-used techniques for image classification as the paper mentions in the section 2.1 Related Work: Image Classification. Specifically, we will develop a content based image retrieval system by using Bag-of-Visual-Words to retrieve similar images given an input image data.


Previous Work – What related work has been done?

  • As described above ShadowDraw, Automatic Photo Pop Up, HelpingHand, and Sketch Classification.

Others:

  • "How do humans sketch objects?" (EITZ, M., HAYS, J., AND ALEXA, M. 2012.): develop a bag-of-features sketch

representation and use multi-class support vector machines, trained on thier sketch dataset, to classify sketches

  • "Sketch Based Object Recognition" (ZHU,B., QUIGLEY, E. Stanford CS231a)
  • "Visual Categorization with Bags of Keypoints" (CSURKA.G, DANCE.C, 2004)
  • "A symmetry-aware flip invariant sketch histogram shape descriptor" (CAO, X., ZHANG, H., LIU, S., GUO, X., AND LIN, L. 2013.)


Approach – What approach are we going to try? Why do we think it will work well?

  • Clean the data set for the sake of efficiency (Delete some images that might confuse the classification algorithm)
  • Extract SURF features from the data set
  • Cluster each SURF vector (corresponding to each image) into k categories via K-means
  • Calculate Visual-Words for each cluster
  • Transform each image into histogram of Visual-Word
  • Use the histogram above to search a similar image
  • Experiments. Observe the results. Comment for future work.


Methodology – What steps (task list) are required? Which of these steps in particular is hard? Contingencies– what to do if the hard steps don’t work out?

  • Understand the learning algorithm in detail to the point where we can actually implment.
  • Learn OpenCV as fast as possible (while doing the project)
  • Need to find a moderate amount of image data set so that we can perform multiple tests and experiments in a short amount of time
  • If we have time, we could experiment a few different ways of measuring similarity to find which is the best; But for now, we will use Histogram Intersection for our similarity measure function
  • The hardest part would be implmentation. Contingencies: Will try to fix bugs as much as possible.


Metrics – How we will we know when we’re done? How will we know whether we have succeeded?

  • The goal is to finish implementing the entire search system from scratch.
  • When the search system outputs reasonable results, it means we have succeeded.


Future Work
We have limited our work in just creating a search engine for the sake of completing the project in a limited time. Often times, this sort of search engine is used in many computer graphics research. For future work, we hope to develop this system to use in an actual research relating to image classification.


Summary – What we will learn by doing this project?

  • We will learn the role of statistical learning, particularly in content image based retreival through OpenCV implementation of image classification. We hope that the knowledge and the experience we will get through this project would benefit us in our future exploration in a wide range of research areas including computer graphics, computer vision, and machine learning.