Python real time image classification problems with Neural Networks
<p>I'm attempting use caffe and python to do real-time image classification. I'm using OpenCV to stream from my webcam in one process, and in a separate process, using caffe to perform image classification on the frames pulled from the webcam. Then I'm passing the result of the classification back to the main thread to caption the webcam stream.</p>

<p>The problem is that even though I have an NVIDIA GPU and am performing the caffe predictions on the GPU, the main thread gets slown down. Normally without doing any predictions, my webcam stream runs at 30 fps; however, with the predictions, my webcam stream gets at best 15 fps. </p>

<p>I've verified that caffe is indeed using the GPU when performing the predictions, and that my GPU or GPU memory is not maxing out. I've also verified that my CPU cores are not getting maxed out at any point during the program. I'm wondering if I am doing something wrong or if there is no way to keep these 2 processes truly separate. Any advice is appreciated. Here is my code for reference</p>

<pre><code>class Consumer(multiprocessing.Process):

    def __init__(self, task_queue, result_queue):
        multiprocessing.Process.__init__(self)
        self.task_queue = task_queue
        self.result_queue = result_queue
        #other initialization stuff

    def run(self):
        caffe.set_mode_gpu()
        caffe.set_device(0)
        #Load caffe net -- code omitted 
        while True:
            image = self.task_queue.get()
            #crop image -- code omitted
            text = net.predict(image)
            self.result_queue.put(text)

        return

import cv2
import caffe
import multiprocessing
import Queue 

tasks = multiprocessing.Queue()
results = multiprocessing.Queue()
consumer = Consumer(tasks,results)
consumer.start()

#Creating window and starting video capturer from camera
cv2.namedWindow("preview")
vc = cv2.VideoCapture(0)
#Try to get the first frame
if vc.isOpened():
    rval, frame = vc.read()
else:
    rval = False
frame_copy[:] = frame
task_empty = True
while rval:
    if task_empty:
       tasks.put(frame_copy)
       task_empty = False
    if not results.empty():
       text = results.get()
       #Add text to frame
       cv2.putText(frame,text)
       task_empty = True

    #Showing the frame with all the applied modifications
    cv2.imshow("preview", frame)

    #Getting next frame from camera
    rval, frame = vc.read()
    frame_copy[:] = frame
    #Getting keyboard input 
    key = cv2.waitKey(1)
    #exit on ESC
    if key == 27:
        break
</code></pre>

<p>I am pretty sure it is the caffe prediction slowing everything down, because when I comment out the prediction and pass dummy text back and forth between the processes, I get 30 fps again.</p>

<pre><code>class Consumer(multiprocessing.Process):

    def __init__(self, task_queue, result_queue):
        multiprocessing.Process.__init__(self)
        self.task_queue = task_queue
        self.result_queue = result_queue
        #other initialization stuff

    def run(self):
        caffe.set_mode_gpu()
        caffe.set_device(0)
        #Load caffe net -- code omitted
        while True:
            image = self.task_queue.get()
            #crop image -- code omitted
            #text = net.predict(image)
            text = "dummy text"
            self.result_queue.put(text)

        return

import cv2
import caffe
import multiprocessing
import Queue 

tasks = multiprocessing.Queue()
results = multiprocessing.Queue()
consumer = Consumer(tasks,results)
consumer.start()

#Creating window and starting video capturer from camera
cv2.namedWindow("preview")
vc = cv2.VideoCapture(0)
#Try to get the first frame
if vc.isOpened():
    rval, frame = vc.read()
else:
    rval = False
frame_copy[:] = frame
task_empty = True
while rval:
    if task_empty:
       tasks.put(frame_copy)
       task_empty = False
    if not results.empty():
       text = results.get()
       #Add text to frame
       cv2.putText(frame,text)
       task_empty = True

    #Showing the frame with all the applied modifications
    cv2.imshow("preview", frame)

    #Getting next frame from camera
    rval, frame = vc.read()
    frame_copy[:] = frame
    #Getting keyboard input 
    key = cv2.waitKey(1)
    #exit on ESC
    if key == 27:
        break
</code></pre>

<p><strong>Some Explanations and Some Rethinks:</strong></p>

<ol>
<li><p>I ran my code below on a laptop with an <code>Intel Core i5-6300HQ @2.3GHz</code> cpu, <code>8 GB RAM</code> and <code>NVIDIA GeForce GTX 960M</code> gpu(2GB memory), and the result was: </p>

<p>Whether I ran the code with caffe running or not(by commenting out or not <code>net_output = this-&gt;net_-&gt;Forward(net_input)</code> and some necessary stuff in <code>void Consumer::entry()</code>), I could always get around 30 fps in the main thread.</p>

<p>The similar result was got on a PC with an <code>Intel Core i5-4440</code> cpu, <code>8 GB RAM</code>, <code>NVIDIA GeForce GT 630</code> gpu(1GB memory).</p></li>
<li><p>I ran the code of <a href="http://stackoverflow.com/q/39522693/6281477">@user3543300</a> in the question on the same laptop, the result was:</p>

<p>Whether caffe was running(on gpu) or not, I could also get around 30 fps.</p></li>
<li><p>According to <a href="http://stackoverflow.com/q/39522693/6281477">@user3543300</a> 's feedback, with the 2 versions of code mentioned  above, @user3543300 could get only around 15 fps, when running caffe(on a <code>Nvidia GeForce 940MX GPU and IntelÂ® Coreâ¢ i7-6500U CPU @ 2.50GHz Ã 4</code> laptop).
And there will also be  a slowdown of frame rate of the webcam when caffe running on gpu as an independent program. </p></li>
</ol>

<p>So I still think that the problem may most possibly lie in hardware I/O limitaions such as gpu memory bandwidth or RAM bandwidth.
Hope <a href="http://stackoverflow.com/q/39522693/6281477">@user3543300</a> can check this or find out the true problem that I haven't realized of.</p>

<p>If the problem  is indeed what I think of above, then a sensible thought would be to reduce memory I/O overhead introduced by the CNN network. In fact, to solve the similar problem on embedded systems with limited hardware resources, there have been some research on this topic, e.g. <a href="https://github.com/wenwei202/caffe/tree/scnn" rel="nofollow">Structurally Sparse Deep Neural Networks</a>, <a href="https://github.com/songhan/SqueezeNet-Deep-Compression" rel="nofollow">SqueezeNet</a>, <a href="https://github.com/songhan/Deep-Compression-AlexNet" rel="nofollow">Deep-Compression</a>. So hopefully, it will also help to improve the frame rate of webcam in the question by applying such skills.</p>

<hr>

<p><strong>Original Answer:</strong></p>

<p>Try this c++ solution. It uses threads for the <a href="https://en.wikipedia.org/wiki/I/O_bound" rel="nofollow">I/O overhead</a> in your task, I tested it using <a href="https://github.com/BVLC/caffe/tree/master/models/bvlc_alexnet" rel="nofollow"><code>bvlc_alexnet.caffemodel</code></a> to do image classification and didn't see obvious slowing down of the main thread(webcam stream) when caffe running(on GPU):</p>

<pre><code>#include &lt;stdio.h&gt;
#include &lt;iostream&gt;
#include &lt;string&gt;
#include &lt;boost/thread.hpp&gt;
#include &lt;boost/shared_ptr.hpp&gt;
#include "caffe/caffe.hpp"
#include "caffe/util/blocking_queue.hpp"
#include "caffe/data_transformer.hpp"
#include "opencv2/opencv.hpp"

using namespace cv;

//Queue pair for sharing image/results between webcam and caffe threads
template&lt;typename T&gt;
class QueuePair {
  public:
    explicit QueuePair(int size);
    ~QueuePair();

    caffe::BlockingQueue&lt;T*&gt; free_;
    caffe::BlockingQueue&lt;T*&gt; full_;

  DISABLE_COPY_AND_ASSIGN(QueuePair);
};
template&lt;typename T&gt;
QueuePair&lt;T&gt;::QueuePair(int size) {
  // Initialize the free queue
  for (int i = 0; i &lt; size; ++i) {
    free_.push(new T);
  }
}
template&lt;typename T&gt;
QueuePair&lt;T&gt;::~QueuePair(){
  T *data;
  while (free_.try_pop(&amp;data)){
    delete data;
  }
  while (full_.try_pop(&amp;data)){
    delete data;
  }
}
template class QueuePair&lt;Mat&gt;;
template class QueuePair&lt;std::string&gt;;

//Do image classification(caffe predict) using a subthread
class Consumer{
  public:
    Consumer(boost::shared_ptr&lt;QueuePair&lt;Mat&gt;&gt; task
           , boost::shared_ptr&lt;QueuePair&lt;std::string&gt;&gt; result);
    ~Consumer();
    void Run();
    void Stop();
    void entry(boost::shared_ptr&lt;QueuePair&lt;Mat&gt;&gt; task
             , boost::shared_ptr&lt;QueuePair&lt;std::string&gt;&gt; result);

  private:
    bool must_stop();

    boost::shared_ptr&lt;QueuePair&lt;Mat&gt; &gt; task_q_;
    boost::shared_ptr&lt;QueuePair&lt;std::string&gt; &gt; result_q_;

    caffe::Blob&lt;float&gt; *net_input_blob_;
    boost::shared_ptr&lt;caffe::DataTransformer&lt;float&gt; &gt; data_transformer_;
    boost::shared_ptr&lt;caffe::Net&lt;float&gt; &gt; net_;
    std::vector&lt;std::string&gt; synset_words_;
    boost::shared_ptr&lt;boost::thread&gt; thread_;
};
Consumer::Consumer(boost::shared_ptr&lt;QueuePair&lt;Mat&gt;&gt; task
                 , boost::shared_ptr&lt;QueuePair&lt;std::string&gt;&gt; result) :
 task_q_(task), result_q_(result), thread_(){

  //for data preprocess
  caffe::TransformationParameter trans_para;
  //set mean
  trans_para.set_mean_file("/path/to/imagenet_mean.binaryproto");
  //set crop size, here is cropping 227x227 from 256x256
  trans_para.set_crop_size(227);
  //instantiate a DataTransformer using trans_para for image preprocess
  data_transformer_.reset(new caffe::DataTransformer&lt;float&gt;(trans_para
                        , caffe::TEST));

  //initialize a caffe net
  net_.reset(new caffe::Net&lt;float&gt;(std::string("/path/to/deploy.prototxt")
           , caffe::TEST));
  //net parameter
  net_-&gt;CopyTrainedLayersFrom(std::string("/path/to/bvlc_alexnet.caffemodel"));

  std::fstream synset_word("path/to/caffe/data/ilsvrc12/synset_words.txt");
  std::string line;
  if (!synset_word.good()){
    std::cerr &lt;&lt; "synset words open failed!" &lt;&lt; std::endl;
  }
  while (std::getline(synset_word, line)){
    synset_words_.push_back(line.substr(line.find_first_of(' '), line.length()));
  }
  //a container for net input, holds data converted from cv::Mat
  net_input_blob_ = new caffe::Blob&lt;float&gt;(1, 3, 227, 227);
}
Consumer::~Consumer(){
  Stop();
  delete net_input_blob_;
}
void Consumer::entry(boost::shared_ptr&lt;QueuePair&lt;Mat&gt;&gt; task
    , boost::shared_ptr&lt;QueuePair&lt;std::string&gt;&gt; result){

  caffe::Caffe::set_mode(caffe::Caffe::GPU);
  caffe::Caffe::SetDevice(0);

  cv::Mat *frame;
  cv::Mat resized_image(256, 256, CV_8UC3);
  cv::Size re_size(resized_image.cols, resized_image.rows);

  //for caffe input and output
  std::vector&lt;caffe::Blob&lt;float&gt; *&gt; net_input;
  std::vector&lt;caffe::Blob&lt;float&gt; *&gt; net_output;

  net_input.push_back(caffe_input_blob);
  std::string *res;

  int pre_num = 1;
  while (!must_stop()){
    std::stringstream result_strm;
    frame = task-&gt;full_.pop();
    cv::resize(*frame, resized_image, re_size, 0, 0, CV_INTER_LINEAR);
    this-&gt;data_transformer_-&gt;Transform(resized_image, caffe_input_blob);
    net_output = this-&gt;net_-&gt;Forward(net_input);
    task-&gt;free_.push(frame);

    res = result-&gt;free_.pop();
    //Process results here
    for (int i = 0; i &lt; pre_num; ++i){
      result_strm &lt;&lt; synset_words_[net_output[0]-&gt;cpu_data()[i]] &lt;&lt; " " 
                  &lt;&lt; net_output[0]-&gt;cpu_data()[i + pre_num] &lt;&lt; "\n";
    }
    *res = result_strm.str();
    result-&gt;full_.push(res);
  }
}

void Consumer::Run(){
  if (!thread_){
    try{
      thread_.reset(new boost::thread(&amp;Consumer::entry, this, task_q_, result_q_));
    }
    catch (std::exception&amp; e) {
      std::cerr &lt;&lt; "Thread exception: " &lt;&lt; e.what() &lt;&lt; std::endl;
    }
  }
  else
    std::cout &lt;&lt; "Consumer thread may have been running!" &lt;&lt; std::endl;
};
void Consumer::Stop(){
  if (thread_ &amp;&amp; thread_-&gt;joinable()){
    thread_-&gt;interrupt();
    try {
      thread_-&gt;join();
    }
    catch (boost::thread_interrupted&amp;) {
    }
    catch (std::exception&amp; e) {
      std::cerr &lt;&lt; "Thread exception: " &lt;&lt; e.what() &lt;&lt; std::endl;
    }
  }
}
bool Consumer::must_stop(){
  return thread_ &amp;&amp; thread_-&gt;interruption_requested();
}


int main(void)
{
  int max_queue_size = 1000;
  boost::shared_ptr&lt;QueuePair&lt;Mat&gt;&gt; tasks(new QueuePair&lt;Mat&gt;(max_queue_size));
  boost::shared_ptr&lt;QueuePair&lt;std::string&gt;&gt; results(new QueuePair&lt;std::string&gt;(max_queue_size));

  char str[100], info_str[100] = " results: ";
  VideoCapture vc(0);
  if (!vc.isOpened())
    return -1;

  Consumer consumer(tasks, results);
  consumer.Run();

  Mat frame, *frame_copy;
  namedWindow("preview");
  double t, fps;

  while (true){
    t = (double)getTickCount();
    vc.read(frame);

    if (waitKey(1) &gt;= 0){
      consuer.Stop();
      break;
    }

    if (tasks-&gt;free_.try_peek(&amp;frame_copy)){
      frame_copy = tasks-&gt;free_.pop();
      *frame_copy = frame.clone();
      tasks-&gt;full_.push(frame_copy);
    }
    std::string *res;
    std::string frame_info("");
    if (results-&gt;full_.try_peek(&amp;res)){
      res = results-&gt;full_.pop();
      frame_info = frame_info + info_str;
      frame_info = frame_info + *res;
      results-&gt;free_.push(res);
    }    

    t = ((double)getTickCount() - t) / getTickFrequency();
    fps = 1.0 / t;

    sprintf(str, " fps: %.2f", fps);
    frame_info = frame_info + str;

    putText(frame, frame_info, Point(5, 20)
         , FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 255, 0));
    imshow("preview", frame);
  }
}
</code></pre>

<p>And in <a href="https://github.com/BVLC/caffe/blob/master/src/caffe/util/blocking_queue.cpp" rel="nofollow">src/caffe/util/blocking_queue.cpp</a>, make a little change below and rebuild caffe:</p>

<pre><code>...//Other stuff
template class BlockingQueue&lt;Batch&lt;float&gt;*&gt;;
template class BlockingQueue&lt;Batch&lt;double&gt;*&gt;;
template class BlockingQueue&lt;Datum*&gt;;
template class BlockingQueue&lt;shared_ptr&lt;DataReader::QueuePair&gt; &gt;;
template class BlockingQueue&lt;P2PSync&lt;float&gt;*&gt;;
template class BlockingQueue&lt;P2PSync&lt;double&gt;*&gt;;
//add these 2 lines below
template class BlockingQueue&lt;cv::Mat*&gt;;
template class BlockingQueue&lt;std::string*&gt;;
</code></pre>

python opencv multiprocessing gpgpu caffe