Skip to content


Saliency

The visual system provides us an enormous amount of information. For processing it in real time to be able to survive, humans (and other animals) have developed an attention system that allows them to filter out non important portions of the scene by just focusing on the most salient parts of what is being observed.

lena being processed with  the fine grained saliency featuresI wrote a paper about using saliency as a new feature for object detection obtaining good results. It was accepted for publication in the Image and Vision Computing (IMAVIS) journal. You can download the manuscript from here. For calculating the new proposed features, I wrote a fine-grained saliency library.  It requires OpenCV to run properly. You can download it from here. You can also download the source code which is released under a BSD license.

UPDATE: I updated the code to be used with the modern C++ OpenCV interface and CMake. You can get this code here.

If you want to see the algorithm applied to your own images quickly, just go to my online fine-grained saliency generator.


35 Responses

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.

  1. Arash Shafiei says

    Hello,
    I’ve translated your code from C to Python:

    http://www.arash-shafiei.com/?p=99

  2. James Monoch says

    Thanks for providing a great resource :)

    PS, You should URL-encode filenames uploaded on the online FGSG or else the results cannot be viewed for certain filenames.

  3. Beiping Hou says

    Thank you very much for your code and paper.
    I have learnt much form your research work.

  4. Medhat Riad says

    Interesting… I read your paper… I also used a Frintrop-related bottom-up saliency in my master thesis… http://igitur-archive.library.uu.nl/student-theses/2010-0623-200147/thesis.pdf

    I have a question though, doesn’t your described method calculate surrounds for fixed centers? Isn’t this different from calculating surrounds for different centers… (moving to a different scale should give a different center(x,y) and not the same i(x,y) value). Right?

  5. samontab says

    Great question Medhat Riad.

    I designed my system to be as similar as possible to the ideas behind Frintrop’s VOCUS, but with an increased resolution in the output.

    Because of that, I decided to keep the original idea that the center will always be I(x, y), which is the minimum possible value that the surround can be compared to. If you enlarge the center area, you may miss fine grained details that are lost in the average of that area.

    Having said that, these differences of sizes probably will not affect the final result in a significant manner. On a later paper, Frintrop also explored the idea of using circular areas instead of squared ones, but the results were very similar.

    I hope I could help you.

  6. PengPai says

    How can I contact Arash Shafiei ?

  7. samontab says

    I do not know PengPai, I do not have any contact information of him.

  8. Matias says

    hi, the code does not seem to be available. could you fix the link please? I’m interested in testing your approach

  9. samontab says

    Hi Matias,

    You will be able to download it now.

  10. Matias says

    Great, thank you!
    Have you tried using more feature channels? such as color opponency, contrast and orientation? Is your approach capable of adding these channels?

  11. Peter says

    Nice work!, What’s the licence of the code? Could you update the sources with the licence?

  12. samontab says

    Thanks Peter,

    It is released under a BSD license, and I just updated the sources to reflect this.

  13. samontab says

    Hi Matias,

    For the purpose of the publication only the intensity channel was used, but it is totally possible to use the same idea to calculate other channels. It will depend on the specific feature that you want, but things like color for example should be really simple to integrate using this method. Other features like orientation may need an alternative representation or an approximation to make use of this technique.

  14. samontab says

    I updated the code to use the C++ interface (cv::Mat instead of IPLImage, etc…), and sent it to opencv contrib github. You can get it here:
    https://github.com/Itseez/opencv_contrib/pull/122

  15. remake says

    which cpp file(s) to download from there? sorry i couldn’t find the files in that link

  16. samontab says

    Well, it is a pull request for the official opencv_contrib repository, so no direct zip.
    You can go to my github project and download the zip there though (click on Download Zip):
    https://github.com/samontab/opencv_contrib
    That will contain the entire opencv_contrib repository with my changes added(fine grained saliency). The files you are after are inside the folder modules/saliency. In particular, take a look at staticSaliencyFineGrained.cpp.
    Hopefully in the future it will be included in OpenCV.

  17. remake says

    Thank you for the rapid reply. Do I have to build whole opencv to use the module? Isn’t there a simple way to just to include the files and get it running? Because in the website it tells me to rebuild the opencv library with the modules and etc, I do not want to rebuild the opencv since this isn’t my own computer, therefore I would like to avoid any possible damage that might arise from rebuilding opencv.

    so in short, is it okay if i just build a project and include the stuff which are only in modules/saliency?

  18. samontab says

    Hi remake,

    Nah, you only need the staticSaliencyFineGrained.cpp file actually. The easiest would be to just download the version from my website to get it compiling with the old interface, and then just replace the code with the new C++ interface from that file.

  19. remake says

    Hi Mr. Montabone,

    I am able to compile and run the old interface one, no problem with that. It works perfectly.

    However, when I try the new interface one, I face problems since I do not have OpenCV 3.0 (and I guess some headers require that). I tried to put the new .cpp file ( staticSaliencyFineGrained.cpp) into the old interface one’s project but it didn’t work. I think it requires a header called precomp, and that header requires some other header and it goes like that, at the end I have all the files in that new module/saliency folder, but I do not know how to get this running. I tried to do it with cmake and all, but no luck. Here is the folder structure that I have:

    -src
    –main.cpp
    –motionSaliency.cpp
    –motionSaliencyBinWangApr2014.cpp
    –objectness.cpp
    –precomp.hpp
    –saliency.cpp
    –saliency_init.cpp
    –staticSaliency.cpp
    –staticSaliencyFineGrained.cpp
    –staticSaliencySpectralResidual.cpp

    -include
    –opencv
    —saliency
    —-saliencyBaseClasses.hpp
    —-saliencySpecializedClasses.hpp
    –saliency.hpp

    and optionally :

    -CMakeLists.txt

    This above setup does not even compile due to Opencv 3.0 issue, some header there requires 3.0 and I don’t have that. So I deleted that file (i guess it was objectness) since i will not use it. However then some other errors occurred, somehow the functions of the fine grain saliency class are not recognized, it treats them like they weren’t predefined.

    I even tried deleting all irrelevant files like motion saliency and others, and just keeping fine grain one, but again, didn’t work.

    I guess i will try to convert this old interface one into the new one in the project that already runs. Sweat work, but better than compilation errors.

  20. samontab says

    Yeah, it is written for OpenCV 3.0, but it should be easy to solve the problems, there are minor changes in the syntax between 2.x and 3.x.

    You could just change the method (functions) from the working old interface using the new one. If you just copy and paste the entire file it will not work, as it is written as a module for opencv.

  21. remake says

    I got what you mean now, and yes that is the easiest way. I did so and it worked, thanks a lot :)

  22. samontab says

    Great to hear that it worked for you, remake

  23. wux says

    Hi samontab!

    I found an example in the opencv_contrib, and I build it. But I still don’t know how to use that software even with your little tips. Could you explain it to me, thank you. Maybe you can add an example usage to that file.

  24. wux says

    I mean what’s the type of saliencyAlgorithm, how to input it correctly.

  25. samontab says

    Hi wux.

    The saliency type is FINE_GRAINED. You need to download the opencv_contrib code from my github account though, which is here:
    https://github.com/samontab/opencv_contrib

  26. samontab says

    Hi wux,

    There is already an example for saliency algorithms in general. You just need to specify the saliency type you want to use.

  27. wux says

    Hi samontab,

    I build your computeSaliency example and named it Saliency, so I call
    ./Saliency
    and it says
    This example shows the functionality of “Saliency “Call:
    ./example_saliency_computeSaliency
    so I input the subtype FINE_GRAINED like this:
    ./Saliency FINE_GRAINED test.avi 3
    and it says
    ***Error in the instantiation of the saliency algorithm…***
    I don’t konw why

  28. wux says

    missing comments:
    This example shows the functionality of “Saliency “Call:
    ./example_saliency_computeSaliency saliencyAlgorithmSubType video_name start_frame

  29. samontab says

    Hi wux,

    Probably you are using the latest opencv_contrib code, and not the fork on my github account.
    This means that the FINE_GRAINED is not defined there
    Make sure that you are using my code and not the latest code from opencv.

  30. samontab says

    Hi wux,

    I updated my github project, and tested it with OpenCV 3.0.0. It is working fine.
    If you still have issues, I also re packaged it into a stand alone application including a CMake project file to make it really easy to use.

    You can get the C++ interface of the fine-grained saliency library with an example usage here:
    http://samontab.com/web/files/saliencyFineGrained.zip

  31. wux says

    Hi samontab,

    I downloaded opencv_contrib from your github, but I can’t build it with my opencv code(also downloaded from itseez), really sad. So I downloaded opencv-3.0.0, but problems still exist.
    Anyway, thank you.
    I think I’ll back to the beginning, see whether these saliency api works or not.

  32. wux says

    didn’t see your latest comments, I’ll test it. thank you!!! :)

  33. wux says

    Hi, samontab!

    Thank you for your update, now I can look into those saliency API.

  34. samontab says

    You are welcome wux.

  35. Gary Lee says

    Sebastian: I am very interested in your masters work and the examples provided via video showing the algorithms in action. Please contact me on the email provided here to start a conversation. Thank you.



Some HTML is OK

or, reply to this post via trackback.