Unlike any other work that we are aware of, the study attempts to automatically detect hate speech using visual information in order to address the problem of hate speech detection in Internet memes. Memes are pixel-based multimedia documents with phrases and images that typically have a humorous connotation when combined. Nevertheless, hate memes are also disseminated via social media, so automatically identifying them would lessen the negative effects they have on society. The model can identify some of the memes, according to the results, but the problem is still not entirely resolved. Our experiments show that the visual modality can be much more informative for hate speech detection than the linguistic one in memes, despite the fact that previous work has focused on linguistic hate speech.