How to cache image requested with AJAX?

I'm creating a simple photo viewer for a project I'm working on. When user views an image it's being requested with AJAX and injected to the DOM. This works fine, however I am struggling to get the cache flag working properly. Every time the image changes, it's being requested AGAIN. How can I force the browser to cache the image? Ideally without using local storage or server-side solution.

EDIT:

Maybe I am being unclear. Let's say there are 5 images in total in the gallery and the user has already seen all 5 of them but wants to see them again - that's when the browser will download them again instead using the cached version. It can be easily seen in the network panel. Images are being loaded twice.

Here's my code:

$.ajax( {

    url: imgUrl,
    type: 'GET',
    cache: true,
    success: function() {

        $slide.prepend( '<img src="' + imgUrl + '" />' );

    }

} );