Fastest way to find reference point in image

i'm searching for the fastest way to find one or more reference points, (could be +, or a rectangle).

My first idea is, to go from the mid point of the image and look iterative in all 4 directions for the coloured pixels. But in this case it is not clear for me how to handle a symbol like + .

Hopefully my description is clear. Greetings