iirc this is an example of a phonemic merger like pin-pen or caught-cot

for me, "our" read on its own is pronounced more similarly to "hour*;" conversely, when 'our' is used in a phrase ("this is our town"), it's pronounced more like "are"

* in northern ireland, this merger is totally complete and "hour" and "are" are pronounced the same