In geometrical optics, how can we say that rays coming from a distant object are parallel to one another?

The answer is because our ken (field of view) subtends an extremely small angle at the source. Even though the source may emit over a wide angular range, we can only receive a small angular range of that light if we have a limited aperture instrument and our distance from the source is large compared with the aperture.
Suppose we look at Alpha Centauri through a 1 meter diameter aperture. Then the range of angles present in the rays that reach us if Alpha Centauri were a true point would be:
$\frac{1\text{ meter}}{4.1\times {10}^{16}\text{ meters}}\approx 2.5\times {10}^{-17}\text{ radians}$
The path difference between a central and edge ray would be:
$\sqrt{(4.1\times {10}^{16}{)}^2+1^2}-4.1\times {10}^{16}\approx 1.25\times {10}^{-17}\mathrm{m}$
or less than one hundredth of an atomic nucleus.
Even when we take account of the fact that the star is an extended source, the range of angles is roughly the star's angular subtense at our position. This is still an extremely small number that has no bearing whatsoever on the diffraction of visible light.