Acos 90 degree matrix transformation. I'm writing a program that transforms a matrix of points by 9

Your two vectors are parallel to the vectors $(1,1,1)$ and $(1,1,-1)$. You can think of each these vectors as being the diagonal of a unit cube - one unit cube sitting above the $x,y$ plane and its mirror image below the $x,y$ plane.
The projection of one of these vectors on the $x,y$ plane has length $\sqrt{2}$. So each of these vectors is the hypotenuse of a triangle with sides $\sqrt{2}$ and $1$. So the angle between each of these vectors and the $x,y$ plane is ${\tan }^{-1}\left(\frac{1}{\sqrt{2}}\right)\approx {35.26}^o$. And the angle between the two vectors is therefore $2\times {35.26}^o={70.52}^o$.
And $\cos (2\times {\tan }^{-1}\left(\frac{1}{\sqrt{2}}\right))=\frac{1}{3}$, which confirms what the dot product calculation tells us. A rotation by ${90}^o$ about the $x$ axis $(x,y,z)\mapsto (x,z,-y)$ will rotate vectors in the $y,z$ plane through ${90}^o$, but vectors like $(1,1,1)$ which are not in the $y,z$ plane will not be rotated by ${90}^o$

When you have a visualization of a 3D objects there always is some kind of projection to a 2D plane – the screen plane. You should not rely on projected lengths, angles, areas etc. when you don’t know the type of projection and understand its properties. They might be misleading.
But your calculations are correct: the vectors are not orthogonal! The angle is ${\cos }^{-1}(\frac{1}{3})\approx {70.53}^{\circ }$