Can't figure out this transformation matrix 1) The positive z axis normalized as Vector(0,0,1) has

(1) Request 2 says that we are working with affine transformations, because you are mapping $O$ to another arbitrary point of ${\mathbb{R}}^3$. But in order to be a valid affine transformation of the form ${\overrightarrow{v}}^{\prime }=A\overrightarrow{v}+b$ (b is a translation), we must have that $detA\ne 0$.to define $A$, you have to figure out how it transform a basis of your space, in your case $\overrightarrow{e_1},\overrightarrow{e_2},\overrightarrow{e_3}$. (they are the normalized of axis x,y,z)(2) $A\overrightarrow{e_3}$ is fixed by request 1, the same for $\overrightarrow{e_2}$ , but you need to know something about the image of $e_1$. Just because even if you want that it,is,orthogonal to the other images of the elements of the basis you have 2 different options for $A\overrightarrow{e_1}$ (the 2 directions of a vector).So you can't build an unique transformation with this informations. In other words if you define $A$ as your matrix, then you can define $A^{\prime }$ with the property of $A^{\prime }\overrightarrow{e_1}=-A\overrightarrow{e_1}$ and this will satisfy your requests. (to build $A^{\prime }$ just switch the signs on the first column of $A$)