Let the transformation matrix be \(A = \begin{bmatrix}P & Q \\ R & S \end{bmatrix}\)

\(A = \begin{bmatrix} X1 \\ Y1 \end{bmatrix} = \begin{bmatrix}P & Q \\ R & S \end{bmatrix} \begin{bmatrix} X1 \\ Y1 \end{bmatrix} = \begin{bmatrix}PX1 + QY1 \\ RX1 + SY1 \end{bmatrix} = \begin{bmatrix} X2 \\ Y2 \end{bmatrix} \ X2 = [PX1 + QY1] \ Y2 = [RX1 + SY1]\)

Putting in thhe given EQN. \([PX1 + QY1]^2 + 6[RX1 + SY1]^2 - 36 = 2 X1^2 - 4 X1Y1 + 5 Y1^2 - 36 \ P2 + 6R2 = 2 \ Q2 + 6S2 = 5 \ 2PQ + 12RS = -4......PQ + 6RS = -2\) Since the conversion is from a non homogeneus to homogeneous QN.,

Which is obtained by rotation through angle tyou can use the following transformation matrix as a formula \(\begin{bmatrix}COS(T) & SIN(T) \\-SIN(T) & COS(T) \end{bmatrix} \begin{bmatrix} X1 \\ Y1 \end{bmatrix} = \begin{bmatrix} X2 \\ Y2 \end{bmatrix}\)

Where \(\displaystyle{T}={\frac{{{1}}}{{{2}}}}{{\tan}^{{-{1}}}{\left({\frac{{{2}{H}}}{{{A}-{B}}}}\right)}}\)

Where \(\displaystyle{H}={C}{O}{E}{F}{F}{I}{C}{I}{E}{N}{T}\ {O}{F}\ {X}{1}{Y}{1}=-{4}\)

\(A = COEFFICIENT\ OF\ X1^2 = 2\)

\(B = COEFFICIENT\ OF\) \(Y1^2 = 5\)

Solving we get \(\displaystyle{P}={0.89443}\ldots\ldots{Q}={0.44721}\ldots\ldots{R}=--{0.44721}\ldots\ldots\ldots{S}={0.89443}\)

Hence the transformation matrix is \(A = \begin{bmatrix} 0.89443 & 0.44721 \\ -0.44721 & 0.89443 \end{bmatrix}\)