Let the transformation matrix be \(\displaystyle{A}={b}{e}{g}\in{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{P}&{Q}\backslash{R}&{S}{e}{n}{d}{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{N}{S}{K}{A}={b}{e}{g}\in{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{X}{1}\backslash{Y}{1}{e}{n}{d}{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}={b}{e}{g}\in{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{P}&{Q}\backslash{R}&{S}{e}{n}{d}{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{b}{e}{g}\in{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{X}{1}\backslash{Y}{1}{e}{n}{d}{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}={b}{e}{g}\in{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{P}{X}{1}+{Q}{Y}{1}\backslash{R}{X}{1}+{S}{Y}{1}{e}{n}{d}{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}={b}{e}{g}\in{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{X}{2}\backslash{Y}{2}{e}{n}{d}{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{N}{S}{K}{X}{2}={\left[{P}{X}{1}+{Q}{Y}{1}\right]}{N}{S}{K}{Y}{2}={\left[{R}{X}{1}+{S}{Y}{1}\right]}\)
Putting in thhe given EQN.
\(\displaystyle{\left[{P}{X}{1}+{Q}{Y}{1}\right]}^{{2}}+{6}{\left[{R}{X}{1}+{S}{Y}{1}\right]}^{{2}}-{36}={2}{X}{1}^{{2}}-{4}{X}{1}{Y}{1}+{5}{Y}{1}^{{2}}-{36}{N}{S}{K}{P}{2}+{6}{R}{2}={2}{N}{S}{K}{Q}{2}+{6}{S}{2}={5}{N}{S}{K}{2}{P}{Q}+{12}{R}{S}=-{4}\ldots\ldots{P}{Q}+{6}{R}{S}=-{2}\)
Since the conversion is from a non homogeneus to homogeneous QN.,
Which is obtained by rotation through angle tyou can use the following transformation matrix as a formula
\(\displaystyle{b}{e}{g}\in{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{C}{O}{S}{\left({T}\right)}&{S}{I}{N}{\left({T}\right)}\backslash-{S}{I}{N}{\left({T}\right)}&{C}{O}{S}{\left({T}\right)}{e}{n}{d}{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{b}{e}{g}\in{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{X}{1}\backslash{Y}{1}{e}{n}{d}{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}={b}{e}{g}\in{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{X}{2}\backslash{Y}{2}{e}{n}{d}{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}\)
Where \(\displaystyle{T}={\frac{{{1}}}{{{2}}}}{{\tan}^{{-{1}}}{\left({\frac{{{2}{H}}}{{{A}-{B}}}}\right)}}\)
Where
\(\displaystyle{H}={C}{O}{E}{F}{F}{I}{C}{I}{E}{N}{T}{O}{F}{X}{1}{Y}{1}=-{4}\) A = COEFFICIENT OF X1^2 = 2ZSK B = COEFFICIENT OF Y1^2 = 5ZSK
Solving we get
\(\displaystyle{P}={0.89443}\ldots\ldots{Q}={0.44721}\ldots\ldots{R}=--{0.44721}\ldots\ldots\ldots{S}={0.89443}\)
Hence the transformation matrix is \(\displaystyle{A}={b}{e}{g}\in{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}{0.89443}&{0.44721}\backslash-{0.44721}&{0.89443}{e}{n}{d}{\left\lbrace{b}{m}{a}{t}{r}{i}{x}\right\rbrace}\)