Determine the value of a if the system x 1 + 4 x 2 −...

We know that a homogeneous linear system has more than one solution if and only if the rank of the matrix of the system is less than the number of unknowns. See Rank-nullity theorem.
One of the possibilities how to find rank is to use row operations. You could start like this:
$\left(\begin{array}{cccc}1& 4& -3& 2\\ 2& 7& -4& 4\\ -1& a& 5& -2\\ 3& 10& -5& a^2+4a+1\end{array}\right)\sim \left(\begin{array}{cccc}1& 4& -3& 2\\ 0& -1& 2& 0\\ 0& a+4& 2& 0\\ 3& 10& -5& a^2+4a+1\end{array}\right)\sim \left(\begin{array}{cccc}1& 4& -3& 2\\ 0& 1& -2& 0\\ 0& a+4& 2& 0\\ 3& 10& -5& a^2+4a+1\end{array}\right)\sim \left(\begin{array}{cccc}1& 4& -3& 2\\ 0& 1& -2& 0\\ 0& a+5& 0& 0\\ 3& 10& -5& a^2+4a+1\end{array}\right)$
If $a+5=0$, i.e. if $a=5$, then you have a matrix which does not have full rank. So in this case you have that there is more than one solution.
If $a+5\ne 0$, you can divide the third row by $(a+5)$
$\left(\begin{array}{cccc}1& 4& -3& 2\\ 0& 1& -2& 0\\ 0& a+5& 0& 0\\ 3& 10& -5& a^2+4a+1\end{array}\right)\sim \left(\begin{array}{cccc}1& 4& -3& 2\\ 0& 1& -2& 0\\ 0& 1& 0& 0\\ 3& 10& -5& a^2+4a+1\end{array}\right)\sim \left(\begin{array}{cccc}1& 0& -3& 2\\ 0& 0& -2& 0\\ 0& 1& 0& 0\\ 3& 0& -5& a^2+4a+1\end{array}\right)\sim \left(\begin{array}{cccc}1& 0& -3& 2\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 3& 0& -5& a^2+4a+1\end{array}\right)\sim \left(\begin{array}{cccc}1& 0& 0& 2\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 3& 0& 0& a^2+4a+1\end{array}\right)$