An automorphism of a group G is an isomorphism from G to itself. Denote by...

Step 1
Let GG be a group. By definition
${\displaystyle Aut\left(G\right)=\{f:G\to G\mid f\text{is an isomorphism of}G\}.}$
Given two automorphisms ${\displaystyle f,g\in Aut\left(G\right)}$, we can consider the composition ${\displaystyle g\times f}$. We claim that ${\displaystyle Aut\left(G\right),\times }$ is a group. We have to check all axioms.
First of all we need to show that ${\displaystyle g\times f}$ is again an automorphism, i.e. a homomorphism that is bijective. Now since g and f are bijective, ${\displaystyle g\times f}$ is bijective. Moreover,
${\displaystyle (g\times f)\left(ab\right)=g\left(f\left(ab\right)\right)=g\left(f\left(a\right)f\left(b\right)\right)=g\left(f\left(a\right)\right)g\left(f\left(b\right)\right)=(g\times f)\left(a\right)(g\times f)\left(b\right),}$
for all ${\displaystyle a,b\in G}$. Hence ${\displaystyle g\times f}$ is a group homomorphism.
Second, we must demonstrate that ${\displaystyle \times }$ is associative, i.e. ${\displaystyle (h\times g)\times f=h\times (g\times f)}$. Just evaluate both morphisms at ${\displaystyle a\in G}$ and see that both expressions coincide due to the associativity of GG.
Thirdly, we must confirm that a neutral component exists for ${\displaystyle \times }$. Clearly ${\displaystyle IdG:G\to G:a\to a}$ is an automorphism. Since ${\displaystyle f\times IdG=IdG\times f}$ for all ${\displaystyle f\in Aut\left(G\right),IdG}$ is the neutral element.
Last but not least, we must ensure that each ${\displaystyle f\in Aut\left(G\right)}$ has an inverse for ${\displaystyle \times }$. Consider the inverse function ${\displaystyle f-1}$. Clearly ${\displaystyle f-1\times f=IdG=f\times f-1}$. So it remains to show that ${\displaystyle f-1}$ is a group morphism. Now it's a very good exercise to prove this last statement.
EDIT: Let's prove the last statement. Suppose that ${\displaystyle f:G\to G}$ is a group isomorphism. We need to show that ${\displaystyle f-1}$ is a group morphism. Let ${\displaystyle a,b\in G}$. By definition there exist a unique ${\displaystyle x,y\in G}$ such that $f(x)=a$ and ${\displaystyle f\left(y\right)=b}$. Hence
${\displaystyle f-1\left(ab\right)=f-1\left(f\left(x\right)f\left(y\right)\right)=f-1\left(f\left(xy\right)\right)=xy}$.
Similarly
${\displaystyle f-1\left(a\right)f-1\left(b\right)=f-1\left(f\left(x\right)\right)f-1\left(f\left(y\right)\right)=xy}$.
Therefore, ${\displaystyle f-1\left(ab\right)=f-1\left(a\right)f-1\left(b\right)}$

Step 1
Let Aut(G) be the set of all automorphisms ${\displaystyle \varphi :G\to G}$. In order to show that this is a group under the operation of composition, we must verify:
1) Is the set is closed under composition? Yes! If you are given isomorphisms ${\displaystyle \varphi ,\psi :G\to G}$, then it is not too tough to show that ${\displaystyle \psi \circ \varphi \text{and}\varphi \circ \psi }$ are isomorphisms. I can expand on this in more detail if you like, but you have probably seen a proof before that a composition of bijective functions is bijective. If a and b are elements of the group,
${\displaystyle \psi \circ \varphi \left(ab\right)=\pi \left(\varphi \left(ab\right)\right)=\psi \left(\varphi \left(a\right)\varphi \left(b\right)\right)}$,
because ${\displaystyle \varphi }$ is an isomorphism. Since ${\displaystyle \psi }$ is also an isomorphism,
${\displaystyle \psi \left(\varphi \left(a\right)\varphi \left(b\right)\right)=\psi \circ \varphi \left(a\right)\psi \circ \varphi \left(b\right)}$,
so the composition ${\displaystyle \psi \circ \varphi }$ preserves products. Thus, ${\displaystyle \psi \circ \varphi }$ is an isomorphism if ${\displaystyle \psi }$ and ${\displaystyle \varphi }$ are.
2) Is the set associative? Yes! All you need to do is show that, for any three isomorphisms ${\displaystyle \varphi ,\psi }$ and
${\displaystyle \xi ,\varphi \circ (\psi \circ \xi )=(\varphi \circ \psi )\circ \xi }$.
To do that, just show that for each x in
${\displaystyle G,\varphi \circ (\psi \circ \xi )\left(x\right)=(\varphi \circ \psi )\circ \xi \left(x\right)=\varphi \left(\psi \left(\xi \left(x\right)\right)\right)}$.
It's just pushing around definitions.
3) Does the set contain an identity element? Yes! Let the identity automorphism ${\displaystyle e:G\to G}$ be the map ${\displaystyle e\left(x\right)=x}$. Clearly, ${\displaystyle e\circ \varphi =\varphi \circ e=\varphi }$.
4) Does each element of the set have an inverse under ${\displaystyle \circ }$? Yes! Since each isomorphism ${\displaystyle \varphi :G\to G}$ is bijective, there is a well-defined inverse map ${\displaystyle {\varphi }^{-1}:G\to G}$. You may have already seen a proof that the inverse of an isomorphism is an isomorphism. If not, it isn't too difficult to prove: I'll leave it to you, but I can expand on it if you need me to. Further, the composition
${\displaystyle {\varphi }^{-1}\circ \varphi =\varphi \circ {\varphi }^{-1}=e}$.
Since Aut(G) satisfies all the group axioms, it forms a group under ${\displaystyle \circ }$, as needed.