a) If f(t)=tmandg(t)=tn, where m and n are positive integers. show that f∗g=tm+n+1∫01um(1−u)ndu b) Use...

(a)
By definition,
${\displaystyle f\cdot g={\int }_0^{t}f\left(\tau \right)g(t-\tau )d\tau ={\int }_0^t{\tau }^m{(t-\tau )}^{n}d\tau }$
Since we must get ${\displaystyle u^m{(1-u)}^n}$, we can try with the substitution ${\displaystyle \tau =ut}$:
${\displaystyle {\int }_0^t{\tau }^m{(t-\tau )}^{n}d\tau =\{(\tau =ut0\mapsto 0),(d\tau =tdut\mapsto 1)\}}$
${\displaystyle ={\int }_0^1{\left(ut\right)}^m{(t-ut)}^{n}tdu}$
${\displaystyle ={\int }_0^1{u}^m{t}^m{\left(t(1-u)\right)}^{n}tdu}$
${\displaystyle =t^{m+n+1}{\int }_0^1{u}^m{(1-u)}^{n}du}$
(b) and (c)
Notice that
${\displaystyle {\int }_0^1{u}^m{(1-u)}^{n}du=B(m,n),}$
where B is $\beta$ function. Recall that,
${\displaystyle B(m,n)=\frac{\mathrm{\Gamma }(m+1)\mathrm{\Gamma }(n+1)}{\mathrm{\Gamma }(m+n+2)},}$
where ${\displaystyle \mathrm{\Gamma }is\gamma }$ function. When m and n are positive integers,
${\displaystyle B(m,n)=\frac{m!n!}{(m+n+1)!}}$
The finally result for (a), consider the subtitution ${\displaystyle \tau =ut}$.
For (b) and (c), notice that this integral is a $\beta$ function