When is examining only one direction of a bijective map phi:X rightarrow Y enough to categorize it as an isomorphism?

Step 1
In the first class, homomorphisms must respect operations defined on all inputs from X:
- For vector spaces: addition: $u+v=w⟹<!--\; ⟹\; -->\mathrm{\varphi <!--\; \varphi \; -->}(u)+\mathrm{\varphi <!--\; \varphi \; -->}(v)=\mathrm{\varphi <!--\; \varphi \; -->}(w)$ and $\mathrm{\alpha <!--\; \alpha \; -->}u=v⟹<!--\; ⟹\; -->\mathrm{\alpha <!--\; \alpha \; -->}\mathrm{\varphi <!--\; \varphi \; -->}(u)=\mathrm{\varphi <!--\; \varphi \; -->}(v)$
- For groups: $ab=c⟹<!--\; ⟹\; -->\mathrm{\varphi <!--\; \varphi \; -->}(a)\mathrm{\varphi <!--\; \varphi \; -->}(b)=\mathrm{\varphi <!--\; \varphi \; -->}(c)$
- For metric spaces: $d(x,y)=r⟹<!--\; ⟹\; -->d(\mathrm{\varphi <!--\; \varphi \; -->}(x),\mathrm{\varphi <!--\; \varphi \; -->}(y))=r$
More generally, $F(x_1,\dots <!--\; \dots \; -->,x_m)=x\text{ or }=c⟹<!--\; ⟹\; -->F(\mathrm{\varphi <!--\; \varphi \; -->}(x_1),\dots <!--\; \dots \; -->,\mathrm{\varphi <!--\; \varphi \; -->}(x_m))=\mathrm{\varphi <!--\; \varphi \; -->}(x)\text{ or }=c$. Then bijectivity of $\mathrm{\varphi <!--\; \varphi \; -->}$ lets us conclude that the same property holds for the inverse map $\mathrm{\psi <!--\; \psi \; -->}$: E.g., if $F(y_1,\dots <!--\; \dots \; -->,y_n)=y$, then $F(\mathrm{\psi <!--\; \psi \; -->}(y_1),\dots <!--\; \dots \; -->,\mathrm{\psi <!--\; \psi \; -->}(y_n))$ is some x and it must be the case that $\mathrm{\varphi <!--\; \varphi \; -->}(x)=y$, hence $x=\mathrm{\psi <!--\; \psi \; -->}(y)$ as desired.
Step 2
This does not work for e.g. topological spaces, even if we define an infinitary operation $F(x_1,\dots <!--\; \dots \; -->)=1$ or $=0$ if $\{x_1,\dots <!--\; \dots \; -->\}$ is open or not. For respecting this F would not define morphisms of Top (i.e., continuous maps). Recall that these are rather defined in a contra-variant fashion, i.e., the preimages of open sets in Y have to be open in X.

Step 1
In arguably all interesting cases you only need to define your type-1 operation and can recover your type-2 operation from that. Let's say we have such a type-1 operation, say,
$$\mathrm{\varphi <!--\; \varphi \; -->}:X\to <!--\; \to \; -->Y,$$
where X, Y are two arbitrary structures of the same kind. For example, X, Y could be groups. Or they could both be vectorspaces. So we assume $\mathrm{\varphi <!--\; \varphi \; -->}$ being such a compatible operation, e.g. for groups it's a group homomorphism, for vectorspaces it's a linear function.
Now I claim I can define your type-2 operation by that:
Definition (Isomorphism): We call a type-1 operation $\mathrm{\varphi <!--\; \varphi \; -->}:X\to <!--\; \to \; -->Y$ an isomorphism iff. there is a type-1 operation $\mathrm{\psi <!--\; \psi \; -->}:Y\to <!--\; \to \; -->X$ such that
$\mathrm{\psi <!--\; \psi \; -->}\circ <!--\; \circ \; -->\mathrm{\varphi <!--\; \varphi \; -->}=i{d}_X$ and
$\mathrm{\varphi <!--\; \varphi \; -->}\circ <!--\; \circ \; -->\mathrm{\psi <!--\; \psi \; -->}=i{d}_Y$
Let's see if my notion of isomorphism coincides with your type-2 operations:
- If X,Y are topological spaces, then $\mathrm{\varphi <!--\; \varphi \; -->}$ needs to be continuous, bijective and open.
Note that you forgot "bijective" in your post. If $\mathrm{\varphi <!--\; \varphi \; -->}:X\to <!--\; \to \; -->Y$ is continuous, it's a type-1 operation for topological spaces. Since it's bijective we have $\mathrm{\psi <!--\; \psi \; -->}:Y\to <!--\; \to \; -->X$ as the inverse function. Does that $\mathrm{\psi <!--\; \psi \; -->}$ fit my isomorphism definition? First, is it a type-1 operation at all? Yes, it is! Openness of $\mathrm{\varphi <!--\; \varphi \; -->}$ implies continuity of $\mathrm{\psi <!--\; \psi \; -->}$. Next: do we have those two equalities with $i{d}_X$ and $i{d}_Y$? Yes, since they are inverse functions by construction.
So a type-2 operation for topological spaces is exactly a type-1 operation which happens to be an isomorphism as defined above.
- If X,Y are vectorspaces, then ϕ needs to be linear and bijective.
Step 2
Left as an exercise for the reader.
You see, there's no need to specify what a type-2 operation is for top. spaces or vectorspaces. It's induced by the definition of the type-1 operation. And that's the case for almost all interesting structures!
To answer your question:
When is examining only one direction of a bijective map $\mathrm{\varphi <!--\; \varphi \; -->}:X\to <!--\; \to \; -->Y$ enough to categorize it as an isomorphism?
The classification as an isomorphism almost always exhibits an inverse $\mathrm{\psi <!--\; \psi \; -->}$ as stated above. It's just the case that in some structures, we have theorems relating a property of $\mathrm{\varphi <!--\; \varphi \; -->}$ to the existence of $\mathrm{\psi <!--\; \psi \; -->}$:
- for ordinary functions solely on sets, bijectivity ensures existence of inverse $\mathrm{\psi <!--\; \psi \; -->}$
- for groups, bijectivity ensures existence of inverse $\mathrm{\psi <!--\; \psi \; -->}$- for topological spaces, bijectivity + openness ensure existence of inverse $\mathrm{\psi <!--\; \psi \; -->}$
(In all cases I assume that $\mathrm{\varphi <!--\; \varphi \; -->}$ is already a type-1 operation.)