Let $G$ be a topological group and $X$ any topological space. We say that $G$ is a topological transformation group of $X$ if $G$ acts on $X$ continuously, in the following sense:

1. there is a continuous function $\alpha:G\times X\to X$ , where $G\times X$ is given the product topology
2. $\alpha(1,x)=x$ , and
3. $\alpha(g_1g_2,x)=\alpha(g_1,\alpha(g_2,x))$ .

The function $\alpha$ is called the (left) action of $G$ on $X$ . When there is no confusion, $\alpha(g,x)$ is simply written $gx$ , so that the two conditions above read $1x=x$ and $(g_1g_2)x=g_1(g_2x)$ .

If a topological transformation group $G$ on $X$ is effective, then $G$ can be viewed as a group of homeomorphisms on $X$ : simply define $h_g:X\to X$ by $h_g(x)=gx$ for each $g\in G$ so that $h_g$ is the identity function precisely when $g=1$ .

Some Examples.

1. Let $X=\mathbb{R}^n$ , and $G$ be the group of $n\times n$ matrices over $\mathbb{R}$ . Clearly $X$ and $G$ are both topological spaces with the usual topology. Furthermore, $G$ is a topological group. $G$ acts on $X$ continuous if we view elements of $X$ as column vectors and take the action to be the matrix multiplication on the left.
2. If $G$ is a topological group, $G$ can be considered a topological transformation group on itself. There are many continuous actions that can be defined on $G$ . For example, $\alpha:G\times G\to G$ given by $\alpha(g,x)=gx$ is one such action. It is continuous, and satisfies the two action axioms. $G$ is also effective with respect to $\alpha$ .