VTRMC 2014 Problem 6

Let $S$ denote the set of $2$ by $2$ matrices with integer entries and determinant $1$ , and let $T$ denote those matrices of $S$ which are congruent to the identity matrix $I\pmod3$ (so $\begin{pmatrix}a&bc&d\end{pmatrix}\in T$ means that $a,b,c,d\in\mathbb Z,ad-bc=1,$ and $3$ divides $b,c,a-1,d-1$ ).

(a) Let $f:T\to\mathbb R$ be a function such that for every $X,Y\in T$ with $Y\ne I$ , either $f(XY)>f(X)$ or $f(XY^{-1})>f(X)$ . Show that given two finite nonempty subsets $A,B$ of $T$ , there are matrices $a\in A$ and $b\in B$ such that if $a'\in A$ , $b'\in B$ and $a'b'=ab$ , then $a'=a$ and $b'=b$ . (b) Show that there is no $f:S\to\mathbb R$ such that for every $X,Y\in S$ with $Y\ne\pm I$ , either $f(XY)>f(X)$ or $f(XY^{-1})>f(X)$ .