Unless you've set up gpm as a repeater for the XFree86 server (using the -R option and /dev/gpmdata fifo), the XFree86 server will take over control of the mouse device and use that ...