Your description lacks some information. If the installation fails, there is an error message displayed (which one is it in your case?) and the error message usually points to a log file (please attach).
If the compilation of the kernel modules fails, the installer will not remove anything because the user is free to install the missing components and can then recompile the kernel modules without the need to reinstall all files. Uninstalling the Guest Additions is clearly documented in the user manual (for your info, section 4.2.2.4). After you've done so, the Guest Additions should be completely vanished from your guest.
The --nox11 option does exactly what it is supposed to do: It would spawn an xterm (or rxvt, dtterm, eterm, Eterm, kvt, konsole, aterm in that order) if you would not start the installer from a terminal. This is a generic option of the makeself installer framework. Probably rarely used but nevertheless works.
