1. Open ports in firewall can be scanned.
2. Telnet connections can be used to probe for installed software and its versions on attacked server. Or perhaps dedicated software, actively or passively.
3. If connection is not encrypted, protocol can be 'sniffed', understood and broken. Hacked client can be made, that can for example send packets that can overflow a buffer, and execute code remotely. If there's open source software on server, it's weaknesses are known and this type of attack is easier to execute.
4. If connection is encrypted, customer or user can be convinced to use 'custom user interface', this way protocol can be broken also. Or hacker can turn off secure connection himself and attempt to break protocol that way.
this opens way for many more moves such as:
1. Intercepting communication, malforming it, pretending to say something instead of someone (for p2p communication).
2. Committing transactions to selected server requests, on behalf of other users.