Together with an IP (which specifies the exact computer, either server or client, with which some computer is trying to connect to) it specifies the exact destination the connection must get to. It is written in the form IP:PORT most commonly.
For example, aMule usually uses port 4662. So let's suppose your IP is 22.214.171.124. Now let's say some computer somewhere sends a packet to 126.96.36.199:4662. This packet is going to be sent to 188.8.131.52, which is your computer, and once there, it will be delivered to port 4662, which most probably will be a port where aMule will be listening, so it will get to aMule. aMule will read the packet and act depending on what its content is.
If the packet was sent to 184.108.40.206:21 it will also get to you, but (probably) not to aMule since aMule will no be listening on that port. If some other application is listening to it (possible some FTP server), it will read its contents and act depending on what they are. If no application is listening to that port, then the packet will just fail to be delivered. Though it got to your computer, no application was waiting for it, so it will be refused.
Other articles you might want to check: