How do you know when your cat needs to see a vet?

One of our cats has obviously not been feeling well for a few days now. He's eating, but not as much as usual. There is some crust around his eyes and nostrils. He's thrown up a few times this week, but never more than once a day and not every day (throwing up is not out of the ordinary for this cat). And he's just not his usual self. This morning he was more lethargic than the last couple of days and didn't want me to pet him, which is very unusual since this cat usually craves attention.

How do I know if this is just a cold or virus that will pass, or something more? At what point is a visit to the vet warranted?

He's had little belly aches before where we knew he wasn't feeling well. But this is the first time that he's avoided me and has had the crusty eyes and nose. It's not a lot of crust, just a little in the corners, but it's been going on for about four days now.

I want to take him to the vet, Hubby thinks we should wait another day or two to see if it passes. Can anyone with more cat experience provide any insight here? Thanks.