Would you say "Come with I"? No you wouldn't. Would you say "Me am going out"? Not unless you are Tarzan. It doesn't take any grammatical analysis - it just sounds obviously wrong, and it is.
The problems seem to start when the sentence is just slightly more complicated. If instead of "I am going" and "Come with me", it happens to be My husband and I are going" and "Come with Beyoncé and me", suddenly all hell breaks loose. You remember hearing the Queen saying "My husband and I" and sounding a bit posh, so you force yourself to say "and I" in all situations because you think that must be the correct thing - perhaps there is a rule about "and" among posh people, or something, after all the Queen says it and it's her English. Or like David Baddiel you decide that you are fed up with all that pretentious twaddle and resolve to say "and me" in all situations and damn the grammarians. Or perhaps you steer a middle course, saying one or the other but not really being sure which is right, or why.
But actually there is no special rule about "and", and Her Majesty does use the word "me". You just need to use the same word that you would have used without the "blah blah and".