Sure, aliens may not be as endangered in science fiction as time or interstellar travel, but they have their detractors as well.
The possibility of there being other intelligent life in the galaxy is typically defined by Drake’s Equation, but the problem is that there are several variables in the equation, very few of which are known. (We don’t know how many stars have solar systems, and how many planets are in a habitable zone, etc.)
Drake’s Equation varies wildly based on what you use as inputs.
Even if we ignore Drake’s Equation, alien opponents point out that we’ve been trying for several decades to both contact and find signals from other civilizations, with no answer. They argue that it’s a fluke that life started on this planet, and that even, by some rare possibility, it managed to happen somewhere else, there’s no guarantee that that life would reach sentience in a time frame that would allow them to interact with us.
What does that mean? It means that even if a civilization reached a level of technology to allow for interstellar communication in a form that we could detect, if they did it a million years before or after we were looking, it would all be for naught.
So some people argue that aliens aren’t scientifically realistic. Where’s the fun in that?
The other argument you run into with aliens is what sort of lifeforms they’ll be. Early Star Trek, for example, explores all sorts of strange non-carbon based lifeforms. But some peope say that since all life here on Earth is carbon-based, that it’s unrealistic to think that life can come in bases.
(However, we recently discovered arsenic-based life here on Earth, so you should ignore those people.)
The argument for or against aliens can be compared to the argument for or against God. There’s not really any evidence one way or the other, so why go with the more depressing option? (Though, I suppose if you believe that most aliens are the invasion type, then not believing they exist is the better option.)