Let’s look at Earth. We have exactly one sentient species: us. Humans. (Although, it can be argued that other advanced species – elephants, gorillas, dolphins, whales – are sentient, depending on what particular factors one’s looking at. But, for the sake of argument, let’s say sentient in this case means regular tool use, complex language, some control over the elements, and self-awareness and consciousness.)
Your average fantasy world, on the other hand, can have several: humans, elves, dwarves, orcs, occasionally dragons, goblins, kobolds, halflings, gnomes, trolls, etc.
These species tend to be on a more or less even footing.
So, how is it that you can have several sentient species on a fantasy planet when science says that competition will limit species of a similar niche?
I know I said the fantasy doesn’t have to follow the laws of science, but on one hand, this is something that bothers me. It may be because a lot of fantasy closely mirrors the real world with the exception of its many species and possible use of magic, so ignoring how evolution works seems a bit odd to me.
The being said, there are ways to have multiple species on the same planet and not break science.
1) You can have them all be related, like emus and ostrichs are related. Common ancestor in the past, isolated populations, etc. This isn’t that hard. Dwarves, for example, typically live underground or in the mountains. In general, most fantasy species come in similar builds and colors, so it’s generally believable.
2) For worlds where you want two widely different sentient species (such as dragons and elves, for example), if you have something that keeps them from competing earlier in the evolutionary phase, such as them being on different continents, or low-birth rates where there’s not a lot of spread, then you’re more likely to have two advanced species come into being.
You can, of course, always make the argument because it doesn’t matter because it’s fantasy. It depends on the reader whether or not they’ll be bothered by it. I don’t tend to worry about it unless there’s a truly ridiculous amount of species or something else stands out as being very strange. I’m more bothered by species that are supposed to be natural but seemingly have no natural reason for existing.