On the surface, it might appear that Science Fiction and Fantasy (by which we mean what is sometimes called "High Fantasy", or "Swords and Sorcery"), are opposites. One is more concerned with science than most fiction, and one is less concerned (magic being more or less by definition something that is unscientific, otherwise we don't call it 'magic' we usually call it 'technology'). And yet, nearly everyone knows that there is more overlap than disconnect between the readership. Plenty of people read both, or neither, each a bigger group than those who read SF-but-not-Fantasy or vice versa. This fact, which ought to be surprising, is not surprising to much of anyone.
Also well known, is that both genres started among nerds, and only slowly spread to other readers (after about a generation's lag). This is a clue as to how SF and F are more alike than different. Let's look at the definitions of "nerd" in wiktionary (at the time of this writing):
So, all of these have something in common: someone who cares more about something else, than they do about social interactions. The "something" may be science, or history, comic books, or really just about anything which most people don't care as much about. The defining fact of a nerd is that they are not as good at social interactions, because they are good at...something else. Even when socializing with one another, nerds utilize that 'something else' as a substitute for social skills of the more ordinary kind. Band. Chemistry. Drawing. Jesus (unless you're in a group that is very outwardly, i.e. socially pious, in which case that doesn't make you a nerd, but if you're the Jesus Kid in a school of atheists you are a nerd).
Before returning to SF and F, let's look at an even more basic question: what is the point of fiction? It's not just a desire for gossip, because there are billions of people out there who we can write (and film and narrate) the real-life stories of. Yet, we have fiction, and as far as we can tell we pretty much always have had fiction, in essentially every culture. Why stories about people who don't exist, when we could use stories about people who do (or did)?
One thing is for certain, it's not because real people are all boring. There is a lot of celebrity gossip that people manage to find interesting, and plenty of historical characters that we manage to make good stories about (or movies, or musicals, or etc). So why, when we have perfectly good and interesting real life histories to go to, do we routinely make (and avidly consume) stories about people who never existed?
I think the reason is kind of like why a diagram is often better than a photograph, or a map is often better than a satellite photo (even if we added labels to the roads in the satellite photo). It's not what's there that makes a fictional story, it's what's not there: the odd fiddly bits of real life. Even when doing non-fiction, authors have to do a lot of removing of inconvenient details: other people, revealing episodes in the main character's life that don't fit with the story arc, etc. We can better understand a story about how Pride Goeth Before A Fall, if we don't have all those bits of humility that the main character also demonstrated prior to the Fall, or all those bits of the Fall that were really about bad luck. We can understand a story about Love Conquers All a lot better, if we don't get the part where one of the characters did something practical and kind of backstabby in order to get or keep their love interest, or the part where they woke up the next morning thinking "geez this person's breath is awful in the morning". We use fiction to illustrate Essential Patterns about how people work (or fail to). This can happen with real historical lives, but quite often even then we will need to be a bit economical with the truth in order to keep the story from being drowned in a sea of inconvenient detail.
It's like lessons in anything else: we need simplified cases to learn from. Not too simple, or we don't learn much, but there is an optimal amount of complexity in any story, and real life normally goes far past that. Too many characters, too many events, too much, too messy. Fiction, when done well, provides just the optimal amount of complexity (which requires an estimation of who the reader is, e.g. a tale for a young audience should have an amount which is right at, neither below nor beyond, the amount of complexity they can handle).
There is, however, a certain implicit assumption in this, and that is that the people in the story, have psychologies which are more or less like ours. Perhaps not precisely the same, but close enough that we could imagine meeting such a person in real life. It is not going to matter to most readers, if the story requires that the laws of physics or chemistry be different than they are (which occasionally happens even in mainstream, non-SFF). It IS going to matter, to most readers, if the characters do not behave in a way which we can imagine real people behaving if they were in that situation. If faced with danger, the author can have the character give in to their fear or overcome it, perhaps even overcome it easily, but if they never really felt any, then that is a flaw in the characterization. They can respond to being rejected in love gracefully, or poorly, or go into denial, but if they don't seem to be impacted by it at all, that is a flaw in the characterization. This is because the purpose of most fiction is to think about how people work, which requires that the reader is able to imagine them as a real person, and if their brains don't work like human brains that hits the fundamental purpose of fiction. Most readers can deal with anthropomorphized animals as characters, with no problem, but if they truly thought like tortoises, or dogs, or frogs, the typical reader will not care to read a story like that.
Which brings us around to Science Fiction and Fantasy, the nerds of fiction genres. Not because they are read by nerds, but because they are nerds, in their essence, i.e. they are most concerned with something other than human psychology. Understanding how people behave, and interact (in couples or small groups), is what fiction is mostly about. SFF is fundamentally about other things.
Now, many will object that it is perfectly possible to write SFF with good characterization, and so it is. Notably, this is something that took hold decades after the birth of Science Fiction. Moreover, there is a pretty clear rift in current fandom between those who want character-driven stories, and those who don't, and the latter have the (often incoherently stated) feeling that they are being driven out of their favorite genre by people who really don't understand it. That is because, in a very real sense, the newer (non-nerd) readers of SFF don't understand it. They often seem to act as if it is normal fiction, with better costumes and fancier names.
It was once said, in Science Fiction, that "The Hero Is The Idea". I don't see this much anymore, because it is no longer the way much of SFF works. There is also a bit of disagreement as to how much "world building" matters. If the battle in the story shows a fundamental disregard for the way in which military tactics work (or fail to), is that important? If the way in which religion is portrayed in that SFF story does not match how religion actually works throughout history, does that matter? If the society does not appear to have any way of feeding itself, given that everyone seems to be going around being Dramatic Heroes or Villains, but nobody is being a peasant, does that matter? It all depends on how much you, as a reader, wish to think about the plausibility of the setting, from the point of view of sociology, anthropology, history, biology. If you care mostly about characters, then the situation that it puts those characters in, is all that really matters. If you care mostly about something other than characters, then how the characters react matters little, if your brain is still thinking, "wait a minute, everyone in this story is trash-talking the religion and its leaders, more or less openly, so why does this religion even exist anymore?" An acquaintance with the history and sociology of religion will tell you that if (like e.g. the Church of Reason in revolutionary France) nobody really believes in the religion, it goes away pretty quickly. But, if you're mostly wanting to explore how the characters deal with the fact that they don't believe their society's religion, this is a detail which doesn't matter.
Of course, even in SF there are physical laws which are ignored. Spaceships travel faster than light with only hand-waving explanations, spacefights happen with audible explosions and flight paths that appear to be taking place in atmospheres rather than the vacuum of space, etc. But the part of the story which is most important, is something other than the normal fare of small-scale human interactions. For example:
In other words, just as fiction creates imaginary people in order to think more clearly about how people work, SFF creates imaginary technologies, and societies, in order to think more clearly about how technologies and societies work.
Now, it is certainly possible to do both at once. A good SFF story can be written to have engaging characters. But, and this is the key difference, SFF stories can actually be good without engaging characters, if the Idea is interesting enough. This, however, requires that you be interested in something else, as much as you are interested in (small-scale) social interactions. If you're not, particularly, then the idea of a really good story with not particularly well fleshed-out characters is a contradiction in terms. If you're a nerd, then it is not. Having well fleshed out characters may be nice, but it's not necessary.
It's much like the difference between getting the forensics right in a novel about a woman whose husband is murdered, who must pick up the shattered pieces of her life and learn to find love again, vs. getting the forensics right in a crime novel. Both have a murder, both are better if they get the forensics right, but if you're writing a whodunnit then the forensics are close to the core purpose of the genre. Well-fleshed-out characters are close to the core purpose of most non-SFF writing. For SFF, they are like forensics in a romance novel, or legalities in something that is not a courtroom drama. Nice, but not essential.
World-building, in SFF, is close to the core purpose. If you get that wrong, you are writing something like a whodunnit with correct punctuation but an implausible way for the detective to catch the criminal, or an historical novel with nice use of language to describe the landscape, but lots of anachronisms. The thing you got right, is less important than the thing you got wrong, for this genre. "World-building", or more generally the Idea, matters more than the characters. If that doesn't seem right to you, then regardless of what kind of fiction you read (or watch, or listen to), you are not a nerd.