A friend of mine served as Editor for the new far-future/post-human weird sci-fi RPG Stillfleet, the core rulebook for which I received as a gift for the holiday this year. The game is a departure from D&D's d20-based systems. There is combat in this game, but it's not emphasized the way it is in D&D - the game is built to allow "killing aliens and robots" to be a focus if that's what you're going for, but also focuses on a kind of far-future Dutch East India Company laying the track for a colonialist expansion across the cosmos and the players' role in either facilitating that or opposing it (the latter generally through subversion than direct opposition, given the concentration of power.)
There are options for crunchy optimization, but it's a very different kind of system - class progression is more about getting more options of what you can do than hitting harder.
It's always hard to get into a game that you haven't played - I'm going to try to look up some actual play videos to help me wrap my head around it as I work my way through the textbook-sized rules, but I've had discussions with this friend and in general about the challenges of, well, being an RPG that isn't Dungeons & Dragons.
When you watch videos by Matt Colville, he'll often dismiss 5E's popularity as more a function of luck than anything special about the system. He is, naturally, inclined to downplay 5E's popularity given that the RPG system he is working on is explicitly being built to compete with it, but we can also reverse cause and effect here as well - he is eager to make his own RPG because of the flaws he has identified in 5th Edition.
The "OGL Debacle" or whatever we wind up calling it from the early part of this year set off a bevy of new projects from longtime 5E contributors.
As I see it, there are four notable projects arising either out of this crisis that were either conceived because of the crisis or, more likely, given higher priority because of the fears of the death of the OGL.
Now, Wizards of the Coast did backpedal hard - 5E's SRD (the sort of primary corpus of the rules that includes things like "this is how Rogues work" and "this is how Advantage works" and, I believe, things like "these are the stats of a Goblin") has been fully handed over to Creative Commons, taking it fully out of Wizards' hands and, essentially, perpetually making the SRD usable by anyone who wants to publish and sell 5E content. I don't know if the changes coming in the 2024 rulebooks will be added to this SRD, but at the very least, nothing is being taken away from content creators, and none of the overzealous language of the original OGL change announcement appears to have survived.
But, these projects were announced and are moving forward:
Kobold Press is working on Tales of the Valiant (under the umbrella of "Project Black Flag.") Kobold Press was founded by former D&D designers and worked on early 5E projects like the Tyranny of Dragons adventure books and Sword Coast Adventurer's Guide, and has published massive monster books in their "Tome of Beasts" 1 and 2 and Creature Codex, among other things. At least in their initial playtest documents, their RPG looks to be replicating very closely the overall form of 5E, but with some interesting new ideas like separating Ancestries and Heritages, for example.
Pathfinder of course came out during 4th Edition, and was built essentially to be a more direct successor to D&D's 3rd Edition (and 3.5th Edition) after 4E had made massive changes. Pathfinder came out with its 2nd Edition a few years ago, but given that it was still built on the OGL, Paizo underwent a project to come out with "2nd Edition Remastered" which sought to utterly strip the game of anything that could be construed as being built on D&D's rules systems. Pathfinder was, for a time, more popular than D&D, but I think 5th Edition pretty handily won back the community, and of course the TTRPG hobby has exploded with 5E's success, meaning that any edge Pathfinder had had before 2014 is basically negligible at this stage.
MCDM's as-yet-unnamed RPG is, of course, the one of these I've been paying closest attention to, but it seems to be built with a mission statement to drop any mechanics that the designers feel are only still around because things have "just always been that way." While Colville and lead designer James Introcaso have been very candid about the design process, I'm not currently aware of how it will all work, exactly, but the core of their design has been to start with four keywords - Tactical Cinematic Heroic Fantasy. By dropping things like intense inventory management, I think they're trying to get the game to play naturally the way that a lot of tables handwave uninteresting (or, more accurately, out-of-genre) mechanics, and with things as radical as getting rid of attack rolls, they're hoping to minimize uninteresting moments in gameplay.
Finally, Darrington Press is working on their Daggerheart game, which is meant to be a more role-play focused fantasy RPG. This latter one is interesting because Critical Role (the parent company of Darrington Press) played a big part in popularizing 5th Edition and the actual play style of entertainment in general (though I always feel a need to shout out Acquisitions Incorporated for both being my introduction to D&D and also sort of inventing the format,) and thus they have a big platform to promote this system - I would not be shocked to find that Critical Role's fourth campaign will use this instead of 5E.
Now, obviously, there have been plenty of non-D&D systems out there since the genre was born in the 1970s. But I think there are points of inertia that make it harder for any of these systems to get the kind of popularity and cultural relevance.
To be sure: lots of these games are successful and profitable. My friend's game, Stillfleet, has now published additional supplements like published ventures (note: not adventures) and they've got other stuff cooking that I don't know if I'm allowed to talk about, so it seems to be going well.
But because TTRPGs require a pretty thorough understanding of the rules to play and certainly to run, it's much harder to just "try out" another system, especially if it has a complexity level similar to D&D. I got the Starfinder core rulebook and its first Alien Archive (and in fact the second one and "Armory") a while ago and have never been able to get my friends to play it, and barely have the energy to run it.
While I jumped into DM'ing immediately, I still benefitted greatly from both watching Acq Inc and having a player who was already familiar with the system. But that requires a kind of continual lineage of familiarity - I ran it for my friend Tim who now runs the game professionally (and has an actual play show: Watch Legacy of Fools on Tuesdays!)
In other words, there's a lot of inertia. WotC is, wisely I think, working very hard not just to achieve the goal of making the 2024 rulebooks compatible with all the 5E products from the last nine years (I don't think they'll accomplish this with 100% compatibility, but it'll be close enough that no table should have any trouble navigating the changes - call it 97.5%). So, they have both the 50-year brand recognition and the 10-year familiarity with the system to make it particularly easy to get into the game.
The other challenge these games have set for themselves is that they all appear to be fully in the same genre as D&D. These days, if you want to play a modern-ish cosmic horror game, Call of Cthulhu is probably your obvious go-to. If you want to play a modern-day goth vampire game, you have Vampire the Masquerade. And if you want to play as fantasy heroes fighting monsters, D&D has you pretty much covered.
So these games need to be really good - they need to be not just better than D&D, but so much better that players will set aside the system mastery they have developed (both in running and playing the game) to go through the growing pains of figuring out a new system.
And to be clear: I hope they succeed! 5E is basically the most fun game I've ever played. So any game that is better than this one that I've become obsessed with for the last eight years would be a phenomenally good game indeed.
I think that a world in which more of these systems are not just around but also popular enough to compete with D&D should, I hope, spur one another to improve. Is 5E's 2024 revision conservative? For sure. Will it not accomplish all the things I hope it will? For sure. But maybe, if we see other games grow in popularity, the designers at WotC (the ones that Hasbro hasn't laid off, at least) will feel more confident in trying new and innovative ideas.
There's a new frontier in TTRPGs opening up, and like any new frontier, it's going to be filled with dangers and failures, but there's the potential for greatness.