Van Richten's gave us two interesting horror monsters - creatures designed to give you the equivalent of a horror-movie slasher - collectively called the Relentless Killers.
In Vecna: Eve of Ruin, the final major adventure book published for 5.0 (and the last "one big adventure" book we've gotten for D&D in a few years) the chapter that takes the party to Ravenloft has them enter the Death House (famed for its level 1-2 adventure in Curse of Strahd) stalked by a new variant that has an impaling spear.
I don't actually have my copy of Eve of Ruin here, so I don't know to what degree the Relentless Nightmare matches that creature, but we'll look at the Horrors Within version of it on its own terms.
All of these are considered fiends, though not necessarily because they are from the Lower Planes. We can probably assume they're primarily found in the Domains of Dread, and their fiendish nature is just to show that they are not undead, but are totally committed to cruelty and pain, no longer worthy of your sympathy.
The reason why they're inhuman might never be known - like Michael Myers, the evil might just be something inherent to them.
The Relentless Nightmare, I think, vaguely gestures toward another icon of the 1980s slasher movie, Freddy Kreuger. While they're not quite as versatile in the ways they can hunt their victims, Relentless Nightmares assault the mind as well as the body, and notably are very difficult to destroy permanently.
Sitting at CR 11, this puts them at just slightly less of a challenge than the Juggernaut (CR 12) and a bit higher than the Slasher (CR 8). For a 4-player party, they should be a high challenge for an 8th level party, a moderate challenge for a 9th level party, and a low challenge for an 11th-level party.
Given the story of these monsters, I don't think it's appropriate to ever have your party face more than one at a time (well, maybe if you want the "double Pyramid Head" boss fight from near the end of Silent Hill 2 you could have them fight two Juggernauts) but these are legendary creatures and I really think are meant to be solo monsters.
Naturally, it's hard to recreate horror as we see it in movies in D&D because player characters are all designed to be capable monster-hunters. We're generally expecting to run toward monsters rather than away from them.
But I think that the Relentless Nightmare has a combination of abilities that can potentially make them profoundly scary:
Each turn, they can use Exhausting Gaze, which can inflict some psychic damage and a level of Exhaustion on a target that fails a wisdom saving throw. Remember that 5.5's exhaustion reduces a creature's speed by 5 feet per level of exhaustion and also reduces the results of their D20 tests by 2 per exhaustion level. Thus, this penalty can compound as it becomes harder to hit the DC.
However, the second part of this one-two punch is that their Haunter's Spear will automatically curse anyone it hits. In addition to the piercing and psychic damage the spear does, the curse prevents the character from reducing their exhaustion by taking a Long Rest.
Now, unlike a Rakshasa, whose curse just prevents a Short or Long rest entirely (truly one of the nastiest monsters,) you do have some recourse here - taking a rest might not cure exhaustion, but you could still swap in a Remove Curse or Greater Restoration spell.
But there are some other factors at play as well:
First off, the Nightmare has a legendary action that allows it to automatically make a cursed creature within 120 feet fall unconscious and take some (light at the level they'd be facing one of these) psychic damage. There's no save against this - if you're cursed, they can just do this. And given that the curse only needs them to hit a target, this is very likely to be available to them.
I think the best use of this is to deal with a tank-like character. If the Nightmare can land a hit (and their other legendary action option allows them to teleport across any distance to a creature it can see and make its Haunter's Spear attack - which actually might make tanks less of a problem anyway) they can knock someone out.
Lastly, they're also very hard to kill permanently. In 1d10 days after they die, they can revive next to an unconscious creature bearing their curse.
Now, if they've only cursed members of the party, this might not be a factor, given that the players likely have figured out that they have a curse and taken steps to cleanse themselves of it. However, if you want to keep this monster around to torment the party, simply have them curse an NPC and leave them alive. They have an Intelligence of 13, so they're rational enough to plan ahead in this way.
If you want to play very strictly, you might say that they had to target a creature with at least 25 HP so that a single hit of their spear wouldn't outright kill them, but this is an area where I think some cutscene magic and a pulled punch could allow them to have struck, like, a child without dealing any damage, using the innocent to keep coming back. Maybe they've cursed tons of people.
I think the numbers we're looking at, and the real necessity to have at least Remove Curse in the party's arsenal, means that we'd for sure not want to deploy these until tier 2 - unless you want a fake-out TPK against a tier 1 party (even then I'd lean toward the later levels of tier 1 just to make sure that the spear attacks don't take people down outright).
But to keep the nightmare factor at play, I wouldn't wait too long to menace the party with one of these guys. Even fudging it a little to send them against a 6th- or 7th-level party (which technically puts it beyond the High Difficulty XP cap for a 4-player party) might be ok to really make them feel like an overwhelming threat. However, this first appearance might simply be to curse a target and then vanish (especially in Ravenloft I think you can feel free to pull some "invisible teleporting" BS to up the spooky factor). The curse might not even be apparent, especially if the characters don't have any levels of exhaustion left.
Indeed, this might be a good monster to deploy suddenly and briefly in a non-combat scene. Maybe the party is doing some general business in town (if it is Ravenloft, that might not necessarily be a place of safety and comfort, but even horror needs some moments of calm to reset the tension) and this thing emerges from some impossible space to jab at the character with their spear before vanishing (they could even use Dreamwalk to teleport to a passing crow or something if you wanted to strictly play by the book). Maybe they jab the target and then use the Narcotize legendary action to immediately cause the character to collapse unconscious, so their escape isn't witnessed.
I do think that probably all of the Relentless Killers are good options for disrupting Pursuer-type monsters, like Mr. X from Resident Evil 2 - if the party is in the middle of some slow dungeon crawl, having an over-leveled Relentless Killer show up can suddenly turn what had been a bunch of gradual exploration-focused rooms into a deadly obstacle course. The Relentless Nightmare can easily catch up with any PCs, so the players really cannot outpace them without blocking line of sight, meaning they're more likely to stand their ground.
If you are deploying the Nightmare against a lower-level party (again, probably not any lower than 6. Maybe, maybe 5 if you want to stretch it) I think you should think about the monster's motivation. Are they going there just to kill people? They certainly have the tools to do so, and the party will probably just fight back and stands a good chance of actually killing it as long as there are some folks with high ACs (it has a +9 to hit, so even with a high AC, the party might just get unlucky.
However, I think the monster might be more interesting if their motives are not to just immediately kill the party, but to disrupt them and separate them. A Relentless Nightmare can pretty quickly and with relative ease knock an entire party unconscious - they can literally do two PCs per round if they use Narcotize with two of their legendary actions, and can even set up to do more of them on a subsequent turn.
The strange thing in horror is that if you just slaughter everyone, it's not really scary anymore, because once all the PCs are irrevocably dead, there's no agency anymore and the player has to separate their feelings from their characters' fates. But if you use this to put them into new and more dangerous positions, that can up the terror.
Perhaps the Nightmare's goal is not to directly kill the party, but instead to knock everyone out and then drag their bodies to some deeper chamber of a dungeon, putting them in the way of other monsters or traps. The panic and desperation of a fight against a Nightmare should probably be built around watching your allies fall unconscious one by one, and forcing you to use your action to wake up an ally rather than put more damage on the monster.
The Nightmarish Restoration element here can make this a recurring threat - I'd be inclined to have them show up a little too early in an unrelated dungeon - maybe the party is going to some dilapidated hunting lodge in the woods, and it's a dungeon that's mostly just about exploration with maybe a few minor threats in it, but then, about halfway through, they look back to some room that they'd already explored, and this thing appears. There's zero explanation for what they are or why they are hunting the party, but after a desperate fight, either the Nightmare is slain or the party is all knocked out and mysteriously finds themselves in some earthen basement below the lodge.
The initial encounter being seemingly at random, the party might not see the thing for a while, until it shows up at some other location to attack them. Eventually, the party might start to learn more about the Nightmare, the folklore surrounding it and the rumors about how it works.
I actually don't think I'd build an entire campaign around one of these monsters, but instead have it be a major recurring threat - a kind of secondary plotline woven through a campaign. If you like to have elements of your campaigns based on your PCs' backstories, this could work very well as a throughline for such a plot.
As a note: while the Narcotize legendary action simply says that it gives the Unconscious condition, I'd interpret that as magically-induced sleep. Thus, if you have an Elf, Reborn, or Warforged member of the party (or any other character that doesn't sleep), you might consider granting them immunity at least to the unconsciousness. This will change the way the encounter works, but remember that you can always knock such characters unconscious simply through damage. Likewise, if you have a Kalashtar, who do sleep but don't dream, I might make it harder for the Nightmare to use its Nightmarish Restoration feature if they're forced to use a Kalshtar character to resurrect.