Rogue Subclasses - PHB

 The Rogue is one of the classic RPG archetypes, though it has come with different names in that history, from Thief to Assassin, sometimes just "Sneak." In D&D, Rogues have, I think, two defining traits. The first is their Sneak Attack, which allows them to do big bursts of damage with single, well-placed hits, which encourages a stealthy approach to foes. The other is their excellence with skills and ability checks, their reliability in the tasks that they are experts in (which is reinforced by the fact that you get four skills from your class rather than two.)

Interestingly, Rogue subclasses don't get a second subclass feature until level 9, which puts a bit of pressure on the level 3 traits to make things interesting for most of tier 2.

Arcane Trickster:

The Arcane Trickster is, along with the Fighter's Eldritch Knight, one of the two "one-third casters" in the game - less magic-based even than Paladins, Rangers, and Artificers, but supplementing their normal playstyle with a handful of magic spells. They get some extra things they can do with those spells, too.

At 3rd level, you get spellcasting, learning two wizard cantrips, along with Mage Hand as a third. You also learn three Wizard spells, but two of those must be from the enchantment or illusion schools. As you level up, you can learn more spells, but each of them has to be from those schools except for the one at 3rd level and the ones you get at 8th, 14th, and 20th level. Your spellcasting ability is Intelligence. Naturally, the enchantment and illusion schools work quite well with the Rogue's typical role of infiltrator, in contrast to the Eldritch Knight's abjuration and evocation, which are more designed for the heat of battle.

At 3 you also get Mage Hand Legerdemain. When you cast Mage Hand, you can make the spectral hand invisible and do additional tasks with it. You can stow or retrieve an object the hand is holding in a container worn or carried by another creature, and you can use thieves' tools to pick locks and disarm traps at range. You can do these without anyone noticing if you succeed on a Dexterity (Sleight of Hand) check against the creature's Wisdom (Perception). Additionally, you can use your Cunning Action to control the hand. Being able to disarm traps from far away is actually really great, making it way safer to do so. Also, being able to invisibly pickpocket other characters can make it a lot safer to make such attempts.

At level 9, you get Magical Ambush. If you are hidden from a creature when you cast a spell on it, the creature has advantage on the saving throw against that spell this turn. This is a cool way of effectively giving you "advantage" on your spells even when it's a saving throw spell - it seems totally logical and cool.

At level 13, you get Versatile Trickster. As a bonus action, you can designate a creature within 5 feet of your Mage Hand, granting you advantage on attacks against that creature until the end of the turn. I like to think that this is basically using the hand to tap them on the shoulder and distract them. Having a free and reliable way to get advantage on attacks as a rogue is really fantastic.

Finally, at level 17, you get Spell Thief. Immediately after a creature casts a spell that targets you or includes you in its area of effect, you can use your reaction to force the creature to make a saving throw with its spellcasting modifier, the DC being equal to your spell save DC. If they fail, the spell is negated and you steal the knowledge of the spell if it's 1st level or higher. For the next 8 hours, you can cast the spell using your spell slots (which means you'll never be able to do more than 4th level spells, but it doesn't have to be a Wizard spell.) That creature, then, can't cast that spell until the 8 hours have passed. You can use this once per long rest.

This feature is really awesome - it's basically a Counterspell, but in a way that not only will sometimes let you cast the spell as well (not always, if it's too high level,) but being able to lock a creature out of casting said spell is especially potent - if you're dealing with some monster with innate spellcasting and only a handful of spells, you can deny them one of their most important features.

So, the Arcane Trickster has a lot to recommend it, and just spellcasting in general gives you a lot of nice features, like being able to turn yourself invisible or charm that guard that spotted you.

Assassin:

The Assassin kind of shares the spot with the Thief as the most quintessential subclass for Rogues. You're all about taking out your target quickly and efficiently, though excitingly, this also has some solid RP material. Let's take a look.

At 3rd level, you get Assassinate. You have advantage on attack rolls against any creature that hasn't taken a turn yet in this combat. Additionally, any attack against a surprised target is a critical hit. Crits for Rogues do massive amounts of damage, so any way to get a guaranteed crit is really exciting, and can actually let you take a foe out in a single strike in the right circumstances. The advantage thing is a little tricky, because I think technically because of the "Surprised" condition being different from previous editions' "Surprise Round" might technically mean that if you have a bad initiative roll, you might not actually benefit from this. That said, you're a Dex-based class, so you'll usually have that benefit.

Additionally at 3, you get proficiency in the disguise kit and the poisoner's kit, which is great for infiltration. I'd recommend DMs consider allowing players to craft more powerful poisons found in the DMG to make the poisoner's kit more interesting.

At level 9, you get Infiltration Expertise. If you spend 7 days and 25 gold, you can establish the history, profession, and affiliations of a false identity (you can't make yourself appear to be another person who exists, though.) You'll have appropriate clothing and certifications and letters of introduction to pass yourself off as this false person. If you adopt this persona, people will accept this unless specifically given a reason not to. While this is really cool, this is also absolutely something I'd already let people do with proper roleplay, so this is weirdly kind of a nothing of a feature, unless you're super strict and don't let players come up with the sort of creative solutions that this feature implies. It's a weird, weird thing to make mechanically part of the subclass.

At level 13, you gain Imposter. If you spend three hours studying another person's speech, writing, and/or behavior, you can mimic these traits, getting advantage on Charisma (Deception) checks to avoid people noticing the falsehood. Again, this is very flavorful, but it's practically just what I think a good DM would give you anyway if you spent the time to do this in-game.

Finally, at level 17, you get Death Strike. When you attack and hit a creature that is surprised, it must make a Constitution save against a DC based on your Dexterity, and if it fails, your attack does double damage against the target. Given that such an attack is also automatically a crit, this is going to do absurd amounts of damage (by level 17 your Sneak Attack is 9d6, so if you're hitting with, like, a shortbow, that's 10d6, which is doubled to 20d6 on a crit, and thus effectively 40d6, which is 140 damage on average (before dex or any magical bonuses.)

So, man. The Assassin is a huge mixed bag. The level 3 and level 17 features are insanely powerful, and the middle ones are beautifully full of flavor but also kind of... don't need to be class features? I'm genuinely kind of in shock at this subclass. It's a fantastic 3-level dip (Gloomstalker/Assassin has maybe the most insane opener possible) and it rewards you nicely for hitting 17, but man are those middle features weird.

Thief:

The Thief is the somewhat less violent other "classic Rogue" archetype. You're all about moving undetected and having a solution to every problem you land in.

At 3rd level, you get Fast Hands. You can use your Cunning Action to make Dexterity (Sleight of Hand) checks, use your thieves' tools to disarm a trap or open a lock, or use an object. Nice expanded utility to get you out of tight situations and act quickly.

You also get Second-Story Work. Climbing no longer costs extra movement (is this the same as a climbing speed?) and if you make a running jump, you can jump a number of feet farther equal to your Dexterity modifier. Tons of flavor here, and again, this makes you great at getting into places you're not supposed to be (and excellent at rooftop chases.)

At level 9, you get Supreme Sneak. You have advantage on Dexterity (Stealth) check if you move no more than half your speed on the same turn. Given that a lot of sneaking happens outside of combat, when movement speed isn't really a concern, this will make you way more effective at sneaking around. But it'll also help if you want to slip up next to a foe in the heat of battle.

At level 13, you get Use Magic Device. You can ignore all class, race, and level requirements to use magic items. This is actually really cool - if you find a Staff of the Magi, there's nothing stopping your Rogue from using that thing. Find a Holy Avenger Scimitar? Use it! It's a great feature, especially getting it at this level, when you're likely to start seeing some of the really crazy items.

Finally, at level 17, you get Thief's Reflexes. You get to take two turns during the first round of any combat. Your first turn happens as normal, and then your second at your initiative score minus 10. You can't use this when you're surprised. Given how crucially important the action economy is in this game, getting a full on extra turn in every single combat is mind-blowingly good. That's an extra sneak attack. That's another movement. It's another reaction (because you get your reaction back at the start of your turn.) It's another bonus action. It's really, really, good.

So, looking at these three, all have some really great strengths. I think the Thief probably winds up being the best overall, though the spellcasting of the Arcane Trickster gives it a ton of versatility.