In 5.5, the old Barbarian feature Brutal Critical was replaced.
The old feature, conceptually, was cool: Barbarians would get particularly nasty critical hits when those landed. But the actual math on them didn't truly do much. Even though your critical strike chance is higher if you're using Reckless Attack (or some other method,) the small chance of actually getting a crit meant that that bonus damage was hardly the equivalent to a Fighter's third attack at level 11 or a Paladin's extra d8s on everything (though it did come two levels earlier). This did give the d12 Greataxe a bit of an advantage over the 2d6 Greatsword or Maul, because the feature only added a single die from the weapon at each upgrade point - so while the Greatsword is adding a d6, then 2d6, then 3d6, the Greataxe is adding a d12, then 2d12, then 3d12.
But when I've calculated that in the past, the small chance of getting a crit meant that it wasn't really until you got that third bonus crit die that the Greataxe finally overtook the damage of a Greatsword or Maul (though I might need to look back and see if I was accounting for Reckless Attack) because the 2d6 weapons do an average of half a point of damage more with each hit.
Brutal Strikes works quite differently. But it's also kind of weird, because while it does add a d10, and later 2d10 damage to your hit, the penalty you take, forgoing advantage, means that in most cases, that extra d10's effective damage per attack (rather than hit) winds up being about a wash, and sometimes even a DPS loss.
It is all dependent, of course, on your hit chance. With a higher attack bonus or lower AC of the target (in other words, the lower we need to roll on the die to hit) that d10 starts to look more appealing. But if I recall correctly, you need to have a pretty high hit chance for that to be the case.
Thus, I think, the real way to treat Brutal Strikes is not as a means for Barbarians to increase their damage output, but more as a means to get some additional utility with the various effects you can choose.
But let's actually figure out the math:
The stakes here are established by what we stand to lose versus what we stand to gain. A miss means we lose out on the rolled damage of our weapon, the Strength bonus, the rage bonus, and any other damage added by a magical weapon or other effect.
To simplify things, we'll ignore magic weapons. Instead, let's make the following assumptions:
In 5.5, it's a bit more encouraged to take General Feats other than Ability Score Improvements, because each of these give you half of one of those anyway along with some potentially big effects. Great Weapon Master is, I think, almost always the best of these for a Strength-based character (though I do think my Eldritch Knight with a Staff managed to beat my Eldritch Knight with a Greatsword, which was surprising). As a note, though, GWM will certainly skew things a bit, because it increases our output with a Great Weapon by a whole lot, and on top of that, it really likes it when we crit, which we will be less likely to do if we forgo advantage.
Is it realistic, then, to assume that we're just going with ASIs? This would make it very easy to have a 20 Strength by level 9 (when we get the feature) and while I doubt many players would eschew those really good martial feats (GWM, Polearm Master, maybe Dual Wielder. Not sure if there are any good ones from Arcana Unleahsed for a Barbarian) the Barbarian is a class that has a lot of stats they might want to boost (it would be pretty cool to have capped both Strength and Constitution before level 20).
Going with flat ASIs will perhaps extend more grace to Brutal Strikes than it's likely to have in a real character build. If it still winds up being pretty underwhelming, that will be doubly damning proof, but it might make it look better than it really is.
Still, we'll go for that because, hey, it makes my math simpler to do.
Thus, by level 9, we'll assume we're capped out at 20 Strength. Our Rage bonus is now +3, and if we're fighting with a Greataxe (while we're no longer incentivized to go with that thanks to the loss of Brutal Critical, it is a classic weapon for a Barbarian) that means our attacks have a +9 to hit and land for 1d12+8, or 14.5 on average, with a bonus from crits of 6.5.
Enemy ACs vary, of course, and especially at these mid levels, it's not impossible that you'll fight some really low-AC minions or some really high-AC legendary monsters. So, and I hope I don't regret this, we're going to see how this goes at ACs of 10, 15, and 20.
AC 10:
Without Brutal Strike:
Ok, here we're taking full advantage of our Advantage. Because our attack bonus is so high, this is actually going to be exactly the same math as if we were attacking something with an AC of 11, because the only way we can miss is thanks to the critical failure rule for rolling a natural 1. Thus, we have a 95% chance to hit, doing so as long as we don't roll a 1.
With advantage, though, the chance of a miss diminishes profoundly, from 5% to 0.25%, or a hit chance of 99.75%. As usual with advantage and no expanded critical hit range, our chance get a critical hit is 9.75%. So, 99.75% of 14.5 is... basically 14.5, if we round to the tenths place, or I guess 14.46 if we want to get slightly more granular. Then, 6.5x9.75% is about .63, so our total damage per attack when we take the advantage is 15.09.
With Brutal Strike:
Our hit chance is still very high here, and we're adding on a d10, so the hit damage jumps up by 5.5 to a flat 20, while our crit damage becomes an extra 12. Our hit chance is now simply 95%, while crits have gone to 5%. So, 95% of 20 is... 19 (this is easy given how familiar we are with probabilities out of 20 given the die we use) and 5% of 12 is .6, so we get 19.6 average damage, with some extra utility to boot.
Clearly, against this low AC, the advantage is not really necessary, and the extra d10 is already nearly doubling the d12 like it would on a crit. So, here, the damage favors Brutal Strikes.
AC 15:
Without Brutal Strike:
Damage numbers on hit/crit are all going to look the same, but the chances have shifted a lot.
So, now our +9 has to land a 6 or higher on the d20 to hit. That's a 75% chance, and with advantage, our 25% miss chance diminishes to a 6.25% chance, or in other words, our hit chance is 93.75% Still quite good. Crits are the same as well, in basically every sense, so we can just talk about that as the same .63. So, we take the 14.5 on a hit times the 93.75% chance to hit, and that gives us 13.59, then add in the crit to bring us to 14.22.
With Brutal Strike:
Here, that 75% chance is fully in effect. Hitting with the Brutal Strike gives us that flat 20, so we're looking at 15 damage for hits, and then our 5% crit chance gives us the same .6 from before, so it's 15.6 average damage.
We're still ahead, but only by a little bit - and once again, we need to remember that this is a character who does not have a magic weapon and isn't using Great Weapon Master.
AC 20:
Without Brutal Strike:
Ok, the only things that change here are the hit chances, so we now need to roll an 11 or higher, which is, very conveniently, a 50% hit chance. Getting advantage turns that into a 75% hit chance. So, 75%x14.5 is 10.88 (again, we're rounding to the hundredths place) and we get that same .63 to add for the crits, giving us 11.51.
With Brutal Strike:
Now we're really at a 50% hit chance, so we take that 20 we get on a hit and turn it into a 10. Then, that .6 for crits gets tacked onto it, and now we're talking a 10.6, simple enough.
So, what does this amount to? Well, at high ACs, the damage you get from Brutal Strikes will, even in a scenario that really favors them, not be enough to make up for the loss of advantage.
That leads us to the following:
What skews this one way or another? (Here I'm going to go more abstract because I'm too lazy to do the math on all of this.)
First off, at higher levels, the bonus damage jumps to 2d10. That will certainly help Brutal Strike look more appealing, and lowering the minimum hit chance it requires to outpace not using it (not to mention the new utility features).
But on the other hand, feats like Great Weapon Master, along with magic weapons like a Vicious Weapon or buff spells from allies like Crusader's Mantle will all work to inflate the value of actually landing hits. That d10 can start go get lost in the crowd if we're adding a d4+2d6. I suspect that even GWM on its own (especially as it scales quite well at higher levels) will overtake Brutal Strike's advantages in terms of pure damage output.
+X magic weapons have a somewhat nonobvious effect, because while they increase the damage of a hit (which would push Brutal Strike down in value,) they also increase your chance to hit (pushing Brutal Strike's value up). In fact, I suspect that the higher hit chance likely outweighs the bonus damage's effect.
Here's what I'll say, and consider as recommendation: in all of our scenarios, the damage output is actually quite close between the two options, whichever is on top. Now, again, maybe some other scenarios will change that (I probably ought to do a GWM version of these). But I think that kind of gives us an answer regardless:
Brutal Stirke is not going to be a major method for directly improving your damage output. The d10, and later 2d10, are there primarily to compensate you for the loss of advantage when going for the various utility options the feature provides.
I doubt that this was the design intent: I imagine that Barbarians were meant to get a nice big damage boost, but I don't think this accomplishes it.
That being said, the utility it provides is very nice, and while some might seem redundant with various weapon masteries, consider first that you can use them while taking advantage of a different weapon mastery, and furthermore, that if you have a weapon mastery that creates a similar effect, you can combine them to have a far greater effect, like pushing a foe 25 feet with a combination of the Push Mastery and Forceful Blow.
Ultimately, if you want to know exactly when to use Brutal Strike or not to maximize your damage, you'll need something like a spreadsheet and also to know an enemy's AC (which your DM might not tell you - I generally let people know if they "just miss," meaning getting an attack roll 1 below their AC, or "just hit" when they hit the AC exactly, but otherwise reserve this information as a secret - though I'll also tell them if it's a humanoid in plate armor, and let them correctly conclude that they have an AC of 18). But I'd also encourage you to just kind of "feel into it," and then also possibly just think more about whether you need the utility effects of the feature for this particular fight (Forceful Blow does a lot more damage if the target is 10 feet from the edge of a cliff, after all!)
EDIT:
Ok, just because it feels worth investigating:
What happens when the bonus jumps to 2d10?
This happens at level 17, meaning we've gotten to tier 4. At this stage, I'd dearly hope that your DM has given you some really powerful magic weapons, even potentially an Artifact (the Barbarian in my longrunning Ravnica game has had a slightly tweaked version of the Sword of Zariel since about level 14 or 15, which I reflavored as being a Benalian weapon passed down by champions of Serra). Any bonuses to our damage when we hit will naturally de-value Brutal Strikes given that we're less likely to get that damage. But with Brutal Strike now dealing twice as much bonus damage, its value is obviously improved.
My conclusion when I wrote the earlier part of this post yesterday was that, given that you only stand to gain about a single point of damage in the most favorable scenarios, we really can't treat Brutal Strike as a means to improve Barbarian damage, and instead just see that d10 as compensation. A fairly equivalent feature could have allowed us to apply the secondary utility effects of the feature to a target when we attack recklessly once per turn without adding any damage if we were simply allowed to keep our advantage.
But how does that start to look when we double the d10s?
Ok, so, at level 17, we'll assume Strength is capped. It would be very reasonable to assume that we have at bare minimum a +1 magic weapon, but as usual, we'll just ignore magic weapons for the sake of this math. Again, I actually suspect that the greater chance to hit with a +X weapon favors Brutal Strike more than the +X bonus to their damage de-values it.
So, actually, then, maybe we want to pour all the benefits into this to prove my thesis that the damage boost is really not enough. So, sure, let's say you have a +3 Greataxe.
This means that, at level 17, we've got a +14 to hit, and our attacks, when raging, deal 1d12+12 damage (5 from Strength, 4 from Rage, 3 from the weapon).
Our previous AC ranges become a bit redundant - against an AC of 10 or 15, we actually have the same hit chance, because the only way to miss anything with an AC of 15 or less is to roll a natural 1. So we can just combine these two. At these levels, though, it might be worth looking into how things go if we attack an AC of 25. Only three creatures, to my knowledge, have that (the three CR 30 monsters in 5E) though technically there's one that can get higher (Sul Khatesh from Eberron has a base AC of 22, but can cast Shield at will, meaning she can effectively have an AC of 27 if the DM chooses to use that reaction every time). It seems worthwhile to give us a range, even if higher ACs will devalue Brutal Strike.
AC 15:
Without Brutal Strikes:
The damage of a hit is going to be 18.5, and a crit gets us, as always, 6.5 extra.
So, this is our 95% hit chance, but because we have advantage, we'll be getting instead a 99.75% chance to hit. 99.75% of 18.5 is 18.45, and then we have that same as always .63 from crits, so the average damage here is 19.08.
With Brutal Strikes:
Now, we add 2d10 to both the hit and crit damage, pumping hit damage up to 29.5 and crits up to 17.5.
95% of 29.5 is 28.03, and 5% of 17.5 is .88 (rounded) so we're looking at 28.91.
Ten damage is maybe not as much as it was at earlier levels, but it's still a pretty significant boost. So, against a low AC, Brutal Strike actually looks quite worth it for the damage.
AC 20:
Ok, now we need to roll a 6 or higher, meaning a 75% chance (huh, these percentages look familiar).
Without Brutal Strike:
The 75% becomes 93.75% with advantage, so we get 18.5x93.75%, or 17.34, and then add the .63 from crits to get 17.97.
With Brutal Strike:
75% for real gives us 29.5x75% for 22.13 (sort of regretting rounding to the hundredths place, because this is a round number if we go to the thousandths) and then we get the .88 crit damage for almost precisely 23 damage.
Our lead has shrunk, but we are still actually ahead. Guess that second d10 does help!
AC 25:
Now, we're going to need to roll an 11 or higher, so we get a 50% hit chance.
Without Brutal Strikes:
Our hit chance becomes 75% with advantage, so we can look at 18.5x75% for 13.88 damage and then the extra .63 for crits, for 14.51 damage.
With Brutal Strikes:
29.5x50% is 14.75, and then an extra .88 for crits puts us at 15.63.
So, yeah, that second d10 does mean that we're looking at a damage boost at all ACs by this level.
However: here's the main caveat: I highly doubt that a 17th level Barbarian will have taken nothing but ASIs. If they're fighting with a Greataxe, they've almost certainly picked up Great Weapon Master by this point.
Now, GWM isn't going to just hand the non-Brutal Strikes option 6 damage that the Brutal Strike version doesn't get. In fact, the difference should be pretty easy to figure out, at least before we get to the Hew part of it. By this level, GWM gives us 6 more damage on a hit. 6x50% is 3 while 6x75% is 4.5. What that means is that the Brutal Strike option is now losing 1.5 damage versus the non Brutal Strike option, which wipes out the lead it had over the eschewing Brutal Strike choice. At lower ACs, of course, this isn't the case.
Having now run a campaign in tier 4 for a couple years (note to future me: don't try to pack so much stuff into a single campaign's story) I will say that even if the big bosses will have ACs in the low 20s, most monsters that the party faces are still looking at ACs in the high teens, so we can honestly expect there to be more scenarios where Brutal Strikes does wind up a damage increase.
I think that the harder but also more important question is whether it's enough of a damage increase. Given that Paladins and especially Fighters get a pretty massive damage boost at middle levels that they don't have to expend any resources to get nor take any kind of penalties for, I do sort of wonder if it would have been better to just give Barbarians the ability to add that d10 (and later 2d10) once on their turn with no strings attached. Indeed, if we compare this to a Paladin's Radiant Strikes, the Paladin gets, effectively, 2d8 extra damage for their two attacks (more if they're dual-wielding) every turn, which is more than the Barbarian gets with a d10. Not, sure, the utility from the Brutal Strike feature is cool - and is the main reason to use it, if you ask me - and might mean that tempering the damage bonus is required.
But especially as the one class that truly cannot be casting spells in combat (ok, yeah, there are some weird builds out there where you multiclass Paladin and get concentration-free spells like Divine Favor) I don't think Barbarians would become suddenly overpowered by making Brutal Strike a pure benefit rather than a kind of "benefit with a cost."
Should Barbarians do the most damage? No, actually. They're already the toughest characters to kill in D&D (believe me, the Ancestral Guardian in my campaign is basically unstoppable at this point) and I think it's reasonable to limit their damage output (this is also why I think Rogues should deal way more damage). But buffing this feature would not, I think, unbalance them.