This is the formula for your max attack/defense. You can see there are (99+99)/2 value on that, so:
formula = (5 * skill + 50) * atk * stance
max damage = (formula * 99) / 10000
With balance stance (1.0):
For 90 melee skill, with stonecutter axe (50 atk), you would have 247,5 max damage
For 90 defense skill, with mastermind shield (37 def), you would have 183,15 max defense.
To determine your random attack/defense, there are 2 rands:
randresult = rand() % 100;
(formula * ((rand() % 100 + randresult) / 2)) / 10000
As you can see, your rand value will vary from 0 to your max value. Both is applied for attack/defense. If you have doubt about any damage and you think we use wrong formulas, you can just get any monster attack/skill value from .mon files and apply on this formula to get their max damage, then diff from your character max defense. If you get hitted (without armors) by a value higher than this diff, so there's something wrong, if not, it's just your superstition.