You are absolutely right. The "rules" only applied to the gentry. If you were of low birth, you were screwed - sometimes literally. Most knights would think nothing of seducing a female serf, kicking a beggar (especially one from another land), or publicly humiliating a servant.

Nice guys.