You couldn’t refuse a duel in these ancient times or you would be considered a nith. And a nith is the worst thing you could be in this era. You’d be labeled a coward, an oathbreaker. (You could also possibly be magic and a homosexual, Boogedy boogedy boo!) If found to be a nithing, you’d be immediately exiled, “no one was allowed to protect, house or feed the outlaw. He had to seek shelter alone in the woods like a wolf.” Your wife would be considered a widower, pretty heavy stuff, not only are you dead to me, your dead to your whole town and everyone you know.
The shame of being a nith was so ingrained in the culture of that time, scoundrels used the threat of Holmgang, and the fear of being branded a Nith to their unscrupulous advantage. Stronger than your neighbor? What’s to stop you from showing up at his doorstep and challenging him to a duel? The choices- accept, die and loose all your stuff or runaway, be labeled a nithing and loose all your stuff. Piracy through shaming was rampant.