Perhaps.
But the setting isn't a battlefield, it is a duel.
While the samurai, like a knight, lived for battle, the duelists of the early-modern period engaged in 100s of duels in their lives in which there was no second place.
A fencing master wasn't necessarily a fop. Many were also officers or NCOs who fought in many battles. Cavalrymen of the age used the saber as their primary weapon.
Napoleon's Marshal Oudinot was wounded 36 times in battle and was reputed as the French Army's finest swordsman in an age of great swordsman. It is said he killed a 100 men in duels that often lasted less than a minute.
A warrior is a warrior. I think it would be very unfortunate for a samurai to underestimate their opponent or his weapon.