i do wonder if magic doesn't have a lot to do with it. because there's a lot of stuff going on when you cast spells. and when ever i get slow downs, it usually is when i'm fighting a mage or a witch.
or a swarm of bears like last night...
i was walking through the woods minding my own business, heard a growl, looked up and see a bear on his hind legs telling me to back off so i back off and try to circumvent his territory, but he didn't like that idea so he ends up chasing me along with 10 or more of his closest bear buddies and they would give up the chase.
i ran out of stamina and had to climb a rock and didn't feel like wasting 100 iron arrows on the bears so i took off after a stamina refill and the only way i could shake them was to lead them to a fort and let them fight some bad guys.