Attack Speed Debuff when holding weapon

It's your Attack Speed for that weapon. Check the topmost line after the weapon's name.

Your weapon's Attack Speed determines your attacks per minute (or the time between each attack), the Mining Fatigue effect is cosmetic and is intentionally not going away so long as you hold the weapon. Worth noting that your weapon's Attack Speed is the base value and can be affected by any Attack Speed modifiers found in armor, accessories or rarely the weapons themselves.

e.g. Rhythm of the Seasons has a Very Fast Attack Speed, and will perform 180 melee attacks per minute, assuming you are executing each attack perfectly.