Ability to Unfollow Tasks assigned to you that you have commented on

There is a pretty useful dashboard widget for seeing "Tasks I Follow." I understand that it shows tasks that you follow that are not assigned to you, except in the case where you comment on those tasks. I'm running into an issue now where I am assigned a task and I commented on it so it appears in the "Tasks I Follow," and I can't "unfollow" it because it is assigned to me. Is there a way to get it out of the "Tasks I Follow" widget? It is kind of cluttering up the widget, and I no longer need to have it in there.