Theoretically it should take the same amount of time (or roughly) to grind each side, pending the bevels were already even-ish to begin with.
So, a couple of possibilities:
Do you lap the stone after grinding one side? When using the process of grinding one side at a time, you have to remember that you started with a clean stone, probably freshly lapped, and it's going to cut much faster. After 2 minutes of rough sharpening the stone will load up a bit. I find getting the swarf off of the stone and giving it a quick lap before starting the other side to greatly speed things up.
Also, do you use 2 hands to sharpen instead of 1? Is one hand putting more pressure than the other. If you are using 1 hand, do you put more pressure on one side than the other? That usually isn't going to be much of an issue after you've sharpened for a while, but more often than not when first starting out you will make asymmetric bevels which will make things take longer than normal (although at your level experience I seriously doubt that is the issue, more or less for other people to read).
About removing fatigued steel: Generally speaking, unless there is a major chip then the majority of fatigued steel is going to be on the edge-of-the-edge. If you start to sharpen and hit the fatigued edge it will simply chip out the first time you raise the burr, so that really shouldn't be an issue IMO. Some steels have more plasticity (Globals) and hold on to a burr or damaged steel a little longer, so that can sometimes create a problem as well, but yet again I highly doubt this is the problem you are having.
I'd dare to say you should just try lapping the stone after getting the first burr and then see what happens. All things equal on the stone side then you can start to explore other possibilities. We WILL get to the bottom of this, I hope.