When working out, you should not stress about fixed rest periods between each set.

You should rest whenever you need. Typically, this will be less in the early sets because your body is fresh.