Oh Dear, I Think I Am In Trouble...

I completed one round of the 30-stitch motif on my Northwest Sunset Fair Isle Vest and I am in trouble. I think my floats are too tight and my gauge is off. The gauge in color pattern is supposed to be 31 stitches in 4". My gauge is something like 35 stitches in 4". When I swatched I was okay but now that I am knittng the actual piece my gauge is not the same.