If it is too big, the risks could be;
1. Excessive flapping in the wind at high speeds - very annoying and potentially dangerous.
2. The armour moving around during a get off - but there is a lot of debate on how effective armour is anyway, and it is very difficult in my opinion to predict if the armour will be in the right spot at the exact time of impact anyway.
3. The chance of the jacket getting pulled up over your torso during a slide.
I would say #1 and #3 are the most interesting to look at and easy to test. Does it flap at speeds? When it is on, can you pull it up around you?
If it flaps a lot or can be pulled up while zipped, you should probably get a smaller size.
mitt