QoS + VPN

I have a fast internet connection -- faster inbound than outbound. But, still ran into issues where a single user uploading a massive file could kill performance for everyone else.

I enabled QoS, and it solved that problem. However, it also had the side effect of causing very poor performance for remote OpenVPN users - even when the network was not under load.

I'm sure others have already ran into this. Any pointers on workable solutions?

Also, the only thing I need out of QoS is to...

QoS + VPN