Well my internet package comes with 16Mbps download/2Mbps upload. My wife is connected to the wifi with her cell phone and iPad. Plus she was on my Xbox watching Netflix. That's why it was 416Kbps.
Its a bit more complicated than that, you may have 0.4Mbps upload at that time, but that is scattered across 1 second. (1000 ms), between 1.6Mbps of other connections at varying degree. Take some overhead into account and it won't run smoothly when 80% of the connection is blocked by other packets.
Netflix and youtube can buffer at intervals and throttle your connection. When you visit a webpage, that image will be downloaded as fast as the server can send, and you can download. These things may lead to spikes and queued connections on your game server.
Upping your upload speed will help against this.
In regular consumer scenario, download speed is mostly being tapped. So a 120Mbps line will heavily prevent these things. Most servers cap user connections to 10 - 100Mbps, so even in big buffer scenarios you will have 20Mbps remaining.
A lower upload speed can also be unstable in general, 0.4Mbps looks like a standardised minimum ADSL configuration. An internet connection usually varies by a few Mbps, which can be critical when you have no upload speed.
By increasing upload speed to 10Mbps, you can depend on always having 5Mbps+ to use with greater reliability.
Although, personally I would rather invest in a proper host than an internet upgrade.
...That being said, my current home network is at 300/300Mbps. And I am not hosting anything at home.