Nick - It'll work, but if I'm following you correctly, you have to make sure that charger doesn't have a common ground - i.e. that it's isolated. I actually bought a bunch of cheap chargers to do a hybrid of what you're suggesting and what 88fiero is suggesting. I ended up going with a boost converter setup (and balance boards in the battery packs), I think that'll ultimately be easiest to implement. Though, as you say, charging will be slow; but you'll almost never need to charge much.