There are two main issues - both related. The brushless controllers you're referring to typically can't handle the sort of eRPM we're looking for. There are a few controllers with ultra-high eRPM options (Kelley is one, but I'm not sure how you'd buy one); and that adds substantially to the cost. The output stage of an ESC typically is MOSFET-based. And since MOSFETs have a measurable capacitance on the the gate (actually, on all three pins), they start to incur switching losses - a discharged capacitor looks like a short to a DC voltage, and it takes time to charge. As more advanced MOSFETs hit the market (as well as newer switching semiconductors - GANFETs, for example), that may change, but for now we have to work with what's available.
And then there's the torque vs RPM issue. The LMT 30100 motor I'm using is rated for a peak of 53hp, and weighs less than 5 lbs. It does that by achieving high rpm - horsepower is a function of torque and speed. Without speed, and only torque, no work gets done. It's like pushing against a wall. Horsepower (and its direct equivalent - watts) represents power. (BTW, we've actually seen well into the 40hp range from the LMT motor, fwiw). So the LMT motor is rated at 53 hp at 50,000 rpm. That means it makes 5.6 ft-lbs of torque at that rpm. You can get away with smaller, lighter parts because torque is lower. BTW, hybrid and EV manufacturers are embracing smaller, faster motors for exactly this reason.
This is a good calculator to convert hp to torque:
https://spicerparts.com/calculators/horsepower-torque-calculator
Now let's say you want to use a lower rpm 50hp motor. Let's say a 1,200 rpm, 50 hp motor like this one:
https://www.electricmotorsforless.c...50uuFr1iRpeiPmf6bqTRhNwXTPzsyNsUaAn93EALw_wcB
It literally makes the same power the LMT motor does. Both are capable of the exact same amount of work. The difference is that motor weighs 878 lbs (!) It only draws 64 amps at 460 volts; the LMT motor draws 700 amps at 52 volts (in my actual testing) - that's 29,440 watts (only 40 hp, not the rated 50) vs 36,400 watts - so the LMT motor is actually MORE powerful, yet only weighs a small fraction as much. Granted, the big bastard could probably run 24 hours a day for a year straight and the LMT can't (unless you had proper cooling; even then, it'd be hard on bearings), but I think you get the point.
Golf cart drivelines are pretty weak by our standards. One day I'd like to put 3-4 castle 2028 motors in something like a Miata just to prove the point.