I'm not sure it would be any better with a rich client form. The root of the problem is the sheer amount of data and the speed of those systems when communicating (network speed, hops, authentication, bandwidth, RAM, processor, etc).
There may be some other way to optimize it for InfoPath, but I don't know of one personally.