Most Rails contributors are not big users of shared hosting and they tend to work on problems or enhancements that'll benefit their own usage of the framework. You don't have to have a degree in formal logic to deduce that work to improve life on shared hosting is not exactly a top priority for these people, myself included.
That's not a value judgement. It's not saying that shared hosting is bad or evil. It's simply saying that the Rails contributors generally don't use it. By extension, it's not something that we are personally invested in solving as a traditional "scratch your own itch" type of development.
Improve what is for profit and fun
I'd love for Rails to be easy as pie to run in a shared hosting environment, though. I'd love for Rails to be easy as pie to run in any environment. In that "more people could have fun learning Rails and deploying their first hobby application" kind of way. But I don't need it in the sense that I'm going to put in the work, personally, to make it happen.
Others might, though. The Dreamhost guys in particular sounds like they're experiencing a lot of hurt running Rails in their shared hosting environments. That should be a great motivator to jump in and help improve things. The work I do every day to improve Rails is usually about removing hurt. Heck, it's currently in the slogan on the Rails site: "Web development that doesn't hurt".
Second, it sounds like they have a substantial economic interest in making the shared hosting scenario for Rails easier. I read that a fair number of their customers are going elsewhere because they can't get Rails to run well at Dreamhost. Before they leave, though, they probably tax the support system quite heavily as well. So there's direct costs, lost revenues, and probably also a great upside waiting if Rails ran great on their system.
That's both a personal motive for having a less stressful day and a profit motive for making your business more money. Sounds like a match made in heaven for someone like Dreamhost to get involved and help do the work to make Rails a great shared host experience. They might not have the man-power in-house today to make that happen, but I'm sure they could easily hire their way out of that. If the plan they want to pursue is a better mod_ruby, I'd start looking at that project for people who've contributed and ask if they'd like to earn a living improving the state of affairs.
We'll work with you if you're willing to work with us
Again, I'd love to see someone tackle this challenge and would be more than happy to work with a group pursuing this to get their results into Rails or working with Rails the best way we can. Consider that an open, standing invitation.
In exchange, I'll ask a few, small favors. Don't treat the current Rails community as your unpaid vendor. Wipe the wah-wah tears off your chin and retract the threats of imminent calamity if we don't drop everything we're doing to pursue your needs. Stop assuming that it's either a "complete lack of understanding of how web hosting works, or an utter disregard for the real world" that we're not working on issues that would benefit your business. Think of it more as we're all just working on the issues that matters most to our business or interests.
The alternatives
Now if you're a user of shared hosting and you're not satisfied with the results you're getting — and you're not getting good vibes that things will be better — there are alternatives. Lots of them in fact. And it doesn't have to cost an arm and a leg. Self-service VPS outfits like Slice Host has plans starting at $20/month that runs Rails great (I use them to run this site). RailsMachine has a Rails-specific setup for $75/month. And for the more high-end stuff, you can get great setups from Joyent, Engine Yard, and tons of others.
So as a programmer looking to deploy Rails, you have tons of options in all price ranges. If you're a shared host looking to capitalize on a framework that's driving a lot of demand, it would seem that your best option is to actually get involved and help the community help you.