Over the past 100 years, San Francisco added numerous restrictions on housing construction. These restrictions are so severe that many existing homes would be illegal to build today. This map shows how many homes would be illegal on each parcel of land:
These buildings aren't illegal to build because of safety reasons or any construction deficiencies - they are only illegal due to regulations called zoning laws that San Francisco passed to restrict the density of housing in certain areas, originally to keep poor minorities out of white neighborhoods.
Starting in 1921, San Francisco has been steadily passing zoning laws to restrict housing construction in wealthy neighborhoods. This accelerated after 1968 when the Fair Housing Act banned explicit race discrimination. Because minorities were poorer than whites and dense housing such as apartments is usually cheaper than single-family homes, the goal was to ban the construction of housing that poor minorities could afford.
These days, many people who don't have explicitly racist motivations support zoning laws because they like the status quo, even though the status quo bans the construction of affordable housing in most of the city.
If your goal is to make housing as expensive as possible, the best way to do it is to require lots of land per home, because land in cities is very expensive. San Francisco does this in a multitude of ways, but I decided to focus on 3 of them:
Whenever San Francisco creates a new restriction, homes that already exist are grandfathered in rather than demolished. That's where all these "homes illegal to build today" come from.
Sorry to break it to you, but politicians lie. Think about it: they say they want more affordable housing all the time, yet very little affordable housing gets built, and San Francisco remains the most expensive city in the world. Meanwhile, you can rent a nice apartment in Houston for $600/month. San Francisco politicians blame all sorts of scapegoats for this: techies, greedy developers, speculators, capitalism, etc. but Houston is also very capitalist, its developers and speculators are just as greedy, and it even has a booming industry that pays engineers high salaries - the main difference is that Houston doesn't have zoning laws.
These laws were originally created for racist reasons - that is a well-documented fact, despite what some San Francisco politicians claim. These days, supporters of zoning probably don't have explicitly racist motivations. Instead, they are concerned about traffic, parking, shadows, "neighborhood character", or just don't like change and aren't personally affected by high rent.
Still, even if the intent of zoning laws isn't racist, their effect has racist impacts. San Francisco's Black population declined from 13% of the city in the 1970s to 5.5% today. In contrast, zoning-free Houston added 100,000 Black residents from 2010 to 2015, and Black people are now 23% of Houston's population.
As far as I know, there is only one organization in San Francisco working to fix this: YIMBY Action. At the California state level, California YIMBY is working on fixing zoning laws statewide. Joining these organizations will teach you how to push your local politicians to make reforms to zoning laws.
Some places such as Berkeley, Minneapolis, and the entire state of Oregon have already taken steps to reform these racist laws, but we can't declare victory yet. Most American cities still have zoning laws that exclude poor people; we have a lot of work left to do.
Well-connected developers can apply for a conditional use permit to get a per-project exemption from zoning laws. The city does not need consistent reasons for granting or rejecting these permits, so this is an easy way for politicians to grant special favors to friendly developers.