Banks have been walking away from low-income homebuyers seeking loans, and that has affordable housing advocates worried.

Newly-released federal data on mortgage lending from the Consumer Financial Protection Bureau shows people with low- and moderate-incomes made up only 26.3% of borrowers in 2017, down from 36.6% in 2009.

In part, that’s due to federal rules that sought to crack down on the subprime lending tactics that helped bring on the financial crisis. Also, skyrocketing housing costs have locked many people of modest means out of the market.

But the data reveals another profound shift. Big banks are moving away from mortgage lending entirely, while independent mortgage companies — or “non-banks” — pick up the slack.

“Non-bank” is a catchall term for financial institutions that don’t take deposits. Non-bank mortgage lenders just do mortgage lending, for example. So in a time of low interest rates and higher regulatory costs, traditional banks have the option of moving into more profitable ventures, like credit cards.

“Profit margins on lending have come down quite a bit,” says Mike Fratantoni, chief economist at the Mortgage Bankers Association, which represents both banks and non-banks. “So a number of banks have de-emphasized their mortgage lending, because there are other business lines they can focus on.”

Non-banks, meanwhile, have doubled down on volume — particularly through refinances — and now originate 56%…