Aryanna Hering stands outside the entrance to her newly purchased home in San Clemente, Calif. After her grandfather died, she jointly inherited the home with other relatives but she eventually got a loan and used it to fully purchase it.

Ms. Hering’s case highlights how a flavor of mortgage once panned for its role in the housing meltdown a decade ago is making a comeback. These loans, aimed at buyers with unusual circumstances such as those who can’t provide the standard proofs of income, are growing rapidly even as rising interest rates and higher home prices crimp demand for mortgages.

Lenders issued $34 billion worth of these unconventional mortgages in the first three quarters of 2018, a 24% increase from the same period a year earlier, according to Inside Mortgage Finance, an industry research group. While that makes up less than 3% of the $1.3 trillion of mortgage originations over that period, the growth is notable because it came as traditional home loans declined. Those originations fell 1.2% over the same period and were on track for a second down year in 2018.

During the financial crisis, many unconventional loans soured after borrowers misstated their incomes and lenders didn’t ask for documentation, earning them the nickname “liar loans.” Today, industry executives say the new unconventional mortgage, now referred to as “nonqualified” in industry jargon, has changed drastically from its crisis-era predecessor and is far safer.

Even so, some regulators, consumer advocates and others worry that the growth in this type of mortgage and rising competition to make such loans could lead to renewed risks for the housing market.

Ms. Hering cooks in the kitchen of her newly purchased home.

Ms. Hering cooks in the kitchen of her newly purchased home. Photo: Jessica Pons for The Wall Street Journal

“It’s a slippery slope,” said Mat Ishbia, the president and CEO of United Wholesale Mortgage, a large nonbank lender that doesn’t issue these loans.

So far, specialty mortgage companies have dominated in making such unconventional loans. But traditional lenders, which are doing less conventional business as interest rates rise, are turning to borrowers with harder-to-document creditworthiness as a new source of revenue and are helping to drive the growth. Nearly half of lenders who participated in a recent survey said they have plans to get into this business, according to Inside Mortgage Finance.

These mortgages don’t…