Affordable housing experts and advocates say it will harm Black communities and vulnerable minorities the most.