Many of those surveyed said concerns about Immigration and Customs Enforcement officers at polling places contributed to their declining trust.