We prove that any continuous map of an N-dimensional simplex Delta_N with colored vertices to a d-dimensional manifold M must map r points from disjoint rainbow faces of Delta_N to the same point in M: For this we have to assume that N \geq (r-1)(d+1), no r vertices of Delta_N get the same color, and our proof needs that r is a prime. A face of Delta_N is a rainbow face if all vertices have different colors. This result is an extension of our recent "new colored Tverberg theorem", the special case of M=R^d. It is also a generalization of Volovikov's 1996 topological Tverberg theorem for maps to manifolds, which arises when all color classes have size 1 (i.e., without color constraints); for this special case Volovikov's proof, as well as ours, work when r is a prime power.