Qualtrics will tend to avoid sending a duplicate message to the same person if the individuals are in multiple segments, receiving the same message and inviting them to the same survey but on a timed stagger.
However, if the different segments are receiving different messages (A/B message testing, etc) or you need the groups to be the same size, then I would suggest that you set each successive segment definition to include a clause that the contents NOT include anyone in the prior segment(s).
Eg Segment 2 excludes Segment 1, Segment 3 excludes Segments 1 & 2, and so on.
I tend to default to adding the criteria, just to be certain of the behavior (because sometimes technololgy gets updated and specs change), but I understand that’s tiresome when your list is large and there will be many segments.