When you are setting up the rule, as long as it is on a field or a control that is outside the repeating table, when you add the condition, in the first drop down, you can pick your repeating group. When you pick a repeating group, one option is "number of instances" - see attached screenshot.
