Hello tafavorite,
one approach without modifying the schema is to use conditional formatting. In this case you define a conditional formatting on the repeating table itself. As the condition you can use the following expression:
count(/my:myFields/my:YourGroup/my:Your repeatingGroup) > YourMaximumNumberofRows
and then select the following option:
Don't allow users to insert or delete this control
Hope that helps.
Regards,