Is landEstimate the field? Or the repeating group?
I'd try something like:
count(../preceding-sibling::my:RepeatingGroup) + 1
However, if your formula syntax is correct and it is just a data type issue (which would sort of surprise me - I'd expect a count + 1 to return a number) you can always add the number function:
number(count(preceding-sibling::landEstimate) + 1)