Max is used on a repeating node to return the largest value from the list. Since these are two fields you need to think simpler.
In your case you would have two conditional rules a>b and b>a and then have your action, maybe set max with value of a if it is greater, etc.