Hi thinking_spot and welcome to the forums!
This does seem like a rather complicated situation. I think it would be easier here to do a string compare. Try this:
Take the tutorial from this post:
http://blogs.msdn.com/infopath/archive/2006/04/05/569338.aspx except you want it to string together all of the values of the B repeating table. Then concatenate this with A and compare to siblings. Your condition will look something like this:
xdMath:Eval(xdMath:Eval(my:group3/my:group4, 'concat(my:field3, ";")'), "..") != xdMath:Eval(xdMath:Eval(../descendant-or-self::node()/my:group2[my:field1 = current()/my:field1]/my:group3/my:group4, 'concat(my:field3, ";")'), "..")
.........
Well, I got really close to finishing this, but I've spent too much time as it is trying to figure it out. Hopefully this will springboard you into the answer! Please reply back when you figure it out so I can see the whole thing!
Matt Faus / Microsoft InfoPath MVPQdabra® Software / Streamline data gathering to turn process into knowledge