You just have to get your logic correct. It can be done easily. When creating data validation logic statements, EXACT syntax is required. Things like "yes" do not equal "YES" or "Yes" and sometimes aren't the same as "TRUE" or "1" so you have to be very careful. Do one step at a time and verify that every individual piece works before combining conditions.
You basically need to say:
If Dropdown equals <value>
and Option not equals TRUE (make sure it's a boolean field or get the text exact if not)
Anyway, I just rigged up a sample file that has all 3 fields with the logic as you described it, I think. Take a look and see if that helps.