Hmmm.
If you are using a filler form (i.e., your form is not going to be used in the browser), you can restrict the file types on the file attachment control itself - that will be under properties.
Since we can't set conditional formatting on a file control - I'd actually put it in a section, and then hide it. To hide it, I'd use conditional formatting with a custom pattern.
So, for the first drop down in your conditional formatting, select the field that has the file name. For the middle drop down, select "does not match pattern". For the third, select custom pattern. The custom pattern would be:
.*(\.xls|\.xlsx|\.txt)
I've attached a simple sample form - save it locally, and right click the saved file to open in design mode. From there, you can preview.