+1 to what Jimmy wrote except that it's not in all form templates. It appears that a list form does not have this option in the properties of a file attachment control. A regular form library template has it, but it is sometimes disabled. It looks like it's Disabled for browser forms but enabled for Filler forms. To be sure, I limited my Filler form template to "docx; xlsx" and it worked well in Filler mode. Interestingly, it actually sets the filter on the popup dialog where you browse for documents. Whatever items you specify as being allowed are the only items that the browse window will let you choose, and it doesn't let you change the searchable file types. This might be why it doesn't work in the browser, because this functionality is invoked on the client machine itself, not in the form as a data validation. I then switched my template back to a browser form, and I get an error on the control saying, "Setting allowable file types is not supported in Web browser forms."
So, I then thought about using data validation to detect the file type of uploaded docs, but the value stored with the attachment is encoded with tons of characters, so I'm not sure yet if that can be done at the time of upload.
In conclusion, if you're just using Filler (rich client) forms, then open the properties of your file attachment control, check the box at the bottom, and insert this text, JPG; PNG; TIFF. What I found was that this feature is case INSENSITIVE, and the spaces after the semicolon delimiters don't matter. I tried uppercase, lowercase, space, and no space - all worked, but the semicolon was the key.