The only time I have personally experienced this is when the page has not finished loading, and I click Add Document. It's like the page doesn't know (yet) that it's supposed to open the form in the browser.
Otherwise, just make sure that the form is browser enabled and that the form library is set to open forms in the browser.
If you can narrow down the behavior (i.e. when does it open in Filler) to a specific set of steps or a few users or a few machines, perhaps that will shed some light on the situation.