I totally agree, I wouldn't worry too much about the latency @mah, in most circumstances it's not an issue, especially if the ER-301 is your only source of drum sounds as @iPassenger says.
I use mine in 48KHz and don't find it a problem, I could make it happen but I'd have to find sounds that were the right type and try and match them to something with a faster response rate outside the ER-301 to create the circumstances in which it would be noticeable.
The style of music you're making will also have a large impact.
Any time I find anything not sitting right I adjust elsewhere. For example the Stillson Hammer is perfect for this kind of timing play, if you want a shuffled hi hat that sits late for a bit of a groove you could just apply a little less delay. I never work anything like this out, just push and pull everything until it sounds right.
Hopefully it will improve as time goes on anyway