http://www.codexon.com/posts/hash-functions-the-modulo-prime-myth-2

Cryptographically secure hash functions generally produce an even distribution or they’re broken. They are also slow, and nobody uses them for hashing in a small hash table with a limited number of buckets, they use them for hashing in essentially infinite sized hash tables distributed all over the net.

I don’t know what the current state of the art is in hash functions typically used for in-memory hash tables, but I bet that while they might be good, there is still unevenness in their distribution. In particular, for one common case, the output of the hash function is likely to almost always be divisible by 4 (or even larger power of 2) since the hash function merely returns the address of the object being hashed.

Your model of /dev/urandom as representing the output of a typical hash function is fatally flawed.

Secondly, of course a bigger table size is going to have more of an effect. In order to have a fair test, I suggest choosing 510481 and 510510 as the table sizes. And use the output of a real hash function. And generate your inputs by taking combinations of two words from an english dictionary. Since you’re using Python, the string hash function would be adequate. 510510 is very composite, and 510481 is a very slightly smaller prime number. For a similar smaller table, choose 30030 (composite) and 30029 (prime).

Personally, I would choose the prime that’s a little larger than the table size you would choose if you chose a composite.

There was a discussion on the speed of the modulus operator on Reddit, and the consensus seemed to be that it was negligible.

Since “mod prime” is a hash function in itself (naive, but still a hash function) my assumption would be that this final step is just to reduce the weaknesses of an algorithm. For example: Adler-32 is just a checksum algorithm, not a cryptographic hash function. It was created for speed and may not produce such perfect hashes in itself.

I must say that I’ve only seen the “mod prime” in quick algorithms suited for checksums and hash tables. I think it’s just a trade-off between speed and hash quality.

The argument that simple implementations win because C was successful is a non sequitur. I can argue just as easily that better performing implementations win because C was successful (it held and in many cases still holds the performance crown). I’m aware of the “less is more” philsophy, but you cannot maintain that that extends all the way down to a single decision in the implementation which does not affect the interface or use in the slightest (which is what that principle is all about). Also, I’d challenge you to take a look at the implementations of these “simple” C functions nowadays — they get quite complicated, to make maximum use of modern hardware.

If we extend the “simpler implementation is better” principle to its logical conclusion, you should not be using a hashtable someone else wrote in the first place, nor should you be writing a general one yourself, since that will always involve two hash functions while you could do with one optimized one. And we’ve already established that in that scenario, the prime bucket approach is irrelevant.

“especially in the embedded world”
The elevator controller argument? Now you’re just moving the goalposts. When you move to embedded, you’re working with a different set of constraints and a whole lot of things change, certainly not in the least the fact that you would not use a general hashtable implementation. You’d want (or be forced to use) custom memory allocation and your application would probably be very limited in scope, so you can (and should) make the hashing much more specific.

If you’re on an embedded system, then yes, again, the whole prime modulo situation becomes irrelevant, and I doubt anybody would argue otherwise. But having said that:

“Embedding a big fat prime table (the real cost of getPrime plus a binary search)”
You have obviously not even looked at implementations that take this approach. I have (one C#, one Java, one C++, quite clearly not based on each other). None of them used tables that could even remotely be called big, and none of them used binary search on these small tables. You’re attacking a naive strawman.

All I’ve been trying to do is explain why people do this in practice — it’s *not* based on superstition or premature optimization, at least not *this* particular use of a prime number of buckets. It’s very much worth mentioning that this is by no means the only way to guard against poor primary hash functions. There are others — http://weblogs.java.net/blog/tchangu/archive/2005/06/hashmap_impleme.html discusses one, and it even references Knuth about power-of-two bucket sizes!

The central point is this: hashtable implementers guard against poor hash functions because this pays off. Using a prime number of buckets is one way of doing this, and it does make sense. Using a prime number of buckets does not *always* make sense, and it’s good to point this out too (there’s plenty of superstition on prime numbers), but it’s not true that it *never* makes sense.

That’s all, folks! I don’t think I can say any more on the subject.

in fact, contrary to your premise, the vast majority of hash functions do any mod p whatsoever, they just do lots of arithmetic and bit-mixing on a 32-bit value and return that directly. at the bottom of the link you provided: http://burtleburtle.net/bob/hash/doobs.html lists a number of hash functions. not a single one of those uses mod p. Adler-32 uses mod p for a very particular reason – and that reason has nothing to do with hash functions in general nor with hash tables. mod p just happens to be a useful operation within Adler-32.

I’d have to say the simpler one wins in real life. The C standard library went with the idea that simpler implementations were better than more complicated ones. Now its one of the longest lasting popular language in existence.

Embedding a big fat prime table (the real cost of getPrime plus a binary search) does not seem like a good idea especially in the embedded world.

But Knuth used 2 modulo prime hash table examples, and Adler-32 uses modulo prime even though its simply a hash function. I am willing to bet many other people use mod prime on every hash.

Many other modern algorithms of hash tables and hash functions don’t need mod prime and achieve even better mixing. (see Bob Jenkin’s website for statistics)

I will have to disagree with you that hash tables have no requirement for collisions. Collisions are the bane of hash tables. Every time you need to repeat a lookup, your once O(1) cost becomes O(n).

hashtables receive hash values. those hash values typically aren’t computed using strong hash functions of the kind you’re referring to, for various reasons. the hashtable uses the mod p as a slightly safer way to distribute the received hash value among its buckets. it is slightly safer in the event that the received hash values aren’t well-distributed. since hash values typically aren’t computed using strong hash functions, this is an important thing to do.

there are various reasons why hash values for a hashtable aren’t computed using strong hash functions. but mainly it’s just not seen as necessary in typical programming scenarios – the consequence of having collisions are not that bad. and the performance penalty for computing a strong hash function is fairly high.

strong hash functions are really only necessary for cryptography, where avoiding collisions is necessary for security. hashtables have no such requirement.