TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Zero Tolerance for Bias

190 点作者 Harmohit12 个月前

8 条评论

IAmLiterallyAB11 个月前
Interesting, but there&#x27;s one part I&#x27;m not sure I agree with.<p>&gt; Pseudo-random number generators are useful for many purposes, but unbiased shuffling isn&#x27;t one of them.<p>A properly seeded CSPRNG is perfectly fine at this. And if it&#x27;s not, then all of our cryptography is pretty much screwed. This is why in modern kernels, &#x2F;dev&#x2F;random and &#x2F;dev&#x2F;urandom are the same (minus differences in behavior when the initialization isn&#x27;t complete). As D.J. Bernstein put it, it&#x27;s superstition to not trust CSPRNGs. <a href="https:&#x2F;&#x2F;www.mail-archive.com&#x2F;cryptography@randombit.net&#x2F;msg04763.html" rel="nofollow">https:&#x2F;&#x2F;www.mail-archive.com&#x2F;cryptography@randombit.net&#x2F;msg0...</a> And if it&#x27;s good enough for crypto, it&#x27;s good enough for card shuffling.<p>FYI I am not a cryptographer
评论 #40620950 未加载
评论 #40620953 未加载
skybrian11 个月前
I recently stumbled across some other ways that random number generation can go wrong.<p>Suppose you want reproducible results from a seed, along with parallelism. Algorithmic random number generators are usually mutable and generate a <i>sequence</i> of results, limiting parallelism. Rather than a sequence, you want something tree-shaped where you can create an independent random stream for each child task. In higher-level API&#x27;s, a <i>jump</i> or <i>split</i> operator can be useful.<p>Counter-based random number generators [1] seem pretty useful in that context. An immutable random number generator works like a hash algorithm that maps each input to an output that&#x27;s difficult to predict. The problem with this is being careful to avoid using the same input twice. You can think of it as allocating random numbers from a very large address space in a reproducible way. How do you partition the address space, predictably, so that every address is used at most once, and nobody runs out?<p>Giving each child a unique ID and generating a stream from that is one way. If the tree is deeper, you&#x27;ll want a unique seed for each <i>path</i>.<p>When a mutable random number generator is copied to a child task (or maybe just to an iterator), the same random numbers might be generated in two places. Avoiding this is the sort of thing that Rust&#x27;s borrow checker can prevent - borrowing is okay, but you want to prevent multiple concurrent ownership.<p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Counter-based_random_number_generator" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Counter-based_random_number_ge...</a>
评论 #40620876 未加载
评论 #40621043 未加载
评论 #40621766 未加载
orlp11 个月前
Here is a trivial shuffle algorithm that is completely unbiased and only requires an unbiased coin (or random number generator giving bits):<p>1. Randomly assign each element to list A or list B. 2. Recursively shuffle lists A and B. 3. Concatenate lists A and B.<p>To prove it&#x27;s correct, note that assigning a random real number to each element and sorting based on that number is an unbiased shuffle. Then we note the above does in fact do that by considering the fractional base-2 expansion of the random numbers, and noting the above is in fact a base-2 radix sort of these numbers. We can sort these random real numbers even though they have an infinite amount of random bits, as we can stop expanding the digits when the prefix of digits is unique (which corresponds to the event that a list is down to a single element).<p>I call the above algorithm RadixShuffle. You can do it in base-2, but also in other bases. For base-2 you can make it in-place similar to how the partition for Quicksort is implemented in-place, for other bases you either have to do it out-of-place or in two passes (the first pass only counting how many elements go in each bucket to compute offsets).<p>The above can be combined with a fallback algorithm for small N such as Fisher-Yates. I believe even though the above is N log N it can be faster than Fisher-Yates for larger N because it is exceptionally cache-efficient as well as RNG-efficient whereas Fisher-Yates requires a call to the RNG and invokes an expected cache miss for each element.<p>---<p>Another fun fact: you can turn any biased memoryless coin into an unbiased one with a simple trick. Throw the coin twice, if it gives HH or TT you throw away the toss, if it&#x27;s HT or TH you use the first toss as your unbiased coin.<p>This works because if p is the probability that heads comes up we have:<p><pre><code> HH: p^2 HT: p(1-p) TH: (1-p)p TT: (1-p)^2 </code></pre> Naturally, p(1-p) and (1-p)p are equiprobable, thus if we reject the other outcomes we have distilled an unbiased coin out of our biased coin.
评论 #40621061 未加载
评论 #40621562 未加载
评论 #40622156 未加载
评论 #40620605 未加载
评论 #40621754 未加载
评论 #40625326 未加载
评论 #40620124 未加载
评论 #40621057 未加载
评论 #40621321 未加载
tomcam11 个月前
I&#x27;m sure I&#x27;ll get roasted for this, but getting the right answer will scratch a years-old itch. Why aren&#x27;t most random number generators seeded using the system clock in addition to the existing algos?
评论 #40621182 未加载
评论 #40621421 未加载
评论 #40623771 未加载
im3w1l11 个月前
Fisher Yates is simple, fast and correct. The only area of improvement I can think of is reducing cache misses.
评论 #40620306 未加载
评论 #40620908 未加载
zug_zug11 个月前
Kinda interesting.<p>There are N! permutations in a shuffle (no duplicates) and there&#x27;s an algorithm where you pick one random number between 1-&gt;N! and then bring up that permutation (though I don&#x27;t know how to do it better than N log N with an order-statistic tree). I like this because it requires exactly one random number.<p>A trivial solution in functional programming (for those of us who find this swap stuff really unreadable and error-prone) would be something like:<p>[1,2,3,4,5,6].map(x =&gt; {return {value: x, order: Math.random()}}).sort((a,b) =&gt; (a.order - b.order)).map(x =&gt; x.value)<p>Of course this is N-Log-N, but personally I think it&#x27;s easy to forget how small logN grows. Like log10(number of atoms in universe) = 82, so if your dataset is smaller than the number of atoms in the universe you could think of it as less than the constant 82.
评论 #40619992 未加载
评论 #40621664 未加载
tbrownaw11 个月前
Trusting a broken library seems like a different <i>kind</i> of error than implementing an algorithm that isn&#x27;t quite the algorithm you meant to implement.
westurner11 个月前
python has random.shuffle() and random.sample() with an MT Mersenne Twister PRNG for random. <a href="https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;library&#x2F;random.html#random.shuffle" rel="nofollow">https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;library&#x2F;random.html#random.shuffle</a> Modules&#x2F;_randommodule.c: <a href="https:&#x2F;&#x2F;github.com&#x2F;python&#x2F;cpython&#x2F;blob&#x2F;main&#x2F;Modules&#x2F;_randommodule.c">https:&#x2F;&#x2F;github.com&#x2F;python&#x2F;cpython&#x2F;blob&#x2F;main&#x2F;Modules&#x2F;_randomm...</a> , Library&#x2F;random.py: <a href="https:&#x2F;&#x2F;github.com&#x2F;python&#x2F;cpython&#x2F;blob&#x2F;main&#x2F;Lib&#x2F;random.py#L354">https:&#x2F;&#x2F;github.com&#x2F;python&#x2F;cpython&#x2F;blob&#x2F;main&#x2F;Lib&#x2F;random.py#L3...</a><p>From &quot;Uniting the Linux random-number devices&quot; (2022) <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=30377944">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=30377944</a> :<p>&gt; <i>&gt; In 2020, the Linux kernel version 5.6 &#x2F;dev&#x2F;random only blocks when the CPRNG hasn&#x27;t initialized. Once initialized, &#x2F;dev&#x2F;random and &#x2F;dev&#x2F;urandom behave the same. [17]</i><p>From <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=37712506">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=37712506</a> :<p>&gt; &quot;lock-free concurrency&quot; [...] <i>&quot;Ask HN: Why don&#x27;t PCs have better entropy sources?&quot; [for generating txids&#x2F;uuids] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=30877296">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=30877296</a></i><p>&gt; <i>&quot;100-Gbit&#x2F;s Integrated Quantum Random Number Generator Based on Vacuum Fluctuations&quot; <a href="https:&#x2F;&#x2F;link.aps.org&#x2F;doi&#x2F;10.1103&#x2F;PRXQuantum.4.010330" rel="nofollow">https:&#x2F;&#x2F;link.aps.org&#x2F;doi&#x2F;10.1103&#x2F;PRXQuantum.4.010330</a></i><p>google&#x2F;paranoid_crypto.lib.randomness_tests: <a href="https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;paranoid_crypto&#x2F;tree&#x2F;main&#x2F;paranoid_crypto&#x2F;lib&#x2F;randomness_tests">https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;paranoid_crypto&#x2F;tree&#x2F;main&#x2F;paranoid...</a>