Andrew's Blog2024-01-01T16:14:18+00:00https://andrew222651.comAndrew MacFieJanuary 2024 updates2024-01-01T00:00:00+00:00https://andrew222651.com/2024/01/01/jan-2024-updates<p>Have paper books with no digital version? <a href="http://www.bookscanning.ca/">Get them scanned into a PDF</a></p>
<p>Read the web in simplified English spelling:
<a href="https://chrome.google.com/webstore/detail/cutspel/hojgjeiaognigdpphinpoobgpmgpkone">this Chrome extension</a>
translates pages to <a href="https://en.wikipedia.org/wiki/Cut_Spelling">“Cut Spelling”</a></p>
<p><a href="https://en.wikipedia.org/wiki/Standard_Canadian_English">The ultimate guide to the Canadian accent</a></p>
<p><a href="https://www.finlywealth.com/calculator">Find the optimal credit card based on your spending</a></p>
<p>Trivial Paleo recipes for <a href="https://docs.google.com/document/d/1z36FpDr08Zk02yZxw_P5xmo-WwbVSkBptERWEYZ0Yu8/edit?usp=sharing">meal replacement powder</a>
and <a href="https://docs.google.com/document/d/13X5egTq2rTxWijhgaam8AwFhpbPYxucJ79Ac8n9AS6I/edit?usp=sharing">dry cereal</a></p>
<p><a href="https://github.com/tiangolo/fastapi/discussions/10743">Alternative to <code class="language-plaintext highlighter-rouge">BackgroundTasks</code> in FastAPI</a></p>
March 2023 updates2023-03-05T00:00:00+00:00https://andrew222651.com/2023/03/05/mar-2023-updates<p><a href="https://github.com/amacfie/public_notes/wiki/Tricky-grammar">New note: Tricky grammar</a></p>
<p><a href="https://github.com/amacfie/public_notes/wiki/Interviewing-programmers">New note: Interviewing programmers</a></p>
<p><a href="https://github.com/amacfie/public_notes/wiki/Problem-solving-strategy-in-math-and-beyond">New note: Problem solving strategy in math and beyond</a></p>
<p><a href="https://amacfie.github.io/solarize">Convert images to the solarized color scheme</a></p>
<p><a href="https://travegeo.com/UNESCO_World_Heritage_Sites">UNESCO World Heritage Site map</a></p>
<p><a href="https://web.archive.org/web/20200611051214/https://inside.trinity.edu/student-life/dean-students/alcohol/online-resources/optimal-buzz">Optimal alcohol</a></p>
<p><a href="https://www.walkscore.com/">Search for apartments within walking distance of a location</a></p>
January 2023 updates2023-01-17T00:00:00+00:00https://andrew222651.com/2023/01/17/jan-2023-updates<p><a href="https://www.citybeetfarm.com/pages/donate-your-yard">A great idea: let people use your yard as a vegetable garden and receive
a percentage of the produce</a></p>
<p><a href="https://github.com/pschanely/CrossHair/discussions/193#discussioncomment-4690860">Lifting CrossHair (automatic bug finder) from Python to SQL via SQLGlot</a></p>
<p><a href="https://photos.app.goo.gl/ix4eceD3UEanQgGx5">Photo gallery: Toronto</a></p>
<p><a href="https://github.com/amacfie/public_notes/wiki/Survey-on-computer-assisted-natural-language-problem-solving-in-the-formal-sciences">New note: Survey on computer assisted natural language problem solving in the
formal sciences</a></p>
<p><a href="https://github.com/amacfie/public_notes/wiki/Global-keyboard-shortcuts-for-ergonomic-editing,-screenshots,-and-media-controls">New note: Global keyboard shortcuts for editing like in emacs, screenshots,
and media controls</a></p>
November 2022 updates2022-11-18T00:00:00+00:00https://andrew222651.com/2022/11/18/nov-2022-updates<p><a href="https://min.news/en/education/8615de9eb4f267be7d3285b35efa4e3f.html?__cf_chl_jschl_tk__=pmd_XvHF0JrEoMkmr5kk_c72vZkpfgzxc4JIIMw0n83GYj8-1631300917-0-gqNtZGzNAlCjcnBszQil">“10 years ago, the Peking University genius Liu Zhiyu, who gave up MIT and
chose to be a monk, how is it now?”</a></p>
<p><a href="https://github.com/amacfie/public_notes/wiki/Showing-the-next-rectangle-in-a-pile">Algorithm for showing overlapping hints in keyboard-driven browser
extensions</a></p>
<p><a href="https://github.com/amacfie/fastapi_returns_inferring_router">FastAPI add-on that integrates the dry-python “returns”
library</a></p>
<p><a href="https://github.com/haslersn/any-nix-shell/issues/18#issuecomment-1316209953">Xonsh users can use any-nix-shell to stay in xonsh when running <code class="language-plaintext highlighter-rouge">nix shell</code></a></p>
<p><a href="https://asciilib.iansinnott.com/">Kaomoji search engine</a></p>
"A Man's A Man for A' That" in Modern English2022-01-03T00:00:00+00:00https://andrew222651.com/2022/01/03/a-mans-a-man<p>On the occasion of <a href="https://www.math.ens.psl.eu/~hoda/">Nima Hoda</a>’s wedding.</p>
<p><a href="https://en.wikipedia.org/wiki/A_Man%27s_A_Man_for_A%27_That">Original</a>:</p>
<blockquote>
<p>Is there for honest Poverty <br />
That hings his head, an’ a’ that; <br />
The coward slave - we pass him by, <br />
We dare be poor for a’ that! <br />
For a’ that, an’ a’ that. <br />
Our toils obscure an’ a’ that, <br />
The rank is but the guinea’s stamp, <br />
The Man’s the gowd for a’ that.</p>
<p>What though on hamely fare we dine, <br />
Wear hodden grey, an’ a that; <br />
Gie fools their silks, and knaves their wine; <br />
A Man’s a Man for a’ that: <br />
For a’ that, and a’ that, <br />
Their tinsel show, an’ a’ that; <br />
The honest man, tho’ e’er sae poor, <br />
Is king o’ men for a’ that.</p>
<p>Ye see yon birkie, ca’d a lord, <br />
Wha struts, an’ stares, an’ a’ that; <br />
Tho’ hundreds worship at his word, <br />
He’s but a coof for a’ that: <br />
For a’ that, an’ a’ that, <br />
His ribband, star, an’ a’ that: <br />
The man o’ independent mind <br />
He looks an’ laughs at a’ that.</p>
<p>A prince can mak a belted knight, <br />
A marquis, duke, an’ a’ that; <br />
But an honest man’s abon his might, <br />
Gude faith, he maunna fa’ that! <br />
For a’ that, an’ a’ that, <br />
Their dignities an’ a’ that; <br />
The pith o’ sense, an’ pride o’ worth, <br />
Are higher rank than a’ that.</p>
<p>Then let us pray that come it may, <br />
(As come it will for a’ that,) <br />
That Sense and Worth, o’er a’ the earth, <br />
Shall bear the gree, an’ a’ that. <br />
For a’ that, an’ a’ that, <br />
It’s coming yet for a’ that, <br />
That Man to Man, the world o’er, <br />
Shall brothers be for a’ that.</p>
</blockquote>
<p>Modern English:</p>
<blockquote>
<p>Should honest poor hang their heads? <br />
We pass by the coward ashamed of his poverty <br />
We dare be poor despite all that! <br />
Despite all that, and all that, <br />
Our humble work, and all that, <br />
Aristocratic rank is but the form that gold is cast into, <br />
The man himself is the gold, despite all that.</p>
<p>So what if we dine on homely fare, <br />
Wear rough grey tweed, and all that? <br />
Give fools their silks, and knaves their wine - <br />
A man is a man despite all that. <br />
Despite all that, and all that, <br />
Their ostentation, and all that, <br />
The honest man, though ever so poor, <br />
Is king of men despite all that.</p>
<p>You see that person called a “lord”, <br />
Who struts, and postures, and all that? <br />
Though hundreds worship at his word, <br />
He is but a fool for all that. <br />
Despite all that, and all that, <br />
His regalia, and all that, <br />
The man of independent mind, <br />
He looks and laughs at all that.</p>
<p>A prince can bestow the title of knight, <br />
Or marquis, duke, and all that! <br />
But an honest man is above all of these - <br />
Good faith, he must not fault that <br />
Despite all that, and all that, <br />
Their titles, and all that, <br />
Strength of sense and pride of merit <br />
Are higher rank than all that.</p>
<p>Then let us pray that it may come <br />
(And it will come despite all that) <br />
That sense and merit over all the earth <br />
Will prevail and all that! <br />
Despite all that, and all that, <br />
It is coming yet despite all that, <br />
That man to man the world over <br />
Will be brothers despite all that.</p>
</blockquote>
Short musings2021-06-20T00:00:00+00:00https://andrew222651.com/2021/06/20/short-musings<script>
// only run if we're on the post's dedicated page
if (window.location.pathname.indexOf("short-musings") > -1) {
window.location.replace("https://github.com/amacfie/public_notes/wiki/Short-musings");
}
</script>
<!-- alternative title: Hot Takes -->
<p>When I was 17 my lowest grade was in math and I thought I wasn’t good at it.
One year later I was obsessed with it. Things can change.</p>
<p>Robin Hanson says that academia views impractical research as more prestigious.
Yes, pure mathematics and theoretical physics are impractical and prestigious
but ceteris paribus a research finding plus an application is more prestigious
than just a research finding.</p>
<p>There’s a meta-contrarian idea that the mechanisms of academia exclude some
really good science that’s just too unconventional. This is not true to the
extent claimed.
<!-- Tyler Cowen said something like this somewhere. --></p>
<p>Computer algebra is useful but discovering new algorithms to automate
mathematical work is hard.</p>
<p>As Robin Hanson and Steve Levitt say, life is long. There’s lots of time to
do lots of different things.</p>
<p>Re: <a href="https://applieddivinitystudies.com/2020/09/05/rationality-winning/">Where are All the Successful Rationalists?</a>,
rationality is an important scientific concept in AI, finance, and statistics;
its value as a self-help technique is not so clear.</p>
<p>Juergen Schmidhuber is right and Tyler Cowen is wrong: China will surpass
the US in dominance this century.</p>
<p>Geoffrey Miller and Robin Hanson have different views on what people are
signaling when they engage in politics: Miller says personal traits and
Hanson says tribal loyalty. Presumably it’s some of each but I find Miller
more convincing.</p>
<p>Robin Hanson says meditation is about signaling who’s a better meditator.
This is an example of meta-contrarianism at one too many levels of meta.</p>
<p><a href="http://mason.gmu.edu/~rhanson/gamblesuits.html">Here</a> Robin Hanson
proposes a much more efficient method of small claims resolution.
The Enlightenment was about such ideas: approaching economic problems
rationally where previously no one realized there was a problem.</p>
<p>The rapid decision-making abilities of basketball and soccer players impress me
as much as the physical.</p>
<p>“Up to 40%” of travelers from developed to developing countries get travelers’
diarrhea; “in the normal population 1% to 2% of persons per year will develop
irritable bowel syndrome (IBS), and 5% to 6% of travelers after traveler’s
diarrhea will develop IBS”; and “the prevalence of depression and anxiety in
IBS patients is 37.1 and 31.4% respectively”.</p>
<p>The Princeton Companion to Mathematics says “algebraists like to work with exact
formulas and analysts use estimates. Or, to put it even more succinctly,
algebraists like equalities and analysts like inequalities”. In computer
science, algebraists like programming languages and analysts like algorithms and
complexity. Or, to put it even more succinctly, algebraists like lambda calculus
and analysts like Turing machines.</p>
The Confucian virtue of learning2021-01-18T00:00:00+00:00https://andrew222651.com/2021/01/18/three-character-classic<p>The Three Character Classic is a 13th century Chinese text with three
characters per line which is traditionally read by children.
Below is an excerpt from the 1812
<a href="https://play.google.com/books/reader?id=YC4OAAAAIAAJ">translation</a>
by Robert Morrison, Presbyterian missionary and author of the first
Chinese-English dictionary.</p>
<!--
Giles literal translation w/ commentary 1910
https://en.wikisource.org/wiki/San_Tzu_Ching/San_Tzu_Ching
according to wikipedia, it was also translated by Hong Xiuquan
https://www.imo-official.org/country_hall.aspx?code=CHN
-->
<blockquote>
<p>Chung-ni [another name for Confucius] once called a boy of ten years of age
his instructor; for, of old, even perfect and wise men learned diligently.</p>
<p>Chao, when he held the office of Chung-ling, read Sun-yu. Though filling so
high a situation, he yet learned diligently – so much so, that he never laid
the book out of his hand.</p>
<p>In the time of the emperor Sung, Lu-wen-shu was constantly looking over the
books engraven on leaves.</p>
<p>Wu-yao made leaves of the reed bamboo, by paring it thin. Though he did not
possess books [as we do], he exerted himself in the pursuit of knowledge.</p>
<p>Sun-king suspended his head by its hair to the beam of his house, to prevent
his sleeping over his books.</p>
<p>Su-tsin pricked his thigh with an awl, to prevent his sleeping.</p>
<p>Those persons, though not taught, of themselves rigorously pursued their
studies.</p>
<p>Che-yin, when a boy, being poor, read his book by the light of a glow-worm
which he confined. And Sun-kang, in winter, read his book by the light reflected
from snow. Though their families were poor they studied incessantly.</p>
<p>Chu-mai-chin, though he subsisted by carrying fire-wood round the town to sell,
yet carefully read his book. At last he became capable of, and filled a public
office.</p>
<p>Li-mie, while watching his cattle in the field, always had his book at hand,
suspended to the horn of a cow. These two persons, though their bodies were
wearied by labor yet studied hard.</p>
<p>Su-lao-tsiuen, at the age of twenty-seven years began to exert himself, and
read a great many books. He, when at that age, repented of his delay: you,
a little boy, should early consider.</p>
<p>Leang-hao, at the age of eighty-two, was permitted to answer the emperor in
his palace, and was placed at the head of all the literati. In the evening of
life his wishes were fulfilled, and all spoke of his extraordinary learning.
You, a little boy, ought to determine to pursue your studies.</p>
<p>Yung, at eight hears of age could recite the Odes. Li-pi, at seven years of age
could play chess. These clever and studious boys were called by everyone
wonderful. You, youths, ought to imitate them.</p>
<p>Tsai-wen-ki could play a stringed instrument. Sie-tao-wen could sing well.
These ladies were clever. You, who are a gentleman, ought at an early time of
life, to perfect that which is suitable.</p>
<p>Chin-tung, a remarkable lad, was raised by the emperor to fill the office of
Ching-tsi. He, though a youth, was made a public officer. Do you, youths,
exert yourselves to learn, and you may arrive at the same. Let all who make
learning their pursuit be as those persons whom we have mentioned.</p>
<p>It is natural for a dog to watch at night, and for a cock to crow in the
morning; if anyone does not learn, how can he be called a man?</p>
</blockquote>
Uncertainty due to computational approximation in Bayesian inference2020-10-31T00:00:00+00:00https://andrew222651.com/2020/10/31/uncertainty-of-approx-bayes<p>In Bayesian inference, we can factor approximate computation (e.g.
<a href="https://www3.nd.edu/~esims1/log_linearization_sp12.pdf">linearization</a>) into the actual
posterior probabilities.</p>
<p>Suppose we have a <a href="https://en.wikipedia.org/wiki/Probability_mass_function">pmf</a>
\(f(x) = P(X=x)\) which is hard to compute.
If we approximate \(f\) by \(\tilde{f}\) then</p>
\[\begin{align*}
P\left(X = a \,|\, \text{we only compute } \tilde{f}\right)
&= \sum_x x P \left(f(a)=x \,|\, a, \tilde{f}(a) \right)\\
&= E\left(f(a) \,|\, a, \tilde{f}(a) \right)
\end{align*}\]
<p>What is \(P\left(f(a)=x \,|\, a, \tilde{f}(a)\right)\)?
Well, if \(f\) is hard to compute then we probably can’t gather much data, so
there are various options to produce a subjective belief:</p>
<ul>
<li>average-case analysis of \(\tilde{f}\) with an uninformed prior, e.g.
probabilistic numerics</li>
<li>reference classes of “similar” cases</li>
<li>uniform distribution across worst-case bounds</li>
<li>past empirical experience</li>
<li>etc.</li>
</ul>
<p>Note that if the mean of the pmf \(P(f(a)=\cdot \,|\, a, \tilde{f}(a))\)
is \(f(a)\) then \(P(X = a \,|\, \text{we only compute } \tilde{f}) = P(X=a)\).
So accounting for uncertainty due to approximation is equivalent to
“de-biasing” it.</p>
<p><em>Example:</em>
Suppose \(f\) has a single atom and our approximation \(\tilde{f}\) is
modeled as \(f\) shifted by some unknown amount:
\(\tilde{f}(x) = f(x + Y - 5)\), where
\(Y \sim {\rm B{\small IN}}(10, 1/2)\).
If \(\tilde{f}(0) = 1\), then</p>
\[\begin{align*}
P(X=0 \,|\, \text{we only compute } \tilde{f})
&= P(f(0) = 1 \,|\, \tilde{f}(0) = 1) \\
&\approxeq P(\tilde{f}(0) = 1 \,|\, f(0) = 1) \\
&=\binom{10}{5} 2^{-10} \doteq 0.246.
\end{align*}\]
<p>(The approximate equality holds if, say, we assume the location of the atom is
a priori uniformly distributed on a large integer interval.)</p>
<p>Note that this is not completely new. E.g. when <a href="https://www.nist.gov/publications/software-testing-statistical-methods-preliminary-success-estimates-approaches-based-0">inferring how likely it is
that software is bug-free based on a finite set of
tests</a>,
we are putting probability distributions on mathematically determined
statements, assuming the software is deterministic.</p>
<p>Inference is approximated for computational reasons in many places such as
linearization as mentioned already, clustering by compression using a zip
algorithm (instead of computing Kolmogorov complexity), PASS-GLM, MCMC
sampling, numerical methods, approximation algorithms, probabilistic
data structures, et cetera.</p>
<p>Is this ultimately rigorous in a decision theoretic sense? I don’t think so,
but
<a href="http://amacfie.github.io/2017/10/10/probability-riemann-hypothesis/">what is rigorous</a>
can easily be mathematically intractable.
So whatever, it’s a heuristic.</p>
Comment ranking algorithms: Hacker News vs. YouTube vs. Reddit2020-05-21T00:00:00+00:00https://andrew222651.com/2020/05/21/comment-ranking<p>Comments on social sites have to be sorted somehow.
How do big platforms do it – is it some complicated mix of
recommender systems,
learning-to-rank algorithms,
Markov decision processes,
neural networks, and
learning automata?
Well, maybe in <a href="https://engineering.linkedin.com/blog/2017/09/serving-top-comments-in-professional-social-networks">some cases</a>
but often it’s just a simple formula.
In this article we put the formulas used by Hacker News, YouTube, and
Reddit, along with a few alternatives, to the test, using virtual comment
section simulations.
Spoiler alert: YouTube does not do well.</p>
<h2 id="the-simulation-model">The simulation model</h2>
<p>240 visitors arrive at equally spaced increments over a 24 hour period.
Each visitor is randomly assigned as a commenter (10%) or a voter (90%).
Commenters leave a single comment, which gets a randomly assigned quality
category: great (10%), mediocre (80%), or stinker (10%).
Great comments have a high probability of receiving upvotes and a low
probability of receiving downvotes;
stinkers are the reverse;
and mediocre comments have a low probability of receiving any votes.
Voters, on the other hand, see the top-ranked comment and vote according
to its probabilities.
At this point they stop reading or keep going based on a probability that
depends on the vote they just gave (0% for upvotes, 50% for downvotes, 15% for
non-votes).
If they don’t leave, they see the next-ranked comment and the process continues
until they finally do leave or they read all the comments.
When the simulation concludes, we log the average number of upvotes per
visitor which we use as our utility function.</p>
<p>See <a href="https://github.com/amacfie/amacfie.github.io/tree/master/_posts/comment_orderings">Python source code</a>
for full details.</p>
<p>Of course this is not a perfect model of every comment section.
These parameter values will not always be accurate, although I did play around
with e.g. the commenter/voter ratio
<!--and how likely a voter is to leave-->
and I got basically the same final conclusions.
Realistically the rate of visitors may vary over time.
A voter’s probability of leaving after a certain comment conditional on the
most recent (non-)vote may also depend on how many comments they’ve already
read.
Comment threads are not represented here.
Vote probabilities may change over time.
Et cetera, et cetera.</p>
<h2 id="the-ranking-formulas">The ranking formulas</h2>
<p>Here we use the following symbols</p>
<ul>
<li>Number of upvotes received so far: \(n_{+}\)</li>
<li>Number of downvotes received so far: \(n_{-}\)</li>
<li>Age of comment, in hours: \(h\)</li>
</ul>
<p>All ranking methods in our analysis rank comments by scoring each comment
and sorting in descending order.
The scores are determined by the formulas below.</p>
<p>Starting with the basics, we have the <em>ratio</em>
\((n_{+} - n_{-})/(n_{+} + n_{-})\)
and the <em>difference</em> \(n_{+} - n_{-}\), a.k.a. the number of net upvotes.
We don’t expect these to be optimal but they’re useful baselines.
Another version of the ratio is
\(n_{+}/(n_{+} + n_{-})\) which performs similarly.</p>
<p>For testing purposes, we have the <em>random</em> ranking which is, well, just
random, and the <em>upvote probability</em> ranking which ranks according to the true
upvote probability.</p>
<p>Reddit’s algorithm, detailed <a href="https://www.evanmiller.org/how-not-to-sort-by-average-rating.html">here</a>,
is a frequentist method for estimating the true voting probabilities
based on \(n_{+}\) and \(n_{-}\).
The <a href="https://districtdatalabs.silvrback.com/computing-a-bayesian-estimate-of-star-rating-means">Bayesian</a>
version of this is what we’ll call the <em>Bayesian average</em>: the same as
<em>ratio</em> but we imagine that a few extra “phantom” votes have been cast, say 3
downvotes and 3 upvotes.</p>
<p>Hacker News <a href="https://medium.com/hacking-and-gonzo/how-hacker-news-ranking-algorithm-works-1d9b0cf2c08d">roughly</a>
uses the formula \((n_{+} - n_{-}) / (h+2)^{1.8}\),
which is like <em>ratio</em>, if we interpret the denominator \((h+2)^{1.8}\)
as an estimate of the number votes cast.
In fact, this denominator is probably more naturally thought of as an
estimate of the number of votes cast including implicit non-votes.
Non-votes (with a value of 0) would not impact the numerator.</p>
<p>To get a sense of how the simulations look, here are the comments as presented
to the 240th visitor from one run using the Hacker News scoring formula:</p>
<table>
<thead>
<tr>
<th>\(h\)</th>
<th>Upvote probability</th>
<th>Downvote probability</th>
<th>\(n_{+}\)</th>
<th>\(n_{-}\)</th>
<th>HN score</th>
</tr>
</thead>
<tbody>
<tr>
<td>7.9</td>
<td>0.671</td>
<td>0.324</td>
<td>47</td>
<td>5</td>
<td>0.657</td>
</tr>
<tr>
<td>14.2</td>
<td>0.671</td>
<td>0.076</td>
<td>82</td>
<td>3</td>
<td>0.515</td>
</tr>
<tr>
<td>21.9</td>
<td>0.496</td>
<td>0.14</td>
<td>110</td>
<td>10</td>
<td>0.324</td>
</tr>
<tr>
<td>23.3</td>
<td>0.434</td>
<td>0.051</td>
<td>72</td>
<td>12</td>
<td>0.174</td>
</tr>
<tr>
<td>8.9</td>
<td>0.162</td>
<td>0.03</td>
<td>8</td>
<td>0</td>
<td>0.094</td>
</tr>
<tr>
<td>14.1</td>
<td>0.112</td>
<td>0.054</td>
<td>12</td>
<td>3</td>
<td>0.060</td>
</tr>
<tr>
<td>10.9</td>
<td>0.184</td>
<td>0.058</td>
<td>6</td>
<td>0</td>
<td>0.059</td>
</tr>
<tr>
<td>5.1</td>
<td>0.151</td>
<td>0.008</td>
<td>2</td>
<td>0</td>
<td>0.058</td>
</tr>
<tr>
<td>12.9</td>
<td>0.226</td>
<td>0.049</td>
<td>6</td>
<td>0</td>
<td>0.046</td>
</tr>
<tr>
<td>15.0</td>
<td>0.114</td>
<td>0.061</td>
<td>10</td>
<td>6</td>
<td>0.024</td>
</tr>
<tr>
<td>7.3</td>
<td>0.021</td>
<td>0.009</td>
<td>1</td>
<td>0</td>
<td>0.017</td>
</tr>
<tr>
<td>13.4</td>
<td>0.071</td>
<td>0.008</td>
<td>1</td>
<td>1</td>
<td>0.0</td>
</tr>
<tr>
<td>5.2</td>
<td>0.489</td>
<td>0.038</td>
<td>0</td>
<td>0</td>
<td>0.0</td>
</tr>
<tr>
<td>3.6</td>
<td>0.151</td>
<td>0.041</td>
<td>1</td>
<td>0</td>
<td>0.0</td>
</tr>
<tr>
<td>1.0</td>
<td>0.579</td>
<td>0.087</td>
<td>0</td>
<td>0</td>
<td>0.0</td>
</tr>
<tr>
<td>0.7</td>
<td>0.047</td>
<td>0.024</td>
<td>0</td>
<td>0</td>
<td>0.0</td>
</tr>
<tr>
<td>21.7</td>
<td>0.158</td>
<td>0.222</td>
<td>19</td>
<td>20</td>
<td>-0.003</td>
</tr>
<tr>
<td>20.7</td>
<td>0.048</td>
<td>0.017</td>
<td>1</td>
<td>3</td>
<td>-0.007</td>
</tr>
<tr>
<td>10.4</td>
<td>0.055</td>
<td>0.044</td>
<td>1</td>
<td>2</td>
<td>-0.010</td>
</tr>
<tr>
<td>11.3</td>
<td>0.041</td>
<td>0.027</td>
<td>0</td>
<td>2</td>
<td>-0.018</td>
</tr>
<tr>
<td>19.5</td>
<td>0.104</td>
<td>0.166</td>
<td>5</td>
<td>10</td>
<td>-0.019</td>
</tr>
<tr>
<td>5.4</td>
<td>0.045</td>
<td>0.604</td>
<td>1</td>
<td>3</td>
<td>-0.054</td>
</tr>
</tbody>
</table>
<p>YouTube also uses a <a href="https://stackoverflow.com/a/39048550">formula</a> that
involves the age of the comment.
Their system additionally factors in the user’s lifetime ratio, which
for our tests we set to 0 as if all users are new.</p>
<p>Lastly, let’s consider how we might modify the Bayesian average to take
time into account.
To make new comments more visible we’ll make the phantom votes all upvotes
at first, then asymptotically reduce them to non-votes.
We’ll also switch to a denominator similar to the Hacker News formula’s in
order to estimate non-votes.
This yields the <em>modified Bayes</em> formula</p>
\[\frac{n_{+} - n_{-} + n_p / (h+1)}{n_p + h},\]
<p>where \(n_p\) is the number of phantom votes.
We use the value \(n_p=7\) in the simulations.</p>
<h2 id="ranking-the-rankings">Ranking the rankings</h2>
<p>I did enough simulation runs (1000-20000) with each formula
to be pretty confident about how they compare.
Without further ado, voila:</p>
<table>
<thead>
<tr>
<th>Ranking algorithm</th>
<th>Average number of upvotes per visitor</th>
</tr>
</thead>
<tbody>
<tr>
<td><em>Upvote probability</em></td>
<td>0.978</td>
</tr>
<tr>
<td><em>Modified Bayes</em></td>
<td>0.916</td>
</tr>
<tr>
<td>Hacker News</td>
<td>0.899</td>
</tr>
<tr>
<td><em>Bayesian average</em></td>
<td>0.878</td>
</tr>
<tr>
<td><em>Difference</em></td>
<td>0.848</td>
</tr>
<tr>
<td>Reddit</td>
<td>0.836</td>
</tr>
<tr>
<td><em>Ratio</em></td>
<td>0.813</td>
</tr>
<tr>
<td>YouTube</td>
<td>0.644</td>
</tr>
<tr>
<td><em>Random</em></td>
<td>0.607</td>
</tr>
</tbody>
</table>
<p>So YouTube is marginally better than <em>random</em>, Reddit is worse than the simple <em>difference</em>, and
Hacker News is the only one of the three better than <em>Bayesian average</em>.
Disappointing but also plausible. How generalizable are the results?
As always, more work required…</p>
Mathematics as a service2020-04-01T00:00:00+00:00https://andrew222651.com/2020/04/01/math-as-a-service<p>What would a market for mathematics look like?</p>
<p>Formal verification might allow an elegant mechanism:
Someone posts a proposition in a formal language like Coq and the first
to submit a proof that passes verification wins the bounty.
Everything can be automated and maybe even trustless.
This has been tried, at proofmarket.org, which was <a href="https://medium.com/@pirapira/ten-ethereum-related-pending-projects-you-could-take-c828a2dce88e">shut
down</a>
due to consistency bugs in the verifier.
Even without bugs, proof assistants are still difficult to use;
mathematician Thomas Hales <a href="https://jiggerwit.wordpress.com/2018/09/18/a-review-of-the-lean-theorem-prover/">says</a>
“It is very hard to learn to use <a href="https://leanprover.github.io/">Lean</a>
proficiently. Are you a graduate student at Stanford, CMU, or Pitt writing a
thesis on Lean? Are you a student at Imperial being guided by Kevin Buzzard?
If not, Lean might not be for you.”</p>
<p>If we stick to natural language to avoid the learning curve, things get messy.
How does the market decide what a complete proof is, which proof is first, and
who did it? Perhaps the only tenable solution is to leave these decisions
up to the individuals who post the bounties. How would we
know that bounties would ever get paid? Stack Exchange forces bounties to be
put in escrow and if they’re not awarded to someone there’s no refund. Another
option is to rely on reputation by using certified identities (e.g. users’
email addresses are verified and
<a href="https://www.gtricks.com/google/hide-email-address-online-recaptcha-tool/">public</a>
so they can be checked against personal webpages).</p>
<p>Something along these lines might be doable (and if someone wants to build it
I’ll donate the domain proofbounty.com) but what’s the use case?
Monetary rewards for mathematical problems are <a href="https://mathoverflow.net/questions/66084/open-problems-with-monetary-rewards">rare</a>
and mathematicians generally already earn a salary, so the interest would
likely be modest.
Students (anywhere in the world) are plausible suppliers though, perhaps
even <a href="https://www.imo-official.org/results.aspx">high school students</a>,
while consumers could be anyone with a research grant usable for paying
“research assistants”, or industry and non-profit research groups.
A market that brings these two sides together could be of some value.</p>
<p>Paid question answering has been tried before, e.g.
<a href="http://answers.google.com/answers/browse/1707.html">Google Answers</a> which
wasn’t very popular.
Did it fail due to lack of network effects,
lack of <a href="https://thenextweb.com/us/2010/07/09/will-you-pay-to-comment-online/">innovative mechanisms</a>,
or an essential flaw in the concept? I don’t know.
<a href="https://www.bountysource.com/">Bounties</a> <a href="https://gitstart.com/">on</a>
GitHub issues seem to be a bit more successful.</p>
<p>In addition to bounties, there could be a prediction market.
The time of resolution may have to be indefinite, though, since
resolving “proposition X will be publicly proved by date Y” would in general
require determining the nonexistence of a public proof, which is at least
somewhat error-prone.
However, prediction markets are basically illegal so it’s a moot point.</p>
March 2020 links2020-03-24T00:00:00+00:00https://andrew222651.com/2020/03/24/links<p>James I’s 1597 book <a href="https://en.wikipedia.org/wiki/Daemonologie"><em>Daemonologie</em></a>,
“a philosophical dissertation on contemporary necromancy … touches on topics
such as werewolves and vampires”.</p>
<p><a href="https://iovs.arvojournals.org/article.aspx?articleid=2166142">96.5% of 19-year-old males in Seoul have myopia</a>.</p>
<p><a href="https://en.wikipedia.org/wiki/List_of_Scottish_Canadians">List of Scottish Canadians</a>.</p>
<p>Free
<a href="https://drive.google.com/file/d/1l0ymFMKh5-BB8FK0Wiw1ytf7eG_74k-c/view?usp=sharing">ebook</a> of
classic novel plot summaries.</p>
<p><a href="https://www.socr.umich.edu/spacekime/">“Kime”: complex-valued time</a>.</p>
Robin Hanson predicts China virus disaster2020-02-17T00:00:00+00:00https://andrew222651.com/2020/02/17/hanson-corona<p>Robin Hanson <a href="https://twitter.com/robinhanson/status/1229209586336489472">says</a>
“In few months, China is likely to be a basket case, having crashed their
economy in failed attempt to stop COVID-19 spreading.” Quantifying the
forecast, he says China’s economy (or growth?) will be “a factor of two to ten
down” and seems to expect dramatic results in 6 months.</p>
Ranking cities by weather2020-02-12T00:00:00+00:00https://andrew222651.com/2020/02/12/weather<p>Let’s analyze data from <a href="https://darksky.net">https://darksky.net</a> from the last 10 years to compare
weather (technically “climate”) in a selection of North American cities.</p>
<script>
// only run if we're on the post's dedicated page (hack)
if (window.location.pathname.indexOf("weather") > -1) {
window.location.replace("https://github.com/amacfie/public_notes/wiki/Comparing-cities-by-climate");
}
</script>
<p>If we define a “nice day” as one where</p>
<ul>
<li>there are at least 10 hours of daylight,</li>
<li>the high apparent temperature is at least 0°C and at most 30°C,</li>
<li>the cloud cover is at most 70%, and</li>
<li>the UV index is at most moderate (unfortunately I used UV index at a single
point in time during the day and didn’t adjust for time zones),</li>
</ul>
<p>we get:</p>
<table>
<thead>
<tr>
<th>City</th>
<th>Probability of nice day</th>
</tr>
</thead>
<tbody>
<tr>
<td>San Diego</td>
<td>0.27</td>
</tr>
<tr>
<td>Los Angeles</td>
<td>0.23</td>
</tr>
<tr>
<td>San Francisco</td>
<td>0.22</td>
</tr>
<tr>
<td>Raleigh</td>
<td>0.22</td>
</tr>
<tr>
<td>Austin</td>
<td>0.2</td>
</tr>
<tr>
<td>Vancouver</td>
<td>0.19</td>
</tr>
<tr>
<td>New York</td>
<td>0.19</td>
</tr>
<tr>
<td>Cambridge</td>
<td>0.19</td>
</tr>
<tr>
<td>Chicago</td>
<td>0.16</td>
</tr>
<tr>
<td>Ottawa</td>
<td>0.16</td>
</tr>
<tr>
<td>Toronto</td>
<td>0.15</td>
</tr>
</tbody>
</table>
<p>What are the nicest months to visit Toronto?</p>
<table>
<thead>
<tr>
<th>Month</th>
<th>Average number of nice days in Toronto</th>
</tr>
</thead>
<tbody>
<tr>
<td>January</td>
<td>0</td>
</tr>
<tr>
<td>February</td>
<td>2.9</td>
</tr>
<tr>
<td>March</td>
<td>9.0</td>
</tr>
<tr>
<td>April</td>
<td>4.7</td>
</tr>
<tr>
<td>May</td>
<td>1.2</td>
</tr>
<tr>
<td>June</td>
<td>0.4</td>
</tr>
<tr>
<td>July</td>
<td>0.5</td>
</tr>
<tr>
<td>August</td>
<td>4.0</td>
</tr>
<tr>
<td>September</td>
<td>12.1</td>
</tr>
<tr>
<td>October</td>
<td>15.8</td>
</tr>
<tr>
<td>November</td>
<td>2.4</td>
</tr>
<tr>
<td>December</td>
<td>0</td>
</tr>
</tbody>
</table>
<p>If we define a “sunny day” as one where</p>
<ul>
<li>there are at least 10 hours of daylight,</li>
<li>the high apparent temperature is at least 15°C, and</li>
<li>the cloud cover is at most 50%,</li>
</ul>
<p>we get:</p>
<table>
<thead>
<tr>
<th>City</th>
<th>Probability of sunny day</th>
</tr>
</thead>
<tbody>
<tr>
<td>Los Angeles</td>
<td>0.69</td>
</tr>
<tr>
<td>Austin</td>
<td>0.56</td>
</tr>
<tr>
<td>San Francisco</td>
<td>0.49</td>
</tr>
<tr>
<td>Raleigh</td>
<td>0.46</td>
</tr>
<tr>
<td>San Diego</td>
<td>0.45</td>
</tr>
<tr>
<td>New York</td>
<td>0.33</td>
</tr>
<tr>
<td>Cambridge</td>
<td>0.32</td>
</tr>
<tr>
<td>Chicago</td>
<td>0.26</td>
</tr>
<tr>
<td>Toronto</td>
<td>0.23</td>
</tr>
<tr>
<td>Vancouver</td>
<td>0.2</td>
</tr>
<tr>
<td>Ottawa</td>
<td>0.18</td>
</tr>
</tbody>
</table>
<p>What are the sunniest months to visit Toronto?</p>
<table>
<thead>
<tr>
<th>Month</th>
<th>Average number of sunny days in Toronto</th>
</tr>
</thead>
<tbody>
<tr>
<td>January</td>
<td>0</td>
</tr>
<tr>
<td>February</td>
<td>0</td>
</tr>
<tr>
<td>March</td>
<td>0.7</td>
</tr>
<tr>
<td>April</td>
<td>2.6</td>
</tr>
<tr>
<td>May</td>
<td>10.0</td>
</tr>
<tr>
<td>June</td>
<td>12.5</td>
</tr>
<tr>
<td>July</td>
<td>17.9</td>
</tr>
<tr>
<td>August</td>
<td>17.8</td>
</tr>
<tr>
<td>September</td>
<td>15.1</td>
</tr>
<tr>
<td>October</td>
<td>6.1</td>
</tr>
<tr>
<td>November</td>
<td>0.4</td>
</tr>
<tr>
<td>December</td>
<td>0</td>
</tr>
</tbody>
</table>
<p>Lastly, if we define a “warm day” as one where</p>
<ul>
<li>the high apparent temperature is at least 15°C and at most 25°C and</li>
<li>the UV index is at most high,</li>
</ul>
<p>we get:</p>
<table>
<thead>
<tr>
<th>City</th>
<th>Probability of warm day</th>
</tr>
</thead>
<tbody>
<tr>
<td>San Diego</td>
<td>0.5</td>
</tr>
<tr>
<td>San Francisco</td>
<td>0.45</td>
</tr>
<tr>
<td>Los Angeles</td>
<td>0.37</td>
</tr>
<tr>
<td>Vancouver</td>
<td>0.33</td>
</tr>
<tr>
<td>Raleigh</td>
<td>0.28</td>
</tr>
<tr>
<td>New York</td>
<td>0.25</td>
</tr>
<tr>
<td>Austin</td>
<td>0.25</td>
</tr>
<tr>
<td>Ottawa</td>
<td>0.23</td>
</tr>
<tr>
<td>Toronto</td>
<td>0.23</td>
</tr>
<tr>
<td>Cambridge</td>
<td>0.22</td>
</tr>
<tr>
<td>Chicago</td>
<td>0.21</td>
</tr>
</tbody>
</table>
<p>What are the warmest months to visit Toronto?</p>
<table>
<thead>
<tr>
<th>Month</th>
<th>Average number of warm days in Toronto</th>
</tr>
</thead>
<tbody>
<tr>
<td>January</td>
<td>0</td>
</tr>
<tr>
<td>February</td>
<td>0.3</td>
</tr>
<tr>
<td>March</td>
<td>1.8</td>
</tr>
<tr>
<td>April</td>
<td>6.9</td>
</tr>
<tr>
<td>May</td>
<td>11.7</td>
</tr>
<tr>
<td>June</td>
<td>11.5</td>
</tr>
<tr>
<td>July</td>
<td>4.8</td>
</tr>
<tr>
<td>August</td>
<td>10.2</td>
</tr>
<tr>
<td>September</td>
<td>19.9</td>
</tr>
<tr>
<td>October</td>
<td>13.7</td>
</tr>
<tr>
<td>November</td>
<td>2.1</td>
</tr>
<tr>
<td>December</td>
<td>0.1</td>
</tr>
</tbody>
</table>
Q&A with William Saunders: Preventing AI catastrophes2020-02-11T00:00:00+00:00https://andrew222651.com/2020/02/11/saunders<p><img src="https://andrew222651.com/public/images/will.jpg" alt="William Saunders" /></p>
<p>William Saunders was a fellow Fellow at <a href="https://intelligence.org/">MIRI</a> in
2016 and now researches AI safety at <a href="https://ought.org/">Ought</a>.
Below we go over his 2017 paper
<a href="https://arxiv.org/pdf/1707.05173.pdf">“Trial without Error: Towards Safe Reinforcement Learning via Human
Intervention”</a>.</p>
<p><strong>Q</strong>:
Say we’re training an autonomous car by running a bunch of practice trips and
letting the model learn from experience. For example, to teach safe driving we
might input a reward if it makes a trip without running anyone over and input a
penalty otherwise. What’s the flaw in this approach, and how serious is this
issue in AI systems present and future?</p>
<blockquote>
<p>Two big flaws, if we use traditional model-free reinforcement learning
algorithms (Deep Q learning, policy gradient):</p>
<ul>
<li>The <a href="https://en.wikipedia.org/wiki/Reinforcement_learning">RL</a> agent won’t learn to avoid running over the human until it
actually runs over the human and recieves the penalty a large number of
times.</li>
<li>The RL agent will suffer “The <a href="https://en.wikipedia.org/wiki/Sisyphus">Sisyphean</a> Curse of RL”. Once it learns
to avoid running over humans, it will keep having new experiences where it
doesn’t run over humans. Eventually, it will forget that running over
humans is bad, and occasionally needing to run over humans a few times and
get penalized in order to remember. This will repeat as long as the agent
is being trained.</li>
</ul>
<p>So, the training process can lead to an arbitrary number of humans being
run over. (In practice of course, you’d stop after the first one if not
sooner).</p>
</blockquote>
<p><strong>Q</strong>:
Your proposal, called Human Intervention Reinforcement Learning (HIRL),
involves using humans to prevent unwitting AIs from taking dangerous
actions.
How does it work?</p>
<blockquote>
<ol>
<li>A human watches the training process. Whenever the RL agent is about to
do something catastrophic, the human intervenes, changing the RL agent’s
action to avoid the catastrophe and giving the RL agent a penalty.</li>
<li>We record all instances when the human intervenes, and train a
supervised learning algorithm (“the blocker”) to predict when the human
intervenes.</li>
<li>When the blocker is able to predict when the human intervenes, we
replace the human with the blocker and continue training. Now the blocker
is called for every new action the agent takes, and decides whether it
should intervene and penalize the agent.</li>
<li>Eventually, the RL agent should learn a policy that performs well on the
task and avoids proposing the blocked actions, which should then be safe
for deployment.</li>
</ol>
</blockquote>
<p><strong>Q</strong>:
What’s a practical example where HIRL might be useful?</p>
<blockquote>
<p>One example might be for a chatbot that occasionally proposes an offensive
reply in a conversation (e.g. <a href="https://en.wikipedia.org/wiki/Tay_(bot)">Microsoft Tay</a>). A human could review
statements proposed by the chatbot and block offensive ones being sent to
end users.</p>
</blockquote>
<p><strong>Q</strong>:
Is there a use case for HIRL in simulated learning environments?</p>
<blockquote>
<p>In simulated environments, one can simply allow the catastrophic action to
happen and intervene after the fact. But depending on the simulation, it
might be more efficient for learning if catastrophic actions are blocked
(if they would end the simulation early, or cause the simulation to run for
a long time in a failed state).</p>
</blockquote>
<p><strong>Q</strong>:
In what situations would human intervention be too slow or expensive?</p>
<blockquote>
<p>Even for self-driving cars, it can be difficult for a safety driver to
detect when something is going wrong and intervene in time. Other robotics
tasks might be similar.
In many domains, it might not be possible to fully hand things over to the
blocker. If the agent doesn’t try some kinds of actions or encounter some
kinds of situations until later in the training process, you either need to
have the human watch the whole time, or be able to detect when new
situations occur and bring the human back in.</p>
</blockquote>
<p><strong>Q</strong>:
How does the applicability of HIRL change (if at all) if the human is part of
the environment?</p>
<blockquote>
<p>HIRL could still apply if the intervening human is part of the
environment, as long as the human supervisor is able to block any
catastrophic action that harms or manipulates the human supervisor, or the
human supervisor’s communication channel.</p>
</blockquote>
<p><strong>Q</strong>:
Theoretically the idea here is to extract, with an accuracy/cost tradeoff, a
human’s beliefs and/or preferences so an AI can make use of them. At a high
level, how big a role do you think direct human intervention will play in this
process on the road to superintelligent AI?</p>
<blockquote>
<p>Ideally, you would want techniques that don’t require the human to be
watching and able to effectively intervene, it would be better if the
blocker could be trained prior to training or if the AI could detect when
it was in a novel situation and only ask for feedback then. I think
HIRL does illustrate how in many situations it’s easier to check whether an
action is safe than to specify the optimal action to perform, and this
principle might end up being used in other techniques as well.</p>
</blockquote>
Whence the English names of countries2020-01-24T00:00:00+00:00https://andrew222651.com/2020/01/24/english-names-countries<p>Some <a href="https://en.wikipedia.org/wiki/English_exonyms">exonyms</a>:</p>
<ul>
<li>India (Hindustan). Same origin: Sanskrit <em>síndhu</em> (“river”), as in Indus River.</li>
<li>Japan (Nihon). Same origin, the English comes via Chinese and Malay.</li>
<li>China (Zhongguo). Via Sanskrit possibly from Qin, the westernmost ancient
Chinese state.</li>
<li>Korea (Hanguk). From Goryeo, a Korean kingdom from 918-1392.</li>
<li>Germany (Deutschland). From Latin name <em>Germānī</em> used for tribes
east of the Rhine.</li>
<li>Finland (Suomi). From Old Norse word for Finland, <em>Finnland</em>.</li>
<li>Greece (Hellas). Via Latin from <em>Graecus</em>, a son of either Zeus or someone
named Thessalus.</li>
<li>Hungary (Magyarország). Via Latin from Turkish name <em>Onoğurs</em> (“ten tribes”)
used for Turkic tribes to the north of Turkey (not where Hungary is).</li>
</ul>
Nov 2019 links2019-11-06T00:00:00+00:00https://andrew222651.com/2019/11/06/links<p><a href="https://www.skyscrapercenter.com/explore-data">Tall buildings by city: look out for Toronto.</a>
The current top cities in North America for sky scrapers are unambiguously
New York, Chicago, and Toronto in that order.
However, if we count proposed buildings and buildings under construction,
Chicago has <em>18</em> at least 150m tall (the dataset is only complete for buildings
at least 150m) and Toronto has <em>90</em>.</p>
<p><a href="http://content.time.com/time/nation/article/0,8599,1662757,00.html">C. Hitchens sometimes just made stuff up.</a></p>
<p><a href="https://hack.chat/">Web LaTeX chat.</a></p>
<p><a href="https://macinjune.com/all-posts/web-tip/%EC%9C%A0%ED%8A%9C%EB%B8%8C-%EB%8F%99%EC%98%81%EC%83%81-%ED%9A%8C%EC%A0%84-%ED%98%B9%EC%9D%80-%EA%B1%B0%EC%9A%B8-%EB%AA%A8%EB%93%9C%EC%A2%8C%EC%9A%B0-%EB%B0%98%EC%A0%84%EB%A1%9C-%EB%B3%B4%EB%8A%94/#i-2">How to mirror YouTube videos (Korean).</a></p>
<p><a href="https://www.youtube.com/watch?v=Fp364ZUWdeM">How to make the weird Australian O sound.</a></p>
Ascending auction bidder strategy2019-11-05T00:00:00+00:00https://andrew222651.com/2019/11/05/smra-strategy<p>Ascending auctions are a common mechanism for selling a set of products.
The basics are covered in this video:</p>
<iframe width="640" height="360" src="https://www.youtube.com/embed/oHkArtG9zO0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>The exact rules of an ascending auction depend on the auctioneer and may
include complexities such as:</p>
<ul>
<li>Activity rules, where bidders can never bid on more products than in previous
rounds</li>
<li>Anonymous bidding, where information on who bid on what is hidden</li>
<li>Whether bid prices in each round are fixed by the auctioneer or chosen by
bidders</li>
</ul>
<p>Below we review some of the main ideas of optimal bidding strategy, give
practice scenarios, and provide pointers to the relevant literature.</p>
<h3 id="terminology">Terminology</h3>
<p>VCG = <a href="https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_auction">Vickrey-Clarke-Groves</a> (sealed bid, Vickrey prices)</p>
<p>SAA = simultaneous (multiple products at once) ascending auction (same as SMRA)</p>
<p>SMRA = simultaneous multiple round ascending (same as SAA)</p>
<p>CCA = Combinatorial clock auction (not the same as SAA/SMRA)</p>
<p><a href="https://en.wikipedia.org/wiki/Focal_point_(game_theory)">Schelling point</a> = a way for independent parties to intentionally coordinate
on one choice among many</p>
<p>Value bidding = selecting a package to maximize value of package minus cost</p>
<h3 id="notation">Notation</h3>
<p>Products = \(\{1, 2, 3, \ldots, n \}\)</p>
<p>Quantities = \(\{q_1, \ldots, q_n\}\)</p>
<p>Bidders = \(\{1, 2, 3, \ldots, m \}\)</p>
<p>Package: \(x = x(1), \ldots, x(n)\), where \(x(i)\) is the quantity of product \(i\)</p>
<p>Valuation of bidder \(i\) of package \(x\): \(v_i(x)\)</p>
<h3 id="nutshell">Nutshell</h3>
<p>Generally SMRA auctions have a cooperative phase and then a competitive phase.
In the cooperative phase, bidders reduce demand (relative to value bidding) in
order to allocate products without bidding prices up.
Bidders must agree on this allocation without communicating.
Typically this implicit allocation is chosen because it’s fair, natural,
symmetric, or otherwise “makes sense” given the context of the auction.</p>
<h3 id="keys-to-the-game">Keys to the game</h3>
<p>(More details below.)</p>
<ul>
<li>Demand reduction negotiation
<ul>
<li>Bidders try to indirectly find agreeable allocation</li>
<li>Selecting quantities: Schelling points based on available info
<ul>
<li>Auction-based, industry-based info</li>
<li>E.g. split product units 50/50 if two bidders are expected to be
interested</li>
</ul>
</li>
<li>Negotiation by sending signals within the auction
<ul>
<li>Presumably cheap talk in this context, but it happens</li>
<li>Much noise little signal in auctions where bids are constrained
or hidden</li>
</ul>
</li>
<li>If there is an activity rule, once you’ve submitted low demand, there is no
way to increase without decreasing somewhere else</li>
</ul>
</li>
<li>Competition
<ul>
<li>Basically value bidding</li>
<li>Usually happens if negotiation fails</li>
</ul>
</li>
<li><a href="https://en.wikipedia.org/wiki/Complementary_good">Complementarity</a>/exposure:
value bidding fails and “cooperation” is inefficient and less likely.
<ul>
<li>Bids for a quantity \(q\) can turn into bids for quantities \(< q\)
so be careful how high you bid if there are complementarities.</li>
<li>See literature review below for more discussion.</li>
</ul>
</li>
<li>Price raising
<ul>
<li>Only do in lots where you’re not going to win anything</li>
<li>Start early (to maintain activity)</li>
</ul>
</li>
</ul>
<h3 id="demand-reduction">Demand reduction</h3>
<p>Value bidding is no longer a dominant strategy, as it is in VCG/CCA.</p>
<p>Say there is a single product and Bidder 1 bids on quantity 1 at
price=\(1,2,\ldots,10\).
Assume \(v_2(1)=9, v_2(2)=10\).</p>
<p>Bidder 2 (B2), strategy 1: bid on \(q=2\) for \(p=1,\ldots,5\), then bid on \(q=1\)
for \(p=6\).
B2 strategy 2: bid on \(q=1\) for \(p=1\).</p>
<p>B2 results:</p>
<ul>
<li>CCA, strategy 1: wins \(q=1\) @ \(p=0\)</li>
<li>
<p>CCA, strategy 2: wins \(q=1\) @ \(p=0\)</p>
</li>
<li>SMRA, strategy 1: wins \(q=1\) @ \(p=6\)</li>
<li>SMRA, strategy 2: wins \(q=1\) @ \(p=1\)</li>
</ul>
<p>Thus reducing demand (strategy 2) pays in the SMRA format where it didn’t in
the CCA.
When both bidders reduce demand, it’s called “cooperation” aka “tacit
collusion”.
See the literature review below for more examples.</p>
<p>However, with the activity rule, there can be a risk to reducing too much
at the beginning if there is uncertainty about the cooperative outcome, so
a somewhat gradual reduction may be wise.</p>
<h3 id="price-raising">Price raising</h3>
<p>Raising prices for other bidders is a realistic motive.
In the SMRA format it’s relatively simple because raising auction price is the
same as raising price paid.
You don’t have to work backwards from Vickrey price calculations to see what
action would cause an increase in price.
Instead, you simply have to create
excess demand on one or more products where there otherwise would not be.
But, it’s risky because your bids might end up being winning bids.</p>
<p>The ideal scenario is as follows:
Two rivals of yours neatly split supply 50-50, and price doesn’t increase.
Then you come in and place a bid for \(q=1\) (no point using higher \(q\) unless
you need the activity) for a few rounds and then get out before
they decrease their bids.</p>
<p>So this can work for disrupting demand reduction, but only for products you
don’t actually want to win (or you’d be raising your own price too).</p>
<h3 id="demystifying-strategies-through-experimentation">Demystifying strategies through experimentation</h3>
<p>Try the following mini scenarios one or multiple times to better understand
tactics.</p>
<ul>
<li>People are assigned to bidders</li>
<li>Bidders’ valuations may be random (independently among bidders)
<ul>
<li>Other bidders know the possible valuations but not which one was selected</li>
</ul>
</li>
<li>People gain points according to their valuation, lose points to pay for
won products</li>
<li>Possible bonus points for raising rivals’ prices</li>
<li>Goal is to maximize points, not to get more points than opponent</li>
<li>In an actual auction, the other bidders may not be rational</li>
</ul>
<p>After gaining familiarity with the mini scenarios, full scale mock auctions may
also be helpful.</p>
<h4 id="scenario-dealing-with-uncertainty">Scenario: Dealing with uncertainty</h4>
<p>1 product, \(q_1=2\), 2 bidders</p>
\[v_1(1) = 2, v_1(2) = 3\]
<ul>
<li>With probability \(1/2\):
\(v_2(1) = 1, v_2(2) = 2\)</li>
<li>With probability \(1/2\):
\(v_2(1) = 0, v_2(2) = 2\)</li>
</ul>
<h4 id="scenario-are-they-price-raising">Scenario: Are they price raising?</h4>
<p>2 products, \(q_1=q_2=2\), 2 bidders</p>
<p>\(v_1(x, 1) = 2x + 2\),
\(v_1(x, 2) = 2x + 3\)</p>
<p>With probability \(1/3\):</p>
<p>\(v_2(1, y) = 2\), \(v_2(2, y) = 3\)</p>
<p>Bonus points for bidder 2 only if its score is positive:
price paid by bidder 1 for product 2</p>
<p>With probability \(2/3\):</p>
\[v_2(x, 1) = v_2(x, 2) = x + 2\]
<h4 id="scenario-cooperating-without-an-obvious-schelling-point">Scenario: Cooperating without an obvious Schelling point</h4>
<p>1 product, \(q_1=3\), 2 bidders</p>
\[v_1(1)=6, v_1(2)=10, v_1(3)=12\]
\[v_2(1)=6, v_2(2)=10, v_2(3)=12\]
<h4 id="scenario-cooperating-with-uncertainty-1">Scenario: Cooperating with uncertainty 1</h4>
<p>1 product, \(q_1=1\), 2 bidders</p>
<ul>
<li>With probability \(1/2\):
\(v_1(1) = 3\)</li>
<li>
<p>With probability \(1/2\):
\(v_1(1) = 5\);</p>
</li>
<li>With probability \(1/2\):
\(v_2(1) = 4\)</li>
<li>With probability \(1/2\):
\(v_2(1) = 6\)</li>
</ul>
<h4 id="scenario-classic-intra-product-exposure">Scenario: Classic intra-product exposure</h4>
<p>1 product, \(q_1=3\), 3 bidders</p>
<p>\(v_1(3) = 10\), otherwise \(v_1(x)=0\)</p>
<ul>
<li>With probability \(1/2\):
\(v_2(1) = v_2(2) = v_2(3) = 5\)</li>
<li>
<p>With probability \(1/2\):
\(v_2(1) = v_2(2) = v_2(3) = 1\);</p>
</li>
<li>With probability \(1/2\):
\(v_3(1) = v_3(2) = v_3(3) = 4\)</li>
<li>With probability \(1/2\):
\(v_3(1) = v_3(2) = v_3(3) = 0\)</li>
</ul>
<h4 id="scenario-cooperating-with-uncertainty-2">Scenario: Cooperating with uncertainty 2</h4>
<p>2 products, \(q_1=q_2=3\), 2 bidders</p>
<p>\(v_1(1, y) = 3 + \sqrt{2y}\), \(v_1(2, y) = 5 + \sqrt{2y}\),
\(v_1(3, y) = 5 + \sqrt{2y}\)</p>
<ul>
<li>With probability \(1/2\):
\(v_2(1, y) = v_2(2, y) = v_2(3, y) = 2 + \sqrt{y}\)</li>
<li>With probability \(1/2\):
\(v_2(1, y) = 4 + \sqrt{y}, v_2(2, y) = 7 + \sqrt{y}, v_2(3, y) = 8 + \sqrt{y}\)</li>
</ul>
<h4 id="scenario-universal-intra-product-complementarity">Scenario: Universal intra-product complementarity</h4>
<p>1 product, \(q_1=3\), 2 bidders</p>
\[v_1(1) = 2, v_1(2) = 5, v_1(3) = 9\]
\[v_2(1) = 1, v_2(2) = 4, v_2(3) = 8\]
<h4 id="scenario-finding-a-schelling-point">Scenario: Finding a Schelling point</h4>
<p>2 products, \(q_1=q_2=3\), 2 bidders</p>
\[v_1(x,y) = v_2(x,y) = \sqrt{x} + \sqrt{y}\]
<h4 id="scenario-inter-product-cooperation-1">Scenario: Inter-product cooperation 1</h4>
<p>2 products, \(q_1=q_2=1\), 2 bidders</p>
<ul>
<li>With probability 1/2:
\(v_1(x,y) = \sqrt{5x + 3y}\)</li>
<li>
<p>With probability 1/2:
\(v_1(x,y) = \sqrt{3x + 5y}\);</p>
</li>
<li>With probability 1/2:
\(v_2(x,y) = \sqrt{5x + 3y}\)</li>
<li>With probability 1/2:
\(v_2(x,y) = \sqrt{3x + 5y}\)</li>
</ul>
<h4 id="scenario-inter-product-cooperation-2">Scenario: Inter-product cooperation 2</h4>
<p>4 products, \(q_1=q_2=q_3=q_4=1\), 2 bidders</p>
<ul>
<li>With probability 1/2:
\(v_1(x_1, x_2, x_3, x_4) = \sqrt{5(x_1+x_2) + 3(x_3+x_4)}\)</li>
<li>
<p>With probability 1/2:
\(v_1(x_1, x_2, x_3, x_4) = \sqrt{3(x_1+x_2) + 5(x_3+x_4)}\);</p>
</li>
<li>With probability 1/2:
\(v_2(x_1, x_2, x_3, x_4) = \sqrt{5(x_1+x_2) + 3(x_3+x_4)}\)</li>
<li>With probability 1/2:
\(v_2(x_1, x_2, x_3, x_4) = \sqrt{3(x_1+x_2) + 5(x_3+x_4)}\)</li>
</ul>
<h3 id="guide-to-the-literature-theoretical">Guide to the literature: Theoretical</h3>
<p>Brusco, Sandro, and Giuseppe Lopomo.
2002.
<a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.394.3028&rep=rep1&type=pdf">“Collusion via Signaling in Simultaneous Ascending Bid Auctions with
Heterogeneous Objects, with and Without Complementarities.”</a>
<em>The Review of Economic Studies</em> 69 (2): 407–36.</p>
<p>Synopsis:
Increasing the ratio of bidders to products decreases cooperation.
Complementaries among products decreases cooperation.
Optimal strategy is attempting to cooperate and value bidding if that fails.
EP is not part of the model.</p>
<p>Grimm, Veronika, Frank Riedel, and Elmar Wolfstetter.
2003.
<a href="https://www.econstor.eu/bitstream/10419/62770/1/724885277.pdf">“Low Price Equilibrium in Multi-Unit Auctions: The Gsm Spectrum Auction in
Germany.”</a>
<em>International Journal of Industrial Organization</em> 21 (10): 1557–69.</p>
<p>Synopsis:
In German 1999 auction, products were split 50-50 between two major players
at relatively low prices.
A simple game is defined.
Assume there are \(m\) bidders, and \(n=mk\) products each with quantity \(1\),
bidders have equal valuations with strictly decreasing marginal values.
The optimal strategy is to bid on \(k\) products each. If someone competes with
you for your \(k\), value bid.</p>
<p>Brusco, Sandro, and Giuseppe Lopomo.
2009.
<a href="https://www.researchgate.net/profile/Giuseppe_Lopomo/publication/23545651_Simultaneous_ascending_auctions_with_complementarities_and_known_budget_constraints/links/556c30c008aeccd7773a2d0e/Simultaneous-ascending-auctions-with-complementarities-and-known-budget-constraints.pdf">“Simultaneous Ascending Auctions with Complementarities and Known Budget
Constraints.”</a>
<em>Economic Theory</em> 38 (1): 105–24.</p>
<p>Synopsis:
Analysis of exposure problem.
For example:
Big bidder has extremely complementary (convex, e.g. \(x^2\)) values. A number of
small bidders have extremely supplementary (concave, e.g. \(\sqrt{x}\)) values.
Due to lack of package bids, big bidder may decide to not bid at all.
However, in spectrum auctions I’m not sure if this is a big factor.
(Not an issue in CCA/VCG.)</p>
<p>Goeree, Jacob K, and Yuanchuan Lien.
2014.
<a href="http://www.centrobaffi.unibocconi.it/wps/allegatiCTP/Exposure-09142010_1.pdf">“An Equilibrium Analysis of the Simultaneous Ascending Auction.”</a>
<em>Journal of Economic Theory</em> 153: 506–33.</p>
<p>Synopsis: Analysis of exposure problem.</p>
<p>Janssen, Maarten, and Vladimir Karamychev.
2017.
<a href="https://homepage.univie.ac.at/maarten.janssen/auctions/Raising%20Rivals'%20Cost%20in%20Multi-unit%20Auctions-V1%2011V-Ref.pdf">“Raising Rivals’ Cost in Multi-Unit Auctions.”</a>
<em>International Journal of Industrial Organization</em> 50: 473–90.</p>
<p>Synopsis:
Discussion of when raising prices is optimal or suboptimal, when bidders
have an interest in doing so.</p>
<h3 id="guide-to-the-literature-empirical">Guide to the literature: Empirical</h3>
<p>Grimm, Veronika, Frank Riedel, and Elmar Wolfstetter.
2003.
<a href="https://www.econstor.eu/bitstream/10419/62770/1/724885277.pdf">“Low Price Equilibrium in Multi-Unit Auctions: The Gsm Spectrum Auction in
Germany.”</a>
<em>International Journal of Industrial Organization</em> 21 (10): 1557–69.</p>
<p>Synopsis: See above.</p>
<p>Kwasnica, Anthony M, and Katerina Sherstyuk.
2007.
<a href="http://test.scripts.psu.edu/users/a/m/amk17/CollEqmSel.pdf">“Collusion and Equilibrium Selection in Auctions.”</a>
<em>The Economic Journal</em> 117 (516): 120–45.</p>
<p>Synopsis:
Lab experiments were conducted on spontaneous cooperation in auctions.
Results (p15): Players cooperate more if they get to play the game many
times.
As the number of bidders per product increases, cooperation decreases.
With complementary products, there was little cooperation.</p>
<p>Cramton, Peter.
2010.
<a href="http://cramton.umd.edu/papers2000-2004/cramton-simultaneous-ascending-auction.pdf">“Simultaneous Ascending Auctions.”</a>
Wiley Encyclopedia of Operations Research and Management Science.</p>
<p>Synopsis (Sec. 5):
Discusses auctions from around 2000 where bidders signaled and coordinated
to reduce demand.</p>
<p>Bichler, Martin, Vitali Gretschko, and Maarten Janssen.
2017.
<a href="https://homepage.univie.ac.at/maarten.janssen/auctions/bargaining-spectrum-auctions.pdf">“Bargaining in Spectrum Auctions: A Review of the German Auction in 2015.”</a>
<em>Telecommunications Policy</em> 41 (5-6): 325–40.</p>
<p>Synopsis:
Analysis of German auction in 2015 which featured cooperation, competition,
and signaling. The auction had high transparency and a great range of actions
(submitting bids higher than clock price). E.g. TEF bids on product A that
VOD was bidding on to send message that VOD should reduce demand in product B
where TEF and VOD are negotiating demand reduction.</p>
<p>Cramton, Peter, and Axel Ockenfels.
2017.
<a href="http://www.cramton.umd.edu/papers2010-2014/cramton-ockenfels-german-4g-auction.pdf">“The German 4G Spectrum Auction: Design and Behaviour.”</a>
Oxford University Press Oxford, UK.</p>
<p>Synopsis:
Analysis of German auction in 2010 which was competitive due to lack of
(or too many) Schelling points.
Specifically there were different ways to divide up the blocks that might have
made sense depending on factors such as future mergers or network sharing
agreements and bidders worked towards conflicting outcomes.</p>
Belief aggregation with computational constraints2019-09-30T00:00:00+00:00https://andrew222651.com/2019/09/30/bayesian-aggregation<p>Imagine a risk-neutral set of traders, each with a common prior which is
updated with some private information.
The traders buy and sell contingent claims until prices reach an equilibrium.
The resultant prices are the conditional expectations of the terminal payoffs
under a probability measure \(\mathbb{P}\).
Is \(\mathbb{P}\) equal to the posterior obtained by updating the
common prior with the combined private information?
At least under certain conditions,
<a href="https://web.stanford.edu/~ost/papers/aggregation.pdf">yes</a>.</p>
<p>Great, so maybe there should be an efficient distributed algorithm to do
Bayesian inference by splitting up the dataset, doing inference on each
worker, and then aggregating the results?
Well, presumably yes – if workers have an exact representation of their
posteriors.
But if the workers obtain their posteriors approximately by MCMC sampling, the
answer so far is no.
Distributed
<a href="http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41849.pdf">Bayesian consensus methods</a>
exist that use heuristics such as weighted averaging but they “lack rigorous
justification and provide no guarantees on the quality of inference”.</p>
<p>So, do precise distributed Bayesian inference methods exist?
If yes, we unlock a new world of Bayesian big data.
If no, what is the character of belief aggregation in markets with
computationally bounded Bayesian traders?</p>
North American population growth2019-08-19T00:00:00+00:00https://andrew222651.com/2019/08/19/population<p>Which places in North America are growing and which aren’t?
City populations are poorly defined and hard to compare but
the boundaries of states and provinces are more objective.
Here is the growth in population since 1970/71
in states/provinces with over 4 million people.</p>
<table>
<thead>
<tr>
<th>Province</th>
<th>1971 Pop.</th>
<th>2016 Pop.</th>
<th>Growth</th>
</tr>
</thead>
<tbody>
<tr>
<td>Alberta</td>
<td>1,627,874</td>
<td>4,067,175</td>
<td>150%</td>
</tr>
<tr>
<td>British Columbia</td>
<td>2,184,621</td>
<td>4,648,055</td>
<td>113%</td>
</tr>
<tr>
<td>Ontario</td>
<td>7,703,106</td>
<td>13,448,494</td>
<td>75%</td>
</tr>
<tr>
<td>Quebec</td>
<td>6,027,764</td>
<td>8,164,361</td>
<td>35%</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>State</th>
<th>1970 Pop.</th>
<th>2016 Pop.</th>
<th>Growth</th>
</tr>
</thead>
<tbody>
<tr>
<td>Arizona</td>
<td>1,770,900</td>
<td>6,931,071</td>
<td>291%</td>
</tr>
<tr>
<td>Florida</td>
<td>6,789,443</td>
<td>20,612,439</td>
<td>204%</td>
</tr>
<tr>
<td>Colorado</td>
<td>2,207,259</td>
<td>5,540,545</td>
<td>151%</td>
</tr>
<tr>
<td>Texas</td>
<td>11,196,730</td>
<td>27,862,596</td>
<td>149%</td>
</tr>
<tr>
<td>Georgia</td>
<td>4,589,575</td>
<td>10,310,371</td>
<td>125%</td>
</tr>
<tr>
<td>Washington</td>
<td>3,409,169</td>
<td>7,288,000</td>
<td>114%</td>
</tr>
<tr>
<td>North Carolina</td>
<td>5,082,059</td>
<td>10,146,788</td>
<td>100%</td>
</tr>
<tr>
<td>California</td>
<td>19,953,134</td>
<td>39,250,017</td>
<td>97%</td>
</tr>
<tr>
<td>Oregon</td>
<td>2,091,385</td>
<td>4,093,465</td>
<td>96%</td>
</tr>
<tr>
<td>South Carolina</td>
<td>2,590,516</td>
<td>4,961,119</td>
<td>92%</td>
</tr>
<tr>
<td>Virginia</td>
<td>4,648,494</td>
<td>8,411,808</td>
<td>81%</td>
</tr>
<tr>
<td>Tennessee</td>
<td>3,923,687</td>
<td>6,651,194</td>
<td>70%</td>
</tr>
<tr>
<td>Maryland</td>
<td>3,922,399</td>
<td>6,016,447</td>
<td>53%</td>
</tr>
<tr>
<td>Minnesota</td>
<td>3,804,971</td>
<td>5,519,952</td>
<td>45%</td>
</tr>
<tr>
<td>Alabama</td>
<td>3,444,165</td>
<td>4,863,300</td>
<td>41%</td>
</tr>
<tr>
<td>Kentucky</td>
<td>3,218,706</td>
<td>4,436,974</td>
<td>38%</td>
</tr>
<tr>
<td>Wisconsin</td>
<td>4,417,731</td>
<td>5,778,708</td>
<td>31%</td>
</tr>
<tr>
<td>Missouri</td>
<td>4,676,501</td>
<td>6,093,000</td>
<td>30%</td>
</tr>
<tr>
<td>Louisiana</td>
<td>3,641,306</td>
<td>4,681,666</td>
<td>29%</td>
</tr>
<tr>
<td>Indiana</td>
<td>5,193,669</td>
<td>6,633,053</td>
<td>28%</td>
</tr>
<tr>
<td>New Jersey</td>
<td>7,168,169</td>
<td>8,944,469</td>
<td>25%</td>
</tr>
<tr>
<td>Massachusetts</td>
<td>5,689,170</td>
<td>6,811,779</td>
<td>20%</td>
</tr>
<tr>
<td>Illinois</td>
<td>11,113,976</td>
<td>12,801,539</td>
<td>15%</td>
</tr>
<tr>
<td>Michigan</td>
<td>8,875,083</td>
<td>9,928,300</td>
<td>12%</td>
</tr>
<tr>
<td>Ohio</td>
<td>10,652,017</td>
<td>11,614,373</td>
<td>9%</td>
</tr>
<tr>
<td>Pennsylvania</td>
<td>11,793,909</td>
<td>12,784,227</td>
<td>8%</td>
</tr>
<tr>
<td>New York</td>
<td>18,236,962</td>
<td>19,745,289</td>
<td>8%</td>
</tr>
</tbody>
</table>
<p>Among the states with population over 10 million, there is a clear clustering
with Florida, Texas, Georgia, North Carolina, and California (all in the
south or west coast) at the top, and
Illinois, Ohio, Pennsylvania, and New York (all in the midwest or northeast) at
the bottom.</p>
Counting solutions to equations2019-04-08T00:00:00+00:00https://andrew222651.com/2019/04/08/equations<p>There are \(\binom{n-1}{m-1}\) integer compositions of \(n\) with \(m\)
parts.
Complex polynomials of degree \(n\) have \(n\) zeros, counting multiplicity.
Where else do we count solutions to equations?
Our criteria are that the equations must be parametrized, and that for each
parameter value there is a finite solution set.
Some examples are given.</p>
<p><br />
<em>Integer solutions in a ball:</em></p>
<p><a href="https://www.jstor.org/stable/2414232">Theorem</a>:
If \(f\) is a polynomial over \(\mathbb{Z}\) in \(n\) variables, let</p>
\[N(f, B) = |\{\mathbf{x} \in \mathbb{Z}^n: f(x_1, \ldots, x_n) = 0,
\max_i |x_i| \leq B \}|.\]
<p>If \(f\) is a singular homogeneous polynomial over \(\mathbb{Z}\) of degree
\(d\) in \(n > (d-1)2^d\) variables, then
\(N(f, B) \sim c_f B^{n-d}, B \to \infty\), under some technical
conditions.</p>
<p><br />
<em>Diophantine equations:</em></p>
<p><a href="https://arxiv.org/abs/1807.10810">Theorem</a>:
Say \(f\) is a polynomial of degree \(d\) over \(\mathbb{Z}_p\) where
\(GCD(p,d) = 1\). If \(N(f)\) is the number of solutions
\(\mathbf{x} \in \mathbb{Z}_p^n\) to \(f(\mathbf{x}) = 0\), then
\(N(f) = p^{n-1} + O(p^{n/2}), p \to \infty\), assuming a non-singularity
condition.</p>
<p><br />
<em>Non-negative integer solutions to linear equations:</em></p>
<p>E.g.
\(\{ (x,y,z) : 3x + 5y + 17z \leq \lambda, x \geq 0, y \geq 0, z \geq 0 \}\).</p>
<p><a href="http://mathworld.wolfram.com/EhrhartPolynomial.html">Theorem</a>:
Let
\(\Delta(\lambda) = \{ \mathbf{x} \in \mathbb{Z}^n: M \mathbf{x} \leq
\lambda\mathbf{b} \}\).
Then \(|\Delta(\lambda)|\) is
a polynomial in \(\lambda\) of degree \(n\).</p>
<p>Note that wlog \(\mathbf{b}\) takes possible values \(-1,0,1\).
If not, multiply \(b_i\) and \([M]_{i,*}\) by \(\textrm{lcm}(\mathbf{b})/b_i\)
and set \(\lambda' = \lambda / \textrm{lcm}(\mathbf{b})\).</p>
<p>If \(\mathbf{b}\) takes possible values \(-1,0,1\),
we may take the difference \(\Delta(\lambda)
\setminus \Delta(\lambda -1)\) to get solutions to an equality.</p>
<p><img src="https://andrew222651.com/public/images/lattice.png" alt="image" />
(Above: Example solution sets for different values of \(\lambda\).)</p>
<p><br />
<em>Locally restricted words over finite groups:</em></p>
<p><a href="https://arxiv.org/abs/1811.10461">Theorem</a>:
If \(G\) is a finite group and \(x_1, \ldots, x_m \in G\), let \(N(m, a)\) be
the number of solutions to
\(x_1 \cdots x_m = a\)
such that \((x_1, \ldots, x_m)\) satisfies a local restriction.
Then under some conditions, as \(m \to \infty\) we have \(N(m, a) \sim
N(m, e)\), where \(e \in G\) is the identity element.</p>
Quote: Bertrand Russell's anecdote2019-02-05T00:00:00+00:00https://andrew222651.com/2019/02/05/quote-russell<blockquote>
<p>To return to my grandmother’s family … the … sister, Lady Charlotte
Portal was … apt to express herself unfortunately.
On one occasion when she had to order a cab for three people, she thought a
hansom would be too small and a four-wheeler too large, so she told the
footman to fetch a three-wheeled cab.
On another occasion, the footman, whose name was George, was seeing her off
at the station when she was on her way to the Continent.
Thinking that she might have to write to him about some household matter she
suddenly remembered that she did not know his surname.
Just after the train had started she put her head out of the window and called
out, ‘George, George, what’s your name?’
‘George, My Lady’, came the answer. By that time he was out of earshot.</p>
<p>-Bertrand Russell, “Autobiography”</p>
</blockquote>
Advice for students2019-01-16T00:00:00+00:00https://andrew222651.com/2019/01/16/advice-students<script>
// only run if we're on the post's dedicated page (hack)
if (window.location.pathname.indexOf("advice-students") > -1) {
window.location.replace("https://github.com/amacfie/public_notes/wiki/Advice-for-students");
}
</script>
<ul>
<li>Be challenged. Seek material at the level and pace appropriate for you.
Learn with people who aren’t all dumber than you.</li>
<li>If you’re motivated to learn or build something, do it. If you’re not
motivated, don’t force yourself.</li>
<li>Don’t be afraid of the unknown. Just because a topic is advanced or
is in an unfamiliar field doesn’t mean it’s difficult to learn; go for it.</li>
<li>Some things can’t be learned from textbooks because the textbooks don’t
exist, e.g. decision theory.</li>
<li>Some concepts take a while to really absorb, possibly years. In the words
of John von Neumann,
“Young man, in mathematics you don’t understand things. You just get used to
them.”</li>
<li>There’s nothing wrong with funding your studies by borrowing against future
earnings as long as you’re not overpaying for your education. Internships
are great too.</li>
<li>Don’t do undergrad if you don’t need to. Subjects like mathematics, computer
science, and economics can be learned conveniently and effectively
using books, videos, and other resources from the internet.
If you need an academic credential, write a paper with a professor in your
city and get it published.
The mentorship will be valuable and with just publications and reference
letters applying to grad school is an option (see e.g.
<a href="https://en.wikipedia.org/w/index.php?title=Vladimir_Voevodsky&oldid=1078337374#Early_life_and_education">link</a>).</li>
<li>If you formally take a course, ideally learn the material by yourself
beforehand (paradoxical as that sounds).</li>
<li>Don’t go to a bad grad school. At the graduate level, low-status schools
have poor funding, low research activity, and few students.</li>
<li>Get a thesis supervisor who is unambiguously an expert in the field.</li>
<li>Attend economics seminars because they’re a blast.</li>
</ul>
<!--
* The world doesn't need more junk research, focus on quality rather than
quantity.
* Sometimes in research it's more productive to explore without a particular
target. Open problems can be daunting. Start with something tractable,
even if it's pretty random. Once you've made progress, you might see how
your approach can be extended.
-->
Qs2018-07-16T00:00:00+00:00https://andrew222651.com/2018/07/16/qs<p>(These are some questions, which may well be already addressed in the
literature, I haven’t checked.)</p>
<script>
// only run if we're on the post's dedicated page
if (window.location.pathname.indexOf("qs") > -1) {
window.location.replace("https://github.com/amacfie/public_notes/wiki/Hansonian-Questions");
}
</script>
<!--
Should job candidates place [conditional
bets](http://mason.gmu.edu/~rhanson/dumpceo.html) on how productive they’d be
if hired?
-->
<!--
Should other people who know the candidate or company be able to bet too?
redundant
-->
<!-- answered
https://www.overcomingbias.com/2022/05/new-hire-prediction-markets.html
-->
<p>Should someone start selling insurance against online mob victimization and
other “life/career ruining” reputation attacks?</p>
<p>How should a country transition to <a href="http://mason.gmu.edu/~rhanson/futarchy.html">futarchy</a>,
if it starts with a highly corrupt government?</p>
<p>Should children be able to sue parents if they divorce, give birth out of
wedlock, etc?</p>
<p>People have a tendency, perhaps intentional, to claim that policies themselves,
rather than the policies’ apparent goals, are axiomatic good things.
Is this a vulnerability in futarchy, since then they would be voted on as
values rather than bet on as beliefs?</p>
<p>Why is betting psychologically safe in finance (that is, trading) but dangerous
in casinos? As binary options are prone to fraud according to Wikipedia, what
does that say about the viability of
<a href="https://en.wikipedia.org/wiki/Prediction_market">prediction markets</a>?</p>
<p>How should we apply and evaluate informal models, e.g. the broad theories of
Carl Jung and
<a href="https://www.wolframscience.com/nks/">Stephen Wolfram</a>?
Does abstract/vague/informal/subjective/qualitative imply unfalsifiable or is
there a way of formalizing the informal?</p>
<p>If the health system is
<a href="http://www.overcomingbias.com/2007/05/rand_health_ins.html">inefficient</a>, how
much of the problem is caused by doctors (and their guilds)? Similarly for
legal system and lawyers.</p>
<p>Why are “theories” helpful in science but “ideologies” unhelpful in politics?</p>
<p>Was Julian Simon right? What is the real relationship between population and
GDP in the short and long run?</p>
<p>People from Bertrand Russell to Tyler Cowen and Peter Thiel have remarked on
increasing societal risk/change aversion and the concomitant stagnation – what
is best to reboot “dynamism”: Move to a different country? Reform the current
one? Seasteading? Form more-dynamic enclave within the country? Work in
digital/crypto realm? Is this a <a href="http://www.overcomingbias.com/2010/10/fear-made-farmers.html">farmer-forager</a>
issue? Can foragers be dynamic?</p>
<p>If <a href="https://www.primalpoly.com/the-mating-mind">certain mental traits</a> are
Zahavian signals that indicate computational resources, does this, pace Miller,
predict asymmetry in cognitive abilities between the sex that signals and the
sex that chooses, assuming \(P \neq NP\)?</p>
<p>In Japan, finding a defendant not guilty is culturally frowned upon, and
the consequences of this are predictably bizarre; see
<a href="https://www.youtube.com/watch?v=IRn4xzaugbk">link1</a>,
<a href="https://en.wikipedia.org/wiki/I_Just_Didn't_Do_It">link2</a>.
Is there a hidden rationality here?</p>
<!-- see comments
The fact that classical utilitarianism and preference utilitarianism are
different is an example of how my preferences for you might not be the same as
your preferences for yourself.
How much of regulatory paternalism is explained by the government having
different preferences about citizens’ lives than the citizens do,
relative to [other theories](http://www.overcomingbias.com/2019/08/explaining-paternalism.html)?
[Robert Wiblin discussed in [2012](http://www.overcomingbias.com/2012/10/paternalism-can-be-kind-just-not-to-present-you.html).]
-->
<p>If tastes in physical beauty
<a href="https://www.nytimes.com/1977/10/23/archives/when-fat-was-in-fashion-abundant-flesh-was-a-thing-of-beauty-to.html">change</a>
a lot, to what extent can they be explained by sexual selection?</p>
<p>Why are criminals so disliked?</p>
<p>In schools, why do teachers but not students get comfortable office chairs?</p>
<p>Why does Microsoft employ developers in Seattle when <a href="https://www.reddit.com/r/cscareerquestions/comments/7ssl0l/why_are_there_such_big_salary_discrepancies/dt79wry/?utm_source=reddit&utm_medium=web2x&context=3">they could get them
for half price</a>
in Vancouver?</p>
<p>What’s the modern appeal of live music? Is there a more interesting way that
musicians can perform live than playing rehearsed songs?</p>
<p>Why do people get temporary disabilities (diseases) but not temporary
superpowers?</p>
Quote: Knuth on Dijkstra2018-06-20T00:00:00+00:00https://andrew222651.com/2018/06/20/quote-knuth-dijkstra<blockquote>
<p>One of the pleasures I’ve had over the years is to play four-hands piano music
with Edsger. … When we’re playing a Haydn waltz the thing I had to get used
to was that Edsger doesn’t count one-two-three, one-two-three it’s always
zero-one-two, zero-one-two.</p>
<p>-Don Knuth, 2000</p>
</blockquote>
Link: Generated poetry2018-05-28T00:00:00+00:00https://andrew222651.com/2018/05/28/auto-poetry<p><a href="https://www.reddit.com/user/haikubot-1911/comments/?sort=top">https://www.reddit.com/user/haikubot-1911/comments/?sort=top</a></p>
<hr />
<p>Procedure:
Search for groups of sentences that form haikus.
Syllable counting can be done with <a href="http://pyphen.org/">pyphen</a>.
Format the haiku-valid text into three lines and do capitalization.
Post the poems and measure quality with votes.</p>
Three excellent lecture videos2018-05-25T00:00:00+00:00https://andrew222651.com/2018/05/25/three-lectures<p>1. Aakar Patel -
<a href="https://www.youtube.com/watch?v=0W2GZwtGcX4">English and its influence on our national priorities</a>
(2016).
This examines the peculiar phenomenon in India where the popular media, due to
the distribution of languages spoken and very high advertising revenues, ends
up being skewed towards elites and their issues, systematically biasing the
propagation of news information.</p>
<p>2. David Starkey -
<a href="https://www.youtube.com/watch?v=5LAaD5kmHg0">When I hear the word ‘art’, I reach for my gun</a>
(2017).
Starkey gives a sweeping account of the history of art, in a broad sense,
and illuminates the phenomena of Dadaism and modern art.</p>
<p>3. Edsger Dijkstra @
<a href="https://www.youtube.com/watch?v=3nHv5MUw7Nc">Joint International Seminar on the Teaching of Computing Science</a>
(1992).
Dijkstra presents an algorithmic problem and walks through a solution based
on ideas in <em>A Discipline of Programming</em>, where formal semantics are used to
guide the search for an algorithm.</p>
Quote: Maud Ray Kent2018-04-07T00:00:00+00:00https://andrew222651.com/2018/04/07/quote-maud-ray<p>Paraphrasing from Macrae’s biography of J. von Neumann:</p>
<blockquote>
<p>In 1940 the Germans captured Denmark plus Niels Bohr. A cable arrived
in Britain from Otto Frisch’s aunt in Sweden saying, “Met Niels and [wife]
Margarethe recently. Both well but unhappy about events. Please inform Cockroft
[British nuclear scientist] and Maud Ray Kent”.</p>
<p>Who or what was “Maud Ray Kent”? The British decided it was an anagram for
“radyum taken” and thus gave warning that the Germans were moving fast to
develop an A-bomb. Perhaps that was the reason the Nazis had occupied Bohr’s
Copenhagen, and were capturing Norway and its heavy water too? Someone
suggested that Maud might actually stand for “Military Application: Uranium
Disintegration”. The British nuclear program commenced under the name “MAUD
Committee”, as a tribute to the brilliant anagram.</p>
<p>Down in her home in Kent, Miss Maud Ray, the English governess to Bohr’s
children, remained uncontacted because nobody had heard of her.</p>
</blockquote>
Classic probability puzzles and their solutions2018-01-17T00:00:00+00:00https://andrew222651.com/2018/01/17/probability-pitfalls<h3 id="envelope-paradox">Envelope paradox</h3>
<p><strong>Problem:</strong>
You are given two blank envelopes which each contain money.
One envelope contains twice as much as the other.
You may choose an envelope and keep the money it contains.
After choosing, you have the option to switch for the other envelope.
Should you switch?</p>
<!--more-->
<p><strong>Pitfall:</strong>
Clearly there is no reason to switch (or not to switch) since the envelopes are
blank and at no point do you learn anything new about their contents.
However, the following argument seems to show you actually should switch.
Let \(X\) be the amount of money in the envelope chosen originally.
The other envelope contains an amount of either \(2X\) or \(X/2\), each with
probability \(1/2\).
Thus the expected value of the other envelope is</p>
\[\frac{1}{2} (2X) + \frac{1}{2} (X/2) = \frac{5}{4} X > X.\]
<p>…so you should switch?</p>
<p><strong>Solution:</strong>
Whenever things get tricky, it’s best to be as formal and methodical as
possible.
What do we actually mean by \(X\)?
We’re taking \(X\) to be the amount of money in the envelope we choose
originally.
That means \(X\) is a random variable whose value depends on the random choice
of the original envelope.
It’s true that \(X\) is equally likely to be either the smaller or larger amount.
Let these be \(x\) and \(2x\).
Then we need to find the expected value of the other envelope.
Let the value of the other envelope be \(X'\).
Implicitly we are finding the expected value of \(X'\) by conditioning
on the value of \(X\).
There are two possibilities, and in each case we know what we get:</p>
\[\begin{align*}
\mathbb{E}(X') &= \mathbb{P}(X' > X)\mathbb{E}(X' | X' > X)
+ \mathbb{P}(X' < X)\mathbb{E}(X' | X' < X) \\
&= \frac{1}{2}\mathbb{E}(X' | X' > X)
+ \frac{1}{2}\mathbb{E}(X' | X' < X)
\end{align*}\]
<p>Note that depending on whether \(X' > X\) or \(X' < X\) the value of \(X\) is
different.
If \(X' > X\), then \(X = x\) and \(X' = 2X = 2x\), and if \(X' < X\), then \(X = 2x\)
and \(X' = X/2 = x\).
So,</p>
\[\begin{align*}
\frac{1}{2}\mathbb{E}(X' | X' > X) + \frac{1}{2}\mathbb{E}(X' | X' < X)
&= \frac{1}{2}\mathbb{E}(X' | X'=2X) \\
&\qquad {} + \frac{1}{2}\mathbb{E}(X' | X'=X/2) \\
&= \frac{1}{2}2x + \frac{1}{2}x \\
&= \frac{3}{2}x \\
&= \mathbb{E}(X).
\end{align*}\]
<p>Comparing with the pitfall solution, the key difference is that we cannot
use the same symbol \(X\) in two cases if our assumption about the value of
\(X\) is different in each case.
Tricky!</p>
<h3 id="monty-hall-problem">Monty Hall problem</h3>
<p><strong>Problem:</strong>
In the TV game show Let’s Make a Deal, you get to win a prize by opening
one of three doors.
Behind one door is a car and behind the others are goats.
You pick a door, say #1, and without opening #1 the host Monty Hall
intentionally shows you that a goat is behind another door, say
#3, and gives you the chance to change to #2.
Should you change doors?</p>
<p><strong>Pitfall:</strong>
You have the choice between two doors, one with a car, the other with a goat.
Reasoning by symmetry, the probability of each configuration is \(1/2\), so there
is no point switching.</p>
<p><strong>Solution:</strong>
It’s true that we know there are two possible configurations at this point,
but that’s not all we know.
We also know that #2 was <em>not</em> a door chosen by Monty as a door with a goat.
This gives us an extra clue that #2 might have the car.</p>
<p>Again, the best way to deal with tricky problems is being as rigorous
as possible.
Let’s explicitly set up a probability model as follows.
Let \(C \in \{1,2,3\}\) be the random variable for the door with the car.
Let \(S_3\) be the event that Monty chooses #3 to show a goat.
Then we can write down the following conditional probabilities.</p>
\[\begin{align*}
\mathbb{P}(S_3|C=1)&=\frac{1}{2} \\
\mathbb{P}(S_3|C=2)&=1 \\
\mathbb{P}(S_3|C=3)&=0.
\end{align*}\]
<p>These are all we need to compute \(\mathbb{P}(C=2|S_3)\), which is the probability
of winning if we switch.
Using Bayes’s rule, we have</p>
\[\begin{align*}
\mathbb{P}(C=2|S_3)
&= \frac{\mathbb{P}(S_3|C=2)\mathbb{P}(C=2)}{\mathbb{P}(S_3)} \\
&=\frac{\mathbb{P}(S_3|C=2)\mathbb{P}(C=2)}{
\sum_{i=1}^3 \mathbb{P}(S_3|C=i)\mathbb{P}(C=i)} \\
&=\frac{\mathbb{P}(S_3|C=2)}{
\mathbb{P}(S_3|C=1)+\mathbb{P}(S_3|C=2)+\mathbb{P}(S_3|C=3)} \\
&=\frac{1}{\frac12+1+0} =\frac23 > \frac{1}{2}.
\end{align*}\]
<h3 id="feminist-bank-teller-question">Feminist bank teller question</h3>
<p><strong>Problem:</strong>
Linda is 31 years old, single, outspoken, and very bright.
She majored in philosophy.
As a student, she was deeply concerned with issues of discrimination and
social justice, and also participated in anti-nuclear demonstrations.</p>
<p>Which is more probable?</p>
<ol>
<li>Linda is a bank teller.</li>
<li>Linda is a bank teller and is active in the feminist movement.</li>
</ol>
<p><strong>Pitfall:</strong>
The description makes it very plausible that Linda is active in the feminist
movement, therefore #2 is more likely than #1.</p>
<p><strong>Solution:</strong>
This is an example of the conjunction fallacy.
Let \(B\) be the event Linda is a bank teller, and let \(F\) be the event that
Linda is active in the feminist movement.
Then we can immediately say \(\mathbb{P}(B) \geq \mathbb{P}(F \cap B)\)
since \(F \cap B \subseteq B\).
And so #2 cannot be more probable than #1.</p>
<p>In general, if more details are added, we cannot become more confident in a
claim.
This goes against a bias we have to consider situations more plausible if
they are specific and vivid.
For more on cognitive biases and heuristics, see
<a href="https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow"><em>Thinking, Fast and Slow</em></a>,
but also see
<a href="https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-train-wreck-how-priming-research-went-of-the-rails/comment-page-1/#comment-1454">the mistakes in that book caused by cognitive biases and heuristics</a>.</p>
<h3 id="base-rate-neglect">Base rate neglect</h3>
<p><strong>Problem:</strong>
A certain disease affects 1 in 1000 people.
A medical diagnostic test for the disease has 95% accuracy, i.e.
95% of the time it gives the correct diagnosis (whether you’re diseased
or not).
<!-- you could have a test that's right 95% of the time for a random person
but always says you have the disease when you have it. so we need the
parenthetical -->
Suppose you take the test and it reads positive, what is the probability
that you have the disease?</p>
<p><strong>Pitfall:</strong>
Since the diagnostic has 95% accuracy, then in my case I can conclude that
the probability I have the disease is 95%.</p>
<p><strong>Solution:</strong>
Let \(D\) be the event I have the disease, and let \(P\) be the event I receive
a positive diagnosis.
We seek \(\mathbb{P}(D|P)\).
Bayes’s rule gives</p>
\[\begin{align*}
\mathbb{P}(D|P) &= \frac{\mathbb{P}(P|D)\mathbb{P}(D)}{\mathbb{P}(P)} \\
&= \frac{\mathbb{P}(P|D)\mathbb{P}(D)}{\mathbb{P}(P|D)\mathbb{P}(D)
+ \mathbb{P}(P|\neg D)\mathbb{P}(\neg D)} \\
&= \frac{(0.95)(0.001)}{(0.95)(0.001)
+ (0.05)(0.999)} \\
&= 0.019 = 1.9\%.
\end{align*}\]
<p>So the probability I have the disease is actually very small, even though
I got a positive test.
This is another case of not using all the information we have.
The 95% accuracy needs to be combined with the very low probability that
anyone has the disease.
Problems of this type have been given to doctors and medical students who
often fail to find the solution.</p>
<h3 id="birthday-paradox">Birthday paradox</h3>
<p><strong>Problem:</strong>
In a group of 23 people, what is the probability that at least 2 of them
have the same birthday?</p>
<p><strong>Pitfall:</strong>
The probability must be small since there are only 23 people and 365 possible
birthdays.</p>
<p><strong>Solution:</strong>
Let \(N\) be the event no 2 people have the same birthday.
The total number of assignments from people to birthdays is
\(365^{23}\).
The total number of assignments from people to birthdays with no repeats
is \(365 \cdot 364 \cdot \,\cdots\, \cdot 343\) since
there are 365 possibilities for the first person’s birthday, which leaves
364 possibilities for the next, and so on.
But</p>
\[\mathbb{P}(N) = \frac{365^{23}}{365 \cdot 364 \cdot\, \cdots\, \cdot 343} = 0.492703.\]
<p>Then the probability that 2 people share a birthday is \(1-0.492703=0.507297\).
So the probability isn’t small; in fact it’s greater than \(1/2\)!
It’s true that any given pair of people are unlikely to share a birthday,
but as the size of the group grows, the probability that all pairs do not
share a birthday becomes small.
There are \(\binom{23}{2} = 253\) different pairs of people in a group of 23.</p>
What's the probability of the Riemann Hypothesis?2017-10-10T00:00:00+00:00https://andrew222651.com/2017/10/10/probability-riemann-hypothesis<p>Usually when we talk about probabilities, we have certain given information,
which takes the form of a \(\sigma\)-algebra of possible events, and there is
also a probability function that assigns values to each event.
The rationality of a probability function is judged based on the relationships
between events.
For example if \(A \subseteq B\) then we must have \(P(A) \leq P(B)\).
But as long as these relationships are satisfied (giving a proper probability
measure), the probabilities could be anything.
As such, we do not judge subjective probabilities based on whether they’re
actually accurate or not, just whether they are consistent with each other.</p>
<p>Now, imagine if information isn’t the limiting factor in our uncertainty,
but rather it’s our lack of mathematical knowledge.
A statement like the Riemann Hypothesis (RH) is unknown even though it is
entirely determined by the axiom system we use, leaving aside issues of
completeness.
Here there’s no given \(\sigma\)-algebra and in fact the relationships
between RH and other statements may themselves be difficult to determine.</p>
<p>A more realistic view is that we have limited computational resources, we want
to solve an intractible problem, and we’ll settle for the best approximation we
can get.
Thus a probability function is seen as a kind of approximation algorithm.
With this algorithmic language, however, we aren’t able to give a very good
answer for single propositions like RH.
If RH is the entire set of inputs, the optimal approximation is the exact
truth value, because it takes a trivial amount of computational resources to
output the constant 1 or 0.
If the set of inputs is infinite, then the particular input corresponding to
RH makes no difference in an asymptotic analysis.
For more elaboration on this theme, see <a href="https://arxiv.org/pdf/1708.09032.pdf">this
paper</a>.</p>
<p>In traditional Bayesianism there is a seemingly ineradicable source of
subjectivity from the choice of prefix Turing machine used to define
Solomonoff’s prior.
Any one input can be assigned a wide range of probabilities.
Perhaps we are left with an analogous but different kind of subjectivity for
mathematical probabilities.</p>
<p><img src="https://andrew222651.com/public/images/presentation.jpg" alt="Andrew Critch" />
(Above: Andrew Critch thinking about this in Berkeley.)</p>
The Fundamental Theorem of Asset Pricing is Bayesianism2017-08-17T00:00:00+00:00https://andrew222651.com/2017/08/17/ftap-bayes<p>If uncertainties encode bet preferences as represented by probabilities,
Bayesianism is a collection of Dutch book arguments proving that probabilities
must be consistent with each other (defining a probability measure) to be
rational.
Weisberg has <a href="https://web.archive.org/web/20181217223439/https://jonathanweisberg.org/pdf/VarietiesvF.pdf">an excellent paper</a> that explains the details.
On the other hand, the Fundamental Theorem of Asset Pricing proves that for
prices to be arbitrage-free, they must be <a href="https://en.wikipedia.org/w/index.php?title=Conditional_expectation&oldid=858130631#Conditional_expectation_with_respect_to_a_sub-%CF%83-algebra">conditional expectations</a>.
<!--more-->
Details on the relevant results are found in
“The Mathematics of Arbitrage”, by Delbaen and Schachermayer.
Having a consistent probability function has been <a href="https://web.archive.org/web/20181030055011/http://www.princeton.edu/~osherson/papers/nuEll11.pdf">shown</a> to be
equivalent to minimizing a proper scoring rule.
And conditional expectations have been <a href="https://www.semanticscholar.org/paper/On-the-optimality-of-conditional-expectation-as-a-Banerjee-Guo/56df317c39f685e75b340c8538b699088ffa918c">shown</a> to minimize
Bregman divergences.
Et cetera.
The correspondance between these theories is alluded to by <a href="https://web.archive.org/web/20180319141859/https://faculty.fuqua.duke.edu/~rnau/definettiwasright.pdf">Nau</a>.</p>
Data from Canadian 700MHz and 2500MHz spectrum auctions2017-06-28T00:00:00+00:00https://andrew222651.com/2017/06/28/canada-spectrum-auctions<p>The 700MHz (2014) and 2500MHz (2015) spectrum auctions generated revenues of
5,270,636,002 CAD from 302 licenses and 755,371,001 CAD from 97 licenses.
Both
auctions used a combinatorial clock auction (CCA) format involving an
ascending clock phase followed by a sealed-bid supplementary stage where
bids could be made on packages of products.
Final prices were determined
using Vickrey pricing with a core-adjustment. An activity
rule was used which required bidders to make bids or lose eligibility to
bid in later clock rounds, along with a revealed preference rule which
allows the eligibility limit to be exceeded as long as consistency
checks are satisfied. For full details on the auction formats see the
official documentation
(<a href="http://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf10573.html#p4">700MHz rules</a>,
<a href="http://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf08697.html">700MHz additional details</a>,
<a href="http://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf10726.html#p4">2500MHz rules</a>);
and the record of bids placed is <a href="http://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf11085.html">here</a>
for 700MHz and
<a href="http://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf11170.html">here</a>
for 2500MHz.</p>
<p><em>Bid consistency</em></p>
<p>The revealed preference rule prevents some inconsistent behavior but not all.
By “truthful”, we mean bids that are true indications of subjective value,
and by “consistent” we mean bids that are indications of some fixed set of
valuations, possibly not the bidder’s actual valuations.</p>
<p>The following table gives the values of Afriat’s critical cost
efficiency index (CCEI) for the 700MHz auction. Recall
that for a CCEI value \(x\), if \(x < 1\) there is at least some
intransitivity in preferences (i.e. inconsistent bidding) and \(1-x\) can be
interpreted as the fraction of expenditure wasted making inefficient
choices (see <a href="http://eml.berkeley.edu/~kariv/CFGK_III_A3.pdf">this</a>
by S. Kariv for more).</p>
<table>
<thead>
<tr>
<th>Bidder</th>
<th>CCEI (clock rounds)</th>
<th>CCEI (clock and supp. rounds)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Bell</td>
<td>0.930</td>
<td>0.417</td>
</tr>
<tr>
<td>Bragg</td>
<td>0.880</td>
<td>0.420</td>
</tr>
<tr>
<td>Feenix</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>MTS</td>
<td>0.996</td>
<td>0.627</td>
</tr>
<tr>
<td>Novus</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>Rogers</td>
<td>0.998</td>
<td>0.742</td>
</tr>
<tr>
<td>SaskTel</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>TBayTel</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>Telus</td>
<td>0.970</td>
<td>0.488</td>
</tr>
<tr>
<td>Videotron</td>
<td>0.879</td>
<td>0.560</td>
</tr>
</tbody>
</table>
<p><a href="https://link.springer.com/article/10.1007/s10726-015-9431-0">Kroemer et al.</a>
conclude for the 700MHz auction, “the
numbers suggest that bidders deviated substantially from straightforward
bidding” in the clock rounds. But “it is not unreasonable to believe
that bidders tried to bid up to their true valuation in the
supplementary stage” because of higher bid amounts compared to the clock
rounds.</p>
<p>The next table gives CCEI values for the 2500MHz
auction. We extend the definition of CCEI to apply to supplementary bids
as in Kroemer’s paper.</p>
<table>
<thead>
<tr>
<th>Bidder</th>
<th>CCEI (clock rounds)</th>
<th>CCEI (clock and supp. rounds)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Bell</td>
<td>0.913</td>
<td>0.712</td>
</tr>
<tr>
<td>Bragg</td>
<td>0.920</td>
<td>0.530</td>
</tr>
<tr>
<td>Corridor</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>MTS</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>Rogers</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>SSi Micro</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>TBayTel</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>Telus</td>
<td>0.997</td>
<td>0.996</td>
</tr>
<tr>
<td>Videotron</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>WIND</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>Xplornet</td>
<td>1</td>
<td>0.578</td>
</tr>
</tbody>
</table>
<p>Kroemer et al. (Sec. 5.2) also point out that the
total number of bids submitted in the 700MHz auction was much smaller
than the number of possible bids, which probably indicates untruthful
bidding since an omitted
package must have valuation less than or equal to its (low) opening price.
The same observation holds for the 2500MHz auction. More exactly, the
auction formats enforced a limit on the number of packages bidders were
allowed to submit, which was in the hundreds, and bidders generally did
not reach the limit.</p>
<p>Ideally, we would determine whether the bids made are consistent with a
non-truthful strategy incorporating gaming and/or coordination.
The papers
Janssen and Karamychev -
<a href="https://homepage.univie.ac.at/maarten.janssen/auctions/Raising%20Rivals'%20Cost%20in%20Multi-unit%20Auctions-V1%2011V-Ref.pdf">“Raising Rivals’ Cost in Multi-unit Auctions”</a>
and
Janssen and Kasberger -
<a href="https://homepage.univie.ac.at/maarten.janssen/working%20papers/CCAclock0911.pdf">“On the Clock of the Combinatorial Auction”</a>
derive Bayesian Nash equilibria under gaming preferences and conclude that
GARP is not violated in equilibrium gaming strategies.
We note that the assumptions in these game models do not include all features
of the CCAs under consideration e.g.
discrete products, many bidders, public aggregate excess
demand, revealed preference rule, initial EP limit,
supplementary package limit, 50% mid-auction deposits.</p>
<p><em>Bids, budgets, and final prices</em></p>
<p>Bidders may have a notion of a budget – the maximum they are willing to
spend. But how should this correspond to the maximum they should bid? In
general, bidders may end up paying the exact amount of their highest
bid, but looking at the data we see bid prices and final prices can be
very different in practice.
The following tables show figures from both auctions that illustrate this
difference.
All prices are given in CAD.</p>
<p>700MHz auction: highest bid placed.
Average ratio: 0.192. Max ratio: 0.766.</p>
<table>
<thead>
<tr>
<th>Bidder</th>
<th>Max bid (\(M\))</th>
<th>Allocation stage final price (\(p\))</th>
<th>Ratio (\(p/M\))</th>
<th>Final clock bid</th>
</tr>
</thead>
<tbody>
<tr>
<td>Bell</td>
<td>3,999,999,000</td>
<td>565,705,517</td>
<td>0.141</td>
<td>1,366,867,000</td>
</tr>
<tr>
<td>Bragg</td>
<td>141,894,000</td>
<td>20,298,000</td>
<td>0.143</td>
<td>38,814,000</td>
</tr>
<tr>
<td>Feenix</td>
<td>60,650,000</td>
<td>284,000</td>
<td>0.005</td>
<td>346,000</td>
</tr>
<tr>
<td>MTS</td>
<td>73,067,000</td>
<td>8,772,072</td>
<td>0.120</td>
<td>10,853,000</td>
</tr>
<tr>
<td>Novus</td>
<td>112,359,000</td>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>Rogers</td>
<td>4,299,949,000</td>
<td>3,291,738,000</td>
<td>0.766</td>
<td>3,931,268,000</td>
</tr>
<tr>
<td>Sasktel</td>
<td>75,000,000</td>
<td>7,556,929</td>
<td>0.101</td>
<td>11,927,000</td>
</tr>
<tr>
<td>TbayTel</td>
<td>7,683,000</td>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>Telus</td>
<td>3,750,000,000</td>
<td>1,142,953,484</td>
<td>0.305</td>
<td>1,313,035,000</td>
</tr>
<tr>
<td>Videotron</td>
<td>677,524,000</td>
<td>233,328,000</td>
<td>0.344</td>
<td>468,530,000</td>
</tr>
</tbody>
</table>
<p><br />
700MHz auction: (highest) bid placed on package eventually won.
Average ratio: 0.447. Max ratio: 0.766.</p>
<table>
<thead>
<tr>
<th>Bidder</th>
<th>Max bid on won package (\(W\))</th>
<th>Allocation stage final price (\(p\))</th>
<th>Ratio (\(p/W\))</th>
<th>Allocation stage Vickrey price</th>
</tr>
</thead>
<tbody>
<tr>
<td>Bell</td>
<td>2,583,868,000</td>
<td>565,705,517</td>
<td>0.219</td>
<td>565,705,000</td>
</tr>
<tr>
<td>Bragg</td>
<td>51,000,000</td>
<td>20,298,000</td>
<td>0.398</td>
<td>20,298,000</td>
</tr>
<tr>
<td>Feenix</td>
<td>425,000</td>
<td>284,000</td>
<td>0.668</td>
<td>284,000</td>
</tr>
<tr>
<td>MTS</td>
<td>40,000,000</td>
<td>8,772,072</td>
<td>0.219</td>
<td>3,198,000</td>
</tr>
<tr>
<td>Novus</td>
<td>N/A</td>
<td>0</td>
<td>N/A</td>
<td>0</td>
</tr>
<tr>
<td>Rogers</td>
<td>4,299,949,000</td>
<td>3,291,738,000</td>
<td>0.766</td>
<td>3,291,738,000</td>
</tr>
<tr>
<td>Sasktel</td>
<td>62,400,000</td>
<td>7,556,929</td>
<td>0.121</td>
<td>2,755,000</td>
</tr>
<tr>
<td>TbayTel</td>
<td>N/A</td>
<td>0</td>
<td>N/A</td>
<td>0</td>
</tr>
<tr>
<td>Telus</td>
<td>1,607,300,000</td>
<td>1,142,953,484</td>
<td>0.711</td>
<td>1,142,953,000</td>
</tr>
<tr>
<td>Videotron</td>
<td>490,000,000</td>
<td>233,328,000</td>
<td>0.476</td>
<td>233,328,000</td>
</tr>
</tbody>
</table>
<p><br />
2500MHz auction: highest bid placed.
Average ratio: 0.135. Max ratio: 0.277.</p>
<table>
<thead>
<tr>
<th>Bidder</th>
<th>Max bid (\(M\))</th>
<th>Allocation stage final price (\(p\))</th>
<th>Ratio (\(p/M\))</th>
<th>Final clock bid</th>
</tr>
</thead>
<tbody>
<tr>
<td>Bell</td>
<td>542,746,000</td>
<td>28,730,000</td>
<td>0.053</td>
<td>76,214,000</td>
</tr>
<tr>
<td>Bragg</td>
<td>35,935,000</td>
<td>4,821,021</td>
<td>0.134</td>
<td>12,091,000</td>
</tr>
<tr>
<td>Corridor</td>
<td>9,300,000</td>
<td>2,299,000</td>
<td>0.247</td>
<td>N/A</td>
</tr>
<tr>
<td>MTS</td>
<td>13,609,000</td>
<td>2,242,000</td>
<td>0.165</td>
<td>2,609,000</td>
</tr>
<tr>
<td>Rogers</td>
<td>304,109,000</td>
<td>24,049,546</td>
<td>0.079</td>
<td>52,343,000</td>
</tr>
<tr>
<td>SSi Micro</td>
<td>851,000</td>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>TBayTel</td>
<td>12,001,000</td>
<td>1,731,000</td>
<td>0.144</td>
<td>1,731,000</td>
</tr>
<tr>
<td>Telus</td>
<td>1,771,723,000</td>
<td>478,819,000</td>
<td>0.270</td>
<td>1,038,472,000</td>
</tr>
<tr>
<td>Videotron</td>
<td>749,128,000</td>
<td>66,552,980</td>
<td>0.089</td>
<td>231,851,000</td>
</tr>
<tr>
<td>WIND</td>
<td>22,609,000</td>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>Xplornet</td>
<td>91,974,000</td>
<td>25,472,454</td>
<td>0.277</td>
<td>57,839,000</td>
</tr>
</tbody>
</table>
<p><br />
2500MHz auction: (highest) bid placed on package eventually won.
Average ratio: 0.235. Max ratio: 0.410.</p>
<table>
<thead>
<tr>
<th>Bidder</th>
<th>Max bid on won package (\(W\))</th>
<th>Allocation stage final price (\(p\))</th>
<th>Ratio (\(p/W\))</th>
<th>Allocation stage Vickrey price</th>
</tr>
</thead>
<tbody>
<tr>
<td>Bell</td>
<td>536,563,000</td>
<td>28,730,000</td>
<td>0.054</td>
<td>28,730,000</td>
</tr>
<tr>
<td>Bragg</td>
<td>19,000,000</td>
<td>4,821,021</td>
<td>0.254</td>
<td>3,536,000</td>
</tr>
<tr>
<td>Corridor</td>
<td>6,440,000</td>
<td>2,299,000</td>
<td>0.357</td>
<td>2,299,000</td>
</tr>
<tr>
<td>MTS</td>
<td>11,000,000</td>
<td>2,242,000</td>
<td>0.204</td>
<td>2,242,000</td>
</tr>
<tr>
<td>Rogers</td>
<td>OR</td>
<td>24,049,546</td>
<td>N/A</td>
<td>21,252,000</td>
</tr>
<tr>
<td>SSi Micro</td>
<td>N/A</td>
<td>0</td>
<td>N/A</td>
<td>0</td>
</tr>
<tr>
<td>TBayTel</td>
<td>12,001,000</td>
<td>1,731,000</td>
<td>0.144</td>
<td>1,731,000</td>
</tr>
<tr>
<td>Telus</td>
<td>1,771,723,000</td>
<td>478,819,000</td>
<td>0.270</td>
<td>478,819,000</td>
</tr>
<tr>
<td>Videotron</td>
<td>358,477,000</td>
<td>66,552,980</td>
<td>0.186</td>
<td>61,092,000</td>
</tr>
<tr>
<td>WIND</td>
<td>N/A</td>
<td>0</td>
<td>N/A</td>
<td>0</td>
</tr>
<tr>
<td>Xplornet</td>
<td>62,200,000</td>
<td>25,472,454</td>
<td>0.410</td>
<td>22,917,000</td>
</tr>
</tbody>
</table>
<p>Across both auctions, we see that bidders paid an average of 16% of
their maximum bid placed, where each bidder is equally weighted.</p>
<p><em>Misc. notes</em></p>
<p>Researchers design approximation algorithms for winner and price
determination in auctions (in e.g. <a href="https://projecteuclid.org/download/pdf_1/euclid.im/1089229504">this paper</a>)
because exact
optimization can be intractable as the number of bids grows, at least in
the worst case. However, in these recent auction instances, theoretical
intractability did not present a problem because the solution was
computable in a small amount of time.
The 2500MHz allocation stage involved 2,239 bids and a
GLPK-powered solver finds the winners and final prices in a couple
minutes on a standard computer.
Simulations involving well over 30,000 random bids still take a feasible amount
of time.</p>
<p>In the 2500MHz auction, there were 4 pairs of package submissions where
the lower-priced package had strictly higher quantities of products.
In this case the lower-priced package is superfluous.
This table shows the packages, submitted by Bell.</p>
<table>
<thead>
<tr>
<th> </th>
<th>Price of larger package (CAD)</th>
<th>Price of smaller package (CAD)</th>
<th>Number of products difference</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>535,917,000</td>
<td>536,214,000</td>
<td>3</td>
</tr>
<tr>
<td>2</td>
<td>536,628,000</td>
<td>536,645,000</td>
<td>3</td>
</tr>
<tr>
<td>3</td>
<td>536,401,000</td>
<td>536,545,000</td>
<td>3</td>
</tr>
<tr>
<td>4</td>
<td>536,434,000</td>
<td>536,563,000</td>
<td>3</td>
</tr>
</tbody>
</table>
<p><em>Acknowledgement</em>: Thanks to Z. Gao for pointers.</p>
Link of the day: Journal of Craptology2017-06-21T00:00:00+00:00https://andrew222651.com/2017/06/21/journal-profile-1<p><a href="http://www.anagram.com/jcrap/">http://www.anagram.com/jcrap/</a></p>
<hr />
<p>The prestigious Journal of Craptology is an open-access publication in the
area of cryptology.</p>
U of Toronto study space info2017-06-09T00:00:00+00:00https://andrew222651.com/2017/06/09/U-of-T-study-spaces<p>(a partial list)</p>
<p><img src="https://andrew222651.com/public/images/cannon.jpg" alt="Cannons" /></p>
<p>(Above: U of T cannons pointed at Ryerson)</p>
<h2 id="libraries"><a href="https://onesearch.library.utoronto.ca/visit">Libraries</a></h2>
<h3 id="robarts-library"><a href="https://www.google.com/maps/place/John+P.+Robarts+Research+Library/@43.664486,-79.4018733,17z/data=!3m1!4b1!4m5!3m4!1s0x882b34be85038f67:0x7cd088dfbb2ef4a9!8m2!3d43.664486!4d-79.39969">Robarts Library</a></h3>
<table>
<tbody>
<tr>
<td>noise</td>
<td>low</td>
</tr>
<tr>
<td>power plugs</td>
<td>yes</td>
</tr>
<tr>
<td>network access</td>
<td>good</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>busy</td>
</tr>
<tr>
<td>typical hours</td>
<td>8:30-11</td>
</tr>
</tbody>
</table>
<h3 id="richard-charles-lee-canada-hong-kong-library">Richard Charles Lee Canada-Hong Kong Library</h3>
<p><img src="https://andrew222651.com/public/images/lee.jpg" alt="table" /></p>
<table>
<tbody>
<tr>
<td>noise</td>
<td>low</td>
</tr>
<tr>
<td>power plugs</td>
<td>yes</td>
</tr>
<tr>
<td>network access</td>
<td>good</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>modest</td>
</tr>
<tr>
<td>typical hours</td>
<td>10-7</td>
</tr>
</tbody>
</table>
<h3 id="gerstein-science-information-centre"><a href="https://www.google.com/maps/place/Gerstein+Science+Information+Centre/@43.662195,-79.3961483,17z/data=!3m2!4b1!5s0x882b34b84d333c2f:0xc015f6f99a229640!4m5!3m4!1s0x882b34b854ad5761:0xd4cee6c2ece27846!8m2!3d43.662195!4d-79.393965">Gerstein Science Information Centre</a></h3>
<!-- visits: many -->
<table>
<tbody>
<tr>
<td>noise</td>
<td>low</td>
</tr>
<tr>
<td>power plugs</td>
<td>some</td>
</tr>
<tr>
<td>network access</td>
<td>poor</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>busy</td>
</tr>
<tr>
<td>typical hours</td>
<td>8:30-11</td>
</tr>
</tbody>
</table>
<h3 id="sanford-fleming-library"><a href="https://www.google.com/maps/place/Engineering+%26+Computer+Science+Library/@43.660171,-79.3970524,17z/data=!3m1!4b1!4m5!3m4!1s0x882b34b884728431:0xcdd91c9b9294d5d0!8m2!3d43.660171!4d-79.3948691">Sanford Fleming Library</a></h3>
<!-- visits: many -->
<table>
<tbody>
<tr>
<td>noise</td>
<td>medium</td>
</tr>
<tr>
<td>power plugs</td>
<td>yes</td>
</tr>
<tr>
<td>network access</td>
<td>poor</td>
</tr>
<tr>
<td>ergonomics</td>
<td>ok</td>
</tr>
<tr>
<td>busyness</td>
<td>busy</td>
</tr>
<tr>
<td>typical hours</td>
<td>9-6</td>
</tr>
</tbody>
</table>
<h3 id="mathematics-library"><a href="https://www.google.com/maps/place/Mathematical+Sciences+Library/@43.6596175,-79.3992823,17z/data=!3m1!4b1!4m5!3m4!1s0x882b34c753034693:0x4002c713c3ce1ea4!8m2!3d43.6596175!4d-79.397099">Mathematics Library</a></h3>
<!-- visits: 1 -->
<p><img src="https://andrew222651.com/public/images/mathlib_chair.jpg" alt="chair" />
<img src="/public/images/mathlib_desk.jpg" alt="desk" /></p>
<table>
<tbody>
<tr>
<td>noise</td>
<td>low</td>
</tr>
<tr>
<td>power plugs</td>
<td>some</td>
</tr>
<tr>
<td>network access</td>
<td>ok</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>busy</td>
</tr>
<tr>
<td>typical hours</td>
<td>9-5</td>
</tr>
</tbody>
</table>
<h3 id="ej-pratt-library"><a href="https://www.google.com/maps/place/E.J.+Pratt+Library/@43.6663191,-79.3934942,17z/data=!4m13!1m7!3m6!1s0x882b34b083e38c11:0x5f73ddb613ec263c!2sE.J.+Pratt+Library,+71+Queen's+Park+Cres+E,+Toronto,+ON+M5S+1K7!3b1!8m2!3d43.6663127!4d-79.3913932!3m4!1s0x882b34b903a1d297:0xf5dde8b4a7aefff!8m2!3d43.666318!4d-79.391286">EJ Pratt Library</a></h3>
<!-- visits: 1 -->
<table>
<tbody>
<tr>
<td>noise</td>
<td>low</td>
</tr>
<tr>
<td>power plugs</td>
<td>some</td>
</tr>
<tr>
<td>network access</td>
<td>?</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>busy</td>
</tr>
<tr>
<td>typical hours</td>
<td>8:30-11:45</td>
</tr>
</tbody>
</table>
<h3 id="architecture-library-eberhard-zeidler-library"><a href="https://www.google.com/maps/place/EBERHARD+ZEIDLER+LIBRARY/@43.659568,-79.4032728,17z/data=!3m1!4b1!4m5!3m4!1s0x882b34c06ea620a7:0xba0239111d9ffc55!8m2!3d43.659568!4d-79.4010895">Architecture Library (Eberhard Zeidler Library)</a></h3>
<!-- visits: 0.5 -->
<table>
<tbody>
<tr>
<td>noise</td>
<td>low</td>
</tr>
<tr>
<td>power plugs</td>
<td>yes</td>
</tr>
<tr>
<td>network access</td>
<td>?</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>busy</td>
</tr>
<tr>
<td>typical hours</td>
<td>9-9</td>
</tr>
</tbody>
</table>
<h3 id="kelly-library"><a href="https://goo.gl/maps/w32oRVh8oz82">Kelly Library</a></h3>
<!--
![desk](https://kek.gg/i/7HY-Dy.jpg)
-->
<table>
<tbody>
<tr>
<td>noise</td>
<td>?</td>
</tr>
<tr>
<td>power plugs</td>
<td>yes</td>
</tr>
<tr>
<td>network access</td>
<td>good</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>?</td>
</tr>
<tr>
<td>typical hours</td>
<td>8:30-11:30</td>
</tr>
</tbody>
</table>
<h3 id="the-inforum-faculty-of-information"><a href="https://goo.gl/maps/LSfZzsMeRk52">The Inforum (Faculty of Information)</a></h3>
<!--
![view from above](https://kek.gg/i/372rmh.jpeg)
![table](https://kek.gg/i/dWxyf.jpg)
-->
<table>
<tbody>
<tr>
<td>noise</td>
<td>?</td>
</tr>
<tr>
<td>power plugs</td>
<td>yes</td>
</tr>
<tr>
<td>network access</td>
<td>good</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>?</td>
</tr>
<tr>
<td>typical hours</td>
<td>9:30-9:30</td>
</tr>
</tbody>
</table>
<h3 id="east-asian-library">East Asian Library</h3>
<!--
![carrel](https://kek.gg/i/4FP_n3.jpg)
-->
<table>
<tbody>
<tr>
<td>noise</td>
<td>?</td>
</tr>
<tr>
<td>power plugs</td>
<td>no</td>
</tr>
<tr>
<td>network access</td>
<td>?</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>?</td>
</tr>
<tr>
<td>typical hours</td>
<td>9-7:30</td>
</tr>
</tbody>
</table>
<h3 id="centre-for-reformation-and-renaissance-studies"><a href="https://goo.gl/maps/d41EMPZ9Sst">Centre for Reformation and Renaissance Studies</a></h3>
<table>
<tbody>
<tr>
<td>noise</td>
<td>?</td>
</tr>
<tr>
<td>power plugs</td>
<td>no</td>
</tr>
<tr>
<td>network access</td>
<td>?</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>?</td>
</tr>
<tr>
<td>typical hours</td>
<td>9-5</td>
</tr>
</tbody>
</table>
<h2 id="local-toronto-public-library-branches"><a href="https://www.torontopubliclibrary.ca/hours-locations/">Local Toronto public library branches</a></h2>
<h3 id="toronto-reference-library"><a href="https://www.google.com/maps/place/Toronto+Public+Library+-+Toronto+Reference+Library/@43.6719053,-79.3888698,17z/data=!3m1!4b1!4m5!3m4!1s0x882b34ae90d62fe1:0x50fbc31e127f9314!8m2!3d43.6719053!4d-79.3866865">Toronto Reference Library</a></h3>
<p><img src="/public/images/trl.jpg" alt="table" /></p>
<table>
<tbody>
<tr>
<td>noise</td>
<td>low</td>
</tr>
<tr>
<td>power plugs</td>
<td>some</td>
</tr>
<tr>
<td>network access</td>
<td>yes</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>medium</td>
</tr>
<tr>
<td>typical hours</td>
<td>9-8:30</td>
</tr>
</tbody>
</table>
<h3 id="lillian-h-smith-public-library"><a href="https://www.google.com/maps/place/Toronto+Public+Library,+Lillian+H.+Smith+Library/@43.6579592,-79.4006306,17z/data=!3m1!4b1!4m5!3m4!1s0x882b34c0de903de1:0x822dc3ec86e6c66a!8m2!3d43.6579592!4d-79.3984473">Lillian H. Smith public library</a></h3>
<!-- visits: 1 -->
<table>
<tbody>
<tr>
<td>noise</td>
<td>low</td>
</tr>
<tr>
<td>power plugs</td>
<td>some</td>
</tr>
<tr>
<td>network access</td>
<td>good</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>medium</td>
</tr>
<tr>
<td>typical hours</td>
<td>9-8:30</td>
</tr>
</tbody>
</table>
<h2 id="other-areas-with-seating">Other areas with seating</h2>
<h3 id="tables-in-bahen-centre"><a href="https://www.google.com/maps/place/Bahen+Centre+for+Information+Technology/@43.6596426,-79.3998509,17z/data=!3m1!4b1!4m5!3m4!1s0x882b34c75165c957:0x6459384147b4b67b!8m2!3d43.6596426!4d-79.3976676">Tables in Bahen Centre</a></h3>
<!-- visits: many -->
<table>
<tbody>
<tr>
<td>noise</td>
<td>moderate</td>
</tr>
<tr>
<td>power plugs</td>
<td>yes</td>
</tr>
<tr>
<td>network access</td>
<td>ok</td>
</tr>
<tr>
<td>ergonomics</td>
<td>poor</td>
</tr>
<tr>
<td>busyness</td>
<td>busy</td>
</tr>
<tr>
<td>typical hours</td>
<td>?</td>
</tr>
</tbody>
</table>
Markov of Chain: Automating Weird Sun tweets2017-05-27T00:00:00+00:00https://andrew222651.com/2017/05/27/markov-of-chain-weird-sun-twitter<p>Let’s use python to train a Markov chain generator using all the tweets from a
certain list of users, say
<a href="https://twitter.com/Grognor/lists/weird-sun-twitter/members">this one</a>.
We’ll use the following libraries.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from functional import seq
import markovify
import re
import tweepy
import unidecode
</code></pre></div></div>
<p>To use the Twitter API, we need to authenticate ourselves.
Register for your personal keys at <a href="https://apps.twitter.com/">https://apps.twitter.com/</a> and then create
a <code class="language-plaintext highlighter-rouge">config.json</code> file that looks like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"consumer_key": "...",
"consumer_secret": "...",
"access_key": "...",
"access_secret": "..."
}
</code></pre></div></div>
<p>Now we can initialize the Twitter API provided by tweepy.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>config = seq.json('config.json').dict()
auth = tweepy.OAuthHandler(
config['consumer_key'], config['consumer_secret'])
auth.set_access_token(config['access_key'], config['access_secret'])
api = tweepy.API(auth)
</code></pre></div></div>
<p>First we write the following function (based on <a href="https://gist.github.com/yanofsky/5436496">this gist</a>)
which returns the most recent tweets of a given user.
The API limits us to at most 3240 tweets per user.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def get_user_tweets(screen_name):
alltweets = []
# 200 is the maximum allowed count
# 'extended' means return full unabridged tweet contents
new_tweets = api.user_timeline(screen_name=screen_name, count=200,
tweet_mode='extended')
alltweets.extend(new_tweets)
# save the id of the oldest tweet less one
oldest_id = alltweets[-1].id - 1
# keep grabbing tweets until there are no tweets left to grab
while len(new_tweets) > 0:
# since we're grabbing 200 at a time, we use `max_id` to
# ask for a certain range of tweets
new_tweets = api.user_timeline(
screen_name = screen_name, count=200,
tweet_mode='extended', max_id=oldest_id)
alltweets.extend(new_tweets)
#update the id of the oldest tweet less one
oldest_id = alltweets[-1].id - 1
print("...{} tweets downloaded so far".format(len(alltweets)))
# put each tweet on a single line
tweet_texts = [re.sub(r'\s*\n+\s*', ' ', tweet.full_text)
for tweet in alltweets]
return tweet_texts
</code></pre></div></div>
<p>The other interaction with Twitter we need to perform is get all users in a
list.
We’ll write a function that fetches the usernames and calls <code class="language-plaintext highlighter-rouge">get_user_tweets</code>
on each:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def get_list_tweets(screen_name, list_name):
'''
params: `screen_name` is the username of the owner of the list,
`list_name` is the name of the list found in the URL
'''
# get list of all users in list
user_names = []
for user in tweepy.Cursor(
api.list_members,
screen_name,
list_name).items():
user_names.append(user.screen_name)
# for each user, get their tweets
list_tweets = []
for user_name in user_names:
list_tweets += get_user_tweets(user_name)
print('Found {1} tweets from @{2}.'
.format(len(list_tweets), user_name))
return list_tweets
</code></pre></div></div>
<p>Let’s run <code class="language-plaintext highlighter-rouge">get_list_tweets</code> and save the output to a file.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tweets = get_list_tweets('Grognor', 'weird-sun-twitter')
with open('data/tweetdump.txt', 'w') as f:
f.write('\n'.join(tweets))
</code></pre></div></div>
<p>With all of the raw data saved, we’re done with the Twitter API and we can
process the data and auto-generate tweets offline.
Assuming the file <code class="language-plaintext highlighter-rouge">tweetdump.txt</code> has a set of tweets, one per line, we
load them as a list of strings <code class="language-plaintext highlighter-rouge">tweets</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tweets = open('data/tweetdump.txt').readlines()
</code></pre></div></div>
<p>Some processing needs to be done in order to get high quality text from the
tweets.
The next function <code class="language-plaintext highlighter-rouge">process_tweet</code> is called on each one.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def process_tweet(tweet):
# convert to ASCII
tweet = unidecode.unidecode(tweet)
# remove URLs
tweet = re.sub(r'http\S+', '', tweet)
# remove mentions
tweet = re.sub(r'@\S+', '', tweet)
tweet = tweet.strip()
# append terminal punctuation if absent
if len(tweet) > 0:
last_char = tweet[-1]
if last_char not in '.!?':
tweet += '.'
return tweet
processed_tweets = [ process_tweet(tweet) for tweet in tweets ]
</code></pre></div></div>
<p>And we remove any tweets that aren’t useful.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def is_excluded(tweet):
ex = False
# no RTs
ex = ex or bool(re.match(r'^RT', tweet))
# remove whitespace-only tweets
ex = ex or bool(re.match(r'^\s*$', tweet))
return ex
good_tweets = [ tweet for tweet in processed_tweets
if not is_excluded(tweet) ]
</code></pre></div></div>
<p>We save the fully processed tweets for easy access later.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>with open('data/processed_tweets.txt', 'w') as f:
f.write('\n'.join(good_tweets))
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">markovify</code> library lets us train, and generate from, a Markov chain very
easily.
Just load the training text and set a state size.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>text = open('data/processed_tweets.txt').read()
text_model = markovify.Text(text, state_size=3)
for x in range(5):
print('* ' + text_model.make_short_sentence(140))
</code></pre></div></div>
<p>Some favorites:</p>
<ul>
<li>“It is no coincidence we call them gods because we suppose they are trying to
convince Robin Hanson.”</li>
<li>“Tell anyone who does not produce Scott Alexander.”</li>
<li>“Weird sun is a costly signal of the ability to remember sources of
information, not just the study of complex manifolds.”</li>
<li>“If you read The Hobbit backwards, it’s about a layer of radioactive ash that
develops the capacity to become larger.”</li>
<li>“When you read a physical book, you get a dust speck in the eye.”</li>
<li>“We all continuously scream about how the people in it are breaking the
awkward silence.”</li>
<li>“People are important, but so are lexicographic preferences.”</li>
<li>“You don’t need an expert Bayesian Epistemologist to ensure it’s not a markov
chain.”</li>
</ul>
Building a shell with JavaScript2017-05-20T00:00:00+00:00https://andrew222651.com/2017/05/20/building-JS-shell<p><a href="https://github.com/shelljs/shelljs">ShellJS</a> is a JS library that provides
functions like <code class="language-plaintext highlighter-rouge">cd()</code> and <code class="language-plaintext highlighter-rouge">ls()</code> which you can use to write Node scripts
instead of bash scripts.
That’s great for scripts, but what about an interactive shell?
Well, we could just run the Node repl and import ShellJS:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ node
> require('shelljs/global');
{}
> pwd()
{ [String: '/tmp']
stdout: '/tmp',
stderr: null,
code: 0,
cat: [Function: bound ],
exec: [Function: bound ],
grep: [Function: bound ],
head: [Function: bound ],
sed: [Function: bound ],
sort: [Function: bound ],
tail: [Function: bound ],
to: [Function: bound ],
toEnd: [Function: bound ],
uniq: [Function: bound ] }
</code></pre></div></div>
<p>Hmm, that’s a little verbose, and we might want to avoid manually importing
ShellJS.
We also might want more features than the Node repl offers, such as vi
keybindings.</p>
<p>We can get vi keybindings with rlwrap, but then tab completion goes away.
The solution is given in <a href="http://stackoverflow.com/a/43677273/371334">this SO answer</a>.
First we need to install an rlwrap filter that negotiates tab-completion with a
Node repl.
The filter file can be found at that link, where it’s called <code class="language-plaintext highlighter-rouge">node_complete</code>.
Put <code class="language-plaintext highlighter-rouge">node_complete</code> in <code class="language-plaintext highlighter-rouge">$RLWRAP_FILTERDIR</code>, which should be the folder on your
system containing the <code class="language-plaintext highlighter-rouge">RlwrapFilter.pm</code> Perl module.
For me it’s <code class="language-plaintext highlighter-rouge">/usr/share/rlwrap/filters</code>.</p>
<p>Now rlwrap is ready to negotiate tab completion, but the Node repl isn’t.
We’ll have to actually write our own Node repl, which is easy because the
<code class="language-plaintext highlighter-rouge">repl</code> module gives us all the tools we need.
We’ll create a file called, say, <code class="language-plaintext highlighter-rouge">myrepl.js</code>, the contents of which are also
given in <a href="http://stackoverflow.com/a/43677273/371334">the SO answer</a>, only 9
lines.
This script starts a repl with a hook to negotiate tab completion with rlwrap.
If <code class="language-plaintext highlighter-rouge">myrepl.js</code> is in <code class="language-plaintext highlighter-rouge">~/bin</code>, now we can run</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ rlwrap -z node_complete -e '' -c ~/bin/myrepl.js
</code></pre></div></div>
<p>and have both JS tab completion and rlwrap features, such as vi keybindings
if that’s what we’ve <a href="https://unix.stackexchange.com/q/22740/15513">configured</a>.
Let’s create a file called <code class="language-plaintext highlighter-rouge">mysh</code> with the following contents:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/usr/bin/env bash
rlwrap -z node_complete -e '' -c ~/bin/myrepl.js
</code></pre></div></div>
<p>Assuming <code class="language-plaintext highlighter-rouge">~/bin</code> is in our path variable, we can put <code class="language-plaintext highlighter-rouge">mysh</code> there and launch
our shell anywhere by just running <code class="language-plaintext highlighter-rouge">mysh</code>.
So far so good but we wanted to automatically import ShellJS.
In <code class="language-plaintext highlighter-rouge">myrepl.js</code>, add the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>var shell = require('shelljs');
Object.assign(myrepl.context, shell);
</code></pre></div></div>
<p>Those two lines add all the ShellJS functions to the JS global object inside
the repl.
We have:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mysh
> pwd()
{ [String: '/tmp']
stdout: '/tmp',
stderr: null,
code: 0,
cat: [Function: bound ],
exec: [Function: bound ],
grep: [Function: bound ],
head: [Function: bound ],
sed: [Function: bound ],
sort: [Function: bound ],
tail: [Function: bound ],
to: [Function: bound ],
toEnd: [Function: bound ],
uniq: [Function: bound ] }
</code></pre></div></div>
<p>Progress.
Now, how do we clean up this output?
The <code class="language-plaintext highlighter-rouge">repl</code> module allows us to define a custom <code class="language-plaintext highlighter-rouge">writer</code>.
This is a function which takes the
output of a line of JS and returns a string to represent the output in the
repl.
What we need to do is intercept objects like the one returned by <code class="language-plaintext highlighter-rouge">pwd()</code> above
and only show the <code class="language-plaintext highlighter-rouge">stderr</code> and <code class="language-plaintext highlighter-rouge">stdout</code> properties.
Add the following near the beginning of <code class="language-plaintext highlighter-rouge">myrepl.js</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>var util = require('util');
var myWriter = function(output) {
var isSS = (
output &&
output.hasOwnProperty('stdout') &&
output.hasOwnProperty('stderr'));
if (isSS) {
var stderrPart = output.stderr || '';
var stdoutPart = output.stdout || '';
return stderrPart + stdoutPart;
} else {
return util.inspect(output, null, null, true);
}
};
</code></pre></div></div>
<p>And load this writer by changing</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>var myrepl = require("repl").start({terminal:false});
</code></pre></div></div>
<p>to</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>var myrepl = require("repl").start({
terminal: false,
writer: myWriter});
</code></pre></div></div>
<p>Now we get</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mysh
> pwd()
/tmp
</code></pre></div></div>
<p>Much better.
However, since the <code class="language-plaintext highlighter-rouge">echo</code> function prints its argument to the console <em>and</em>
returns an object with it in the <code class="language-plaintext highlighter-rouge">stdout</code> property, we get this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mysh
> echo('hi')
hi
hi
</code></pre></div></div>
<p>I haven’t solved this issue quite yet although I’d be surprised if there isn’t
a reasonable solution out there.
You can add to <code class="language-plaintext highlighter-rouge">mysh</code> and <code class="language-plaintext highlighter-rouge">myrepl.js</code> to get more features, such as colors,
custom evaluation, custom pretty printing, other pre-loaded libraries, et
cetera.
The sky is the limit.
I added an <code class="language-plaintext highlighter-rouge">inspect</code> function which allows us to see the full ShellJS output
of a command if we really want it.
My complete <code class="language-plaintext highlighter-rouge">myrepl.js</code> file is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/usr/bin/env node
var util = require('util');
var colors = require('colors/safe');
var inspect = function(obj) {
if (obj && typeof obj === 'object') {
obj['__inspect'] = true;
}
return obj;
};
var myWriter = function(output) {
var isSS = (
output &&
output.hasOwnProperty('stdout') &&
output.hasOwnProperty('stderr') &&
!output.hasOwnProperty('__inspect'));
if (isSS) {
var stderrPart = output.stderr || '';
var stdoutPart = output.stdout || '';
return colors.cyan(stderrPart + stdoutPart);
} else {
if (typeof output === 'object') {
delete output['__inspect'];
}
return util.inspect(output, null, null, true);
}
};
// terminal:false disables readline (just like env NODE_NO_READLINE=1):
var myrepl = require("repl").start({
terminal: false,
prompt: colors.green('% '),
ignoreUndefined: true,
useColors: true,
writer: myWriter});
var shell = require('shelljs');
Object.assign(myrepl.context, shell);
myrepl.context['inspect'] = inspect;
// add REPL command rlwrap_complete(prefix) that prints a simple list
// of completions of prefix
myrepl.context['rlwrap_complete'] = function(prefix) {
myrepl.complete(prefix, function(err,data) {
for (x of data[0]) {console.log(x);}
});
};
</code></pre></div></div>
<p>So this is basically what we wanted.
We have a JS repl with convenient ShellJS commands.
We also have vi keybindings, and tab completion for JS and filenames.
It’s very rough around the edges, but it was really simple to make.
GitHub user <code class="language-plaintext highlighter-rouge">streamich</code> built a <a href="https://github.com/streamich/jssh">more advanced form</a>
of this, called <code class="language-plaintext highlighter-rouge">jssh</code> which adds many features but lacks some too.
The bottom line is, if you know JS, you might be surprised at what you can
build.</p>
Modeling aesthetics in mathematics2016-07-05T00:00:00+00:00https://andrew222651.com/2016/07/05/math_beauty<p>What exactly is beautiful math?</p>
<blockquote>
<p>[A]bove all, adepts [of mathematics] find therein delights analogous to those
given by
painting and music. They admire the delicate harmony of numbers and forms; they
marvel when a new discovery opens to them an unexpected perspective; and has
not the joy they thus feel the esthetic character, even though the senses take
no part therein? Only a privileged few are called to enjoy it fully, it is
true, but is not this the case for all the noblest arts?</p>
<p>-Henri Poincaré, <em>The Value of Science</em></p>
</blockquote>
<blockquote>
<p>One expects a mathematical theorem or a mathematical theory not only to
describe and to classify in a simple and elegant way numerous and a priori
disparate special cases. One also expects “elegance” in its “architectural”,
structural makeup. Ease in stating the problem, great difficulty in getting
hold of it and in all attempts at approaching it, then again some very
surprising twist by which the approach, or some part of the approach, becomes
easy, etc. Also, if the deductions are lengthy or complicated, there should be
some simple general principle involved, which “explains” the complications and
detours, reduces the apparent arbitrariness to a few simple guiding
motivations, etc. These criteria are clearly those of any creative art.</p>
<p>-John von Neumann, <em>The Mathematician</em></p>
</blockquote>
<blockquote>
<p>The moral: a good proof is one that makes us wiser.</p>
<p>-Yuri Manin, <em>A Course in Mathematical Logic for Mathematicians</em></p>
</blockquote>
<p>My hypothesis is that generally when people talk about beauty in mathematics
they’re talking about things that teach us something useful for proving new
facts.
For example, proving a difficult but simple theorem is useful because its
difficulty means it may imply other previously difficult theorems, and its
simplicity means it may show up and be used often.
A theorem that establishes a connection between two previously disparate areas
of mathematics is considered beautiful, and
such a connection allows knowledge from one are to be applied to the other,
potentially cracking new problems.
An unexpected proof – “an unexpected perspective” or “surprising twist” –
offers something new to be learned, something that can then be used for other
problems.</p>
Quote of the day: Yuri Gurevich2016-04-29T00:00:00+00:00https://andrew222651.com/2016/04/29/g-quote<blockquote>
<p>I remember, in a geometry class, my teacher wanted to prove the congruence of
two triangles. Let’s take a third triangle, she said, and I asked where do
triangles come from. I worried that there may be no more triangles there. Those
were hard times in Russia and we were accustomed to shortages. She looked at me
for a while and then said: ‘Shut up’.</p>
<p>-<a href="http://research.microsoft.com/en-us/um/people/gurevich/Opera/123.pdf">Platonism, Constructivism, and Computer Proofs vs. Proofs by Hand</a></p>
</blockquote>
Link of the day: Learn the Greek alphabet2016-04-03T00:00:00+00:00https://andrew222651.com/2016/04/03/link-greek-alphabet<p><a href="http://www.cram.com/flashcards/the-greek-alphabet-both-cases-290740">A handy flashcards web app for memorizing all the Greek letters</a></p>
Link of the day: Jon Skeet speaks2016-03-15T00:00:00+00:00https://andrew222651.com/2016/03/15/link-skeet-data-types<p>Jon Skeet on the tricky edge cases that can show up with basic data types
and how they model reality.
<a href="https://www.youtube.com/watch?v=l3nPJ-yK-LU">Back to basics: the mess we’ve made of our fundamental data types</a></p>
Command line Grooveshark post-Grooveshark2015-08-28T00:00:00+00:00https://andrew222651.com/2015/08/28/groove<p>In this post I’ll share a way to get a Grooveshark-like experience with
a Linux command-line application.</p>
<p>Step 1: Build a YouTube playlist with some music you like.</p>
<p>Step 2: Go to your channel, select <em>Playlists</em>, find the one you just made,
and click <em>View full playlist</em>.
Make sure the privacy setting is either public or unlisted.
Copy the playlist ID in the URL (after <code class="language-plaintext highlighter-rouge">list=</code>).
For example, the playlist ID of one of my playlists is
<code class="language-plaintext highlighter-rouge">PLmaRvdyzIrIGkRbl7jJdzEYrRcXe4od9G</code>.</p>
<p>Step 3: Install <code class="language-plaintext highlighter-rouge">mpv</code> and <a href="`https://github.com/mps-youtube/mps-youtube`"><code class="language-plaintext highlighter-rouge">mps-youtube</code></a>.</p>
<p>Step 4: Run</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mpsyt pl <playlist ID>, dump, \*
</code></pre></div></div>
<p>You’re streaming the playlist!
Press <code class="language-plaintext highlighter-rouge"><space></code> to pause/play, <code class="language-plaintext highlighter-rouge"><</code> to play previous track, <code class="language-plaintext highlighter-rouge">></code> to play next
track.</p>
<p>To play the playlist shuffled, run this instead:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mpsyt pl <playlist ID>, dump, shuffle \*
</code></pre></div></div>
<p>I have this alias in my <code class="language-plaintext highlighter-rouge">.zshrc</code> file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>alias playlistName="mpsyt pl PLmaRvdyzIrIGkRbl7jJdzEYrRcXe4od9G, dump, shuffle \*"
</code></pre></div></div>
<p>so I can get music playing with just one command.</p>
<p>If, as in Grooveshark, your YouTube playlists are public, you can open
<code class="language-plaintext highlighter-rouge">mpsyt</code> and run <code class="language-plaintext highlighter-rouge">userpl <YouTube username></code> to see your YouTube playlists
and select one to play.</p>
<p>The good thing is that the whole videos aren’t streamed, just high-quality
audio.</p>
<p>Among other features, <code class="language-plaintext highlighter-rouge">mpsyt</code> allows you to search for YouTube videos and
create local playlists (not connected to a YouTube account), which you can do
if you want to avoid the YouTube web interface completely.</p>
Status of the day2015-04-21T00:00:00+00:00https://andrew222651.com/2015/04/21/szpank<p>Wojciech Szpankowski:</p>
<p><img src="https://andrew222651.com/public/images/szpank_face.jpg" alt="WS" /></p>
<p>From Wojciech Szpankowski’s <a href="http://amzn.com/047124063X">book</a>, drawn by
Philippe Jacquet:</p>
<p><img src="https://andrew222651.com/public/images/szpank_drawing.jpg" alt="drawing" /></p>
<p><a href="https://andrew222651.com/public/images/szpank_finger.jpg">.</a></p>
Quote of the day2015-04-19T00:00:00+00:00https://andrew222651.com/2015/04/19/mill<blockquote>
<p>The first intellectual operation in which I arrived at any proficiency, was
dissecting a bad argument, and finding in what part the fallacy lay; and though
whatever capacity of this sort I attained was due to the fact that it was an
intellectual exercise in which I was most perseveringly drilled by my father,
yet it is also true that the school logic, and the mental habits acquired in
studying it, were among the principal instruments of this drilling. I am
persuaded that nothing, in modern education, tends so much, when properly used,
to form exact thinkers, who attach a precise meaning to words and propositions,
and are not imposed on by vague, loose, or ambiguous terms. The boasted
influence of mathematical studies is nothing to it; for in mathematical
processes, none of the real difficulties of correct ratiocination occur.</p>
<p>-J.S. Mill, <a href="http://oll.libertyfund.org/titles/mill-the-collected-works-of-john-stuart-mill-volume-i-autobiography-and-literary-essays"><em>Autobiography</em></a></p>
</blockquote>
<p>I believe “school logic” is a.k.a. <a href="http://en.wikipedia.org/wiki/Scholasticism">scholastic</a>
logic and is something along the lines of “philosophical” logic and what Mill
covered in his <em>A System of Logic</em>.
Sometimes I found combinatorics problems to require careful thinking in order
to avoid plausible-looking mistakes.
At <a href="http://www.artofproblemsolving.com/wiki/index.php/Combinatorics/Olympiad">Art of Problem Solving</a>,
instructors suggest “counting in two ways”, i.e. using two different counting
strategies and comparing the results.</p>
Quote of the day2015-03-27T00:00:00+00:00https://andrew222651.com/2015/03/27/socphil<blockquote>
<p>Why not experiment with wholesome personal relationships? Some recreational
socialising?
Maybe.
I dabble.
I’m a social primate susceptible to peer
pressure.
No man is an island, etc, even those with a metaphysic evocative of
windowless <a href="http://en.wikipedia.org/wiki/Monadology">monads</a>.
[I fret that the <em>only</em> thing one can ever know is the
intrinsic subjective properties of whichever configuration of matter and energy
one happens to instantiate. <em>cf</em>. <a href="http://www.hedweb.com/lockwood.htm">Lockwood</a>.]
Sadly, my conspecifics are a dangerous, expensive and tainted source of opioid
supply with endogenous habits of their own to feed.
Moreover my inferential realism about so-called
<a href="http://en.wikipedia.org/wiki/Philosophy_of_perception">perception</a> is a deterrent to intimacy: personal relationships become a branch
of speculative metaphysics if each virtual world resembles a solipsistic
island-universe.
Indeed I worry that the closest I come to staying in touch
with the real world is eating bits of it: the epistemology of
<a href="http://www.moodfoods.com/">food</a>, so to
speak.
And germs.
If I meet my end under the wheels of the proverbial London bus, my faith in the
mind-independent world will be posthumously vindicated.
As
it happens, I have just finished reading Simon Critchley’s <em>The Book of Dead</em>
<em>Philosophers</em> (2008); perhaps my best hope of immortality lies, not churning out
philosophical verbiage, but having the equivalent of a tortoise fall on my
head.
This untimely fate exhausts my knowledge of Aeschylus.</p>
<p>-David Pearce, <a href="http://www.hedweb.com/diarydav/2008.html">Diary Update and Idle Musings</a></p>
</blockquote>
Quote of the day2014-10-14T00:00:00+00:00https://andrew222651.com/2014/10/14/crock<p>Programming wisdom:</p>
<blockquote>
<p>Someone wrote to me once suggesting that JSLint should give a warning when a
<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/switch">case</a>
falls through into another case.
He pointed out that this is a very common source of errors, and it is a
difficult error to see in the code.
I answered that that was all true, but that the benefit of compactness
obtained by falling through more than compensating for the chance of error.
The next day, he reported that there was an error in JSLint.
…
I investigated, and it turned out that I had a case that was falling through.
…
I no longer use intentional fall throughs.</p>
<p>-<a href="http://amzn.com/0596517742">Douglas Crockford</a></p>
</blockquote>
Quote of the day2014-08-18T00:00:00+00:00https://andrew222651.com/2014/08/18/cam<blockquote>
<p>This reminds me of a story from the time when Queen Mary and Westfield College,
University of London decided to change its name to Queen Mary, University of
London.
My colleague Wilfrid Hodges was giving a lecture in Germany, and put up his
first slide, giving his name and affiliation as “Wilfrid Hodges, Queen Mary,
University of London”.
Somebody asked, “Is that a joint publication?”</p>
<p>-<a href="http://cameroncounts.wordpress.com/2014/08/08/google-scholar-citations/">Peter Cameron</a></p>
</blockquote>
Link of the day2014-06-29T00:00:00+00:00https://andrew222651.com/2014/06/29/usde<p>An interesting comparison of the cultures of the US and Germany:</p>
<p><a href="http://math-www.uni-paderborn.de/~axel/us-d.html">http://math-www.uni-paderborn.de/~axel/us-d.html</a></p>
<p>When I travel internationally, I wish I could absorb this level of detail about
a culture, but of course it’s impossible without living there for an extended
period of time.</p>
Some resources for learning Mandarin2014-05-26T00:00:00+00:00https://andrew222651.com/2014/05/26/mandarin_resources<p>(This is an old list from when I <a href="http://rationalconspiracy.com/2012/05/29/foreign-languages-are-a-waste-of-time/">was studying it</a>
in around 2009.)</p>
<ul>
<li>This is enough to learn how to pronounce Chinese people’s names: <a href="https://www.youtube.com/watch?v=b9Ayvjy-Dgs">https://www.youtube.com/watch?v=b9Ayvjy-Dgs</a></li>
<li><a href="http://learn.rocketlanguages.com/chinese/premium/">Rocket Chinese</a> Grammar & Culture Series with MegaChinese vocabulary software</li>
<li><a href="http://www.echineselearning.com/newsletter/index.html?ecl=emEEEEEEis61mn04">eChineseLearning newsletter</a></li>
<li>Serge Melnyk’s <a href="http://www.melnyks.com/">lessons</a></li>
<li><a href="http://www.rosettastone.com/learn-chinese">Rosetta Stone software</a></li>
<li>Conversion tools:
<ul>
<li>Characters -> pinyin: <a href="http://translate.google.com">http://translate.google.com</a></li>
<li>Characters <-> english: <a href="http://translate.google.com">http://translate.google.com</a></li>
<li>Pinyin -> sound: <a href="http://www.quickmandarin.com/chinesepinyintable/">http://www.quickmandarin.com/chinesepinyintable/</a></li>
</ul>
</li>
<li>Desktop translation tools:
<ul>
<li><a href="http://www.allvirtualware.com/languages/chinese_translation_software.htm">LEC Translate Chinese</a></li>
<li><a href="http://lingoes.net/">Lingoes</a></li>
</ul>
</li>
<li><a href="http://english.cntv.cn/learnchinese/">CCTV Learn Chinese</a> Videos</li>
<li><a href="http://watchtolearnchinese.com/">WatchtoLearnChinese.com</a> Videos</li>
<li><a href="http://resources.echineselearning.com/funstuff/funstuff-chinese-20.html">http://resources.echineselearning.com/funstuff/funstuff-chinese-20.html</a> Various resources</li>
<li><a href="http://digchinese.com/en/measure-words">http://digchinese.com/en/measure-words</a> A guide to measure words</li>
<li><a href="http://www.nciku.com/conversation">http://www.nciku.com/conversation</a> Conversation texts in chinese, pinyin and english</li>
<li><a href="http://www.wenlin.com/">Wenlin Software for Learning Chinese</a></li>
<li>A list of language learning web apps: <a href="http://translate.google.com/translate?hl=en&sl=auto&tl=en&u=http%3A%2F%2Fwww.cnbeta.com%2Farticles%2F84038.htm">http://translate.google.com/translate?hl=en&sl=auto&tl=en&u=http%3A%2F%2Fwww.cnbeta.com%2Farticles%2F84038.htm</a></li>
<li><a href="http://kidschinesepodcast.com">Kids Chinese Podcast</a></li>
<li><a href="http://amzn.com/0824810686">John DeFrancis - Chinese Language: Fact and Fantasy</a></li>
</ul>
Quote of the day2014-05-25T00:00:00+00:00https://andrew222651.com/2014/05/25/quote<p>From <a href="http://amzn.com/B00CMNTKZ4">Modern Computer Algebra</a>:
“We start by using [Newton iteration] to find a custom-Taylored division
algorithm that is as fast as multiplication…”</p>
LessWrong gems2014-05-22T00:00:00+00:00https://andrew222651.com/2014/05/22/lwgems<p><a href="http://lesswrong.com/lw/csw/son_of_shit_rationalists_say/6pz1">lukeprog ridicules Will_Newsome’s “post-rationality”</a></p>
<p><a href="http://lesswrong.com/r/discussion/lw/aw7/muehlhausergoertzel_dialogue_part_1/62b7">Will_Newsome defends XiXiDu</a></p>
<p><a href="http://lesswrong.com/lw/ol/penguicon_blook/">proposed cover design for book of the sequences</a></p>
<p><a href="http://lesswrong.com/lw/4g/eliezer_yudkowsky_facts/">Eliezer Yudkowsky facts</a></p>
<p><a href="http://lesswrong.com/r/discussion/lw/egt/random_lwparodying_statement_generator/7ena">Random LW-parodying Statement Generator</a></p>
<p><a href="http://lesswrong.com/lw/d2w/cards_against_rationality/">Cards Against Rationality</a></p>
<p><a href="https://andrew222651.com/public/images/bayes_mag.jpg">Bayes: The Exclusive Less Wrong Magazine</a></p>
<p><a href="http://kruel.co/lw/There%20is%20no%20Science%20but%20Bayes%20and%20it%20is%20our%20Method.txt">LessWrongers’ creed</a></p>
<p><a href="http://lesswrong.com/lw/k2s/open_thread_april_16_april_22_2014/atpy">The end is nigh</a></p>
<p><a href="http://lesswrong.com/lw/ec0/dealing_with_metadiscussion_and_the_signal_to/7bi6">Signal to signaling ratio</a></p>
<p><a href="http://www.quickmeme.com/Overcoming-bias-guy/page/1/">Overcoming bias guy</a></p>
<p><a href="http://lesswrong.com/lw/btc/how_can_we_get_more_and_better_lw_contrarians/6e33">Stop mass greenvoting!</a></p>
<p><img src="https://andrew222651.com/public/images/louie.jpg" alt="Louie Helm" />
(Above: Louie Helm.)</p>
Status of the day2014-05-21T00:00:00+00:00https://andrew222651.com/2014/05/21/Status<p>AC in pop culture:
Pitbull fans must exist but it’s impossible to find an explicit example.</p>
Status of the day2014-05-18T00:00:00+00:00https://andrew222651.com/2014/05/18/status<p>Linux tip: Install Helvetica and/or Helvetica Neue on your computer and some
websites will look better</p>
Quote of the day2014-05-15T00:00:00+00:00https://andrew222651.com/2014/05/15/quote<p>From a ratemyprofessors.com review of Mike Newman’s teaching:
‘Clearly LOVES math, makes tons of jokes; ie, someone asks how many questions
on the midterm and he replies “well… it’s an integer number”.’</p>
Link of the day2014-02-27T00:00:00+00:00https://andrew222651.com/2014/02/27/link<p><a href="http://www.psychologytomorrowmagazine.com/jeff-warren-neuroscience-suffering-end/">http://www.psychologytomorrowmagazine.com/jeff-warren-neuroscience-suffering-end/</a></p>
General thoughts on computer security2014-01-07T00:00:00+00:00https://andrew222651.com/2014/01/07/sec<p>The following is a document I wrote in mid 2008.
I haven’t edited it, although I was tempted to since parts of it sound a bit
strange now.
Rather than improve on it, I’ll just clarify what it was attempting to say:
The connections between devices and/or agents have a finite set of security
properties which determine the options available to secure the assets involved.
These properties can be simply classified, providing a way to systematically
explore the space of security configurations.
I felt that the ideas in the document were a good summary of some facts about
computer security.</p>
<hr />
<p>The culmination of my independent studies in computer security involved the
development of a system of detailed original models and axioms to describe
every aspect of the security state of digital systems by generalizing all the
disparate case-by-case knowledge I had compiled. I had heard of no similar
methods or structure from anywhere else because the vast majority of IT
security knowledge is in applied form for various technical domains. Indeed,
when work on this system concluded, I no longer needed to continue learning to
increase my understanding of computer security. While the concepts used apply
to security of all domains, it the models are optimized for computerized
resources. A summary of it follows.</p>
<p>The fundamental principles of security are authentication and authorization. An
asset is secure to the degree that the owner is willing to make it if these
properties are properly configured to function at that degree (or higher) and
understood by its owner to function at that degree (or lower). Security is
needed when access is to the asset is enabled. In the context of digital
security, in which the asset is a digital substance (hardware or software),
access is generally arbitrary usage, reading, and/or writing.</p>
<p>If one is to determine the optimal expenditure of funds on security, one must
first find the optimal system of security, on a per-asset or per-asset-group
basis. The asset involved will have one or more points of access
(security/trust chains, or simply chains). Each chain will have one or more
links, with the distinction between links depending on the granularity of the
analysis. The conceptual chain will “connect” the potential user to the asset
in question (with no regard for physical circumstances). Each link of the chain
involves an authentication/authorization process of which there are three
primary orders.</p>
<p>The first order is preventive, and akin to the wall and gate around a medieval
castle. This order provides access control at the site of the access in full
context of the asset’s execution environment. This is also the order at work in
file-access Access Control Lists (ACLs) in an operating system. The second is
pre-preventive, and akin to a security check before boarding an airplane. This
order is used by an intermediate system before the request arrives at the
destination, but strives to serve the same purpose as first order security. It
may be cheaper to implement, especially if one specialized device can be used
to serve many others. The principal drawback is that it must be aware of the
full system state of the destination device (asset) in order to rule on the
safety of the incoming request. This is often not the case in practice, but the
benefit in economy may outweigh the cost. The other main drawback to second
order security procedures is that even if enough information about the asset is
known at the time of access to make a ruling, there may not be enough to
“sanitize” the incoming request, or provide an effective, safe alternative to
the requested access, and may instead simply drop it; this would be unintended
in some circumstances. The third order, also called policy, involves a time
delay, the need for evidence, the tracking of identity, the requirement for
effective deterrence, a willing judiciary, and a cost-effective punishment
system. The costs involved with the use of this method in the digital world
dictate that its use tends to be primarily as a backup, although there have
been questions about the merits of its existence at all.</p>
<p>Each link in the security/trust chain will either provide access control on one
of the three levels, involve trusting an uncontrolled third party to determine
the security of the access, or do neither. In practical modeling, links of the
final option would likely only be included to illustrate failures of security.</p>
<p>All security systems in use in digital environments can be described using this
method. A web of chains can be used to illustrate trust relationships between
end devices and the role they play in the security chain of others. A potential
attacker would be presumed to use what would be perceived by him to be the
“weakest” chain, or sequence of connections of chains, that leads him to the
asset. Note that what is often derided as “security through obscurity” is often
quite secure indeed. For example, a file on an Internet-connected Web server
that is only meant to be accessible by a specific individual could possibly be
protected in a number of ways. Two of these are storing the file in a
password-protected folder, or putting the file in a hard-to-guess directory.
The latter is considered “security through obscurity”, but requires almost the
exact same security procedures to access as the former (except for the traffic
patterns created, types and permissions of log files stored, etc., none of
which are necessarily risks for a given situation). Ultimately, all chains from
the user to the asset need only one strong link, but combinations of the
different kinds are used in different domains. Possible examples of the system
in practice are numerous, a few follow (with thorough background).</p>
<p>An example of a simplistic security chain model illustrating the typical
protection used against XSS (cross-site scripting) attacks involves three
entities: the server hosting the vulnerable Web page, the attacker, and the
victim user. In a persistent XSS attack, the chain begins with a second order
link in between the attacker and the server. The server will inspect the input
sent to determine whether or not it will violate the user’s presumed (and
usually generalized) security policy. In the case of JavaScript malware, the
server checks to make sure none of the input could be executed (in the [likely]
context of the eventual Web page it will be rendered in) and either encodes it
to prevent this happening, or cancels the request if the check fails. Note that
this process may have to be looped to prevent some encoding bugs, until the
result is safe. The user then requests the Web page that the attacker’s input
is found in (the reason for the visit may be the result of the function of
another chain). Generally, the chain link here will be one of trust between the
user and server: the user trusts that the server (uncontrolled third party)
will not serve anything dangerous. If dynamic content does get served and
attempts to connect to a remote host other than the original server (subject to
a few loopholes and exceptions), the connection will be blocked. This is called
the Same Origin Policy (SOP). This requires the removal of functionality (and
the need for chains of security) for controlling this kind of execution access,
which could potentially be legitimate. However, the check involved in comparing
the server and the remote host is done using domain names. This causes problems
when the SOP (showing second-order characteristics) presumes it knows the way
in which requests are sent in the user’s network, and is known to have
weaknesses. The user’s understanding of his security state is fundamentally to
blame in this case. Note that the integrity of the network all devices are on
and other aspects of the scenario represent the need for additional security
chains, but are ignored as perfectly secure in this example.</p>
<p>The final link in the most common scenario for persistent XSS defense can be
the user’s disallowing of JavaScript altogether on a per-domain basis.
Essentially similar to the SOP, this involves overriding the trust in the
domain from the previous link, and in effect closes access (removes the need
for a security chain altogether, but at the cost of functionality) to
JavaScript-enabled Web sites that are not on the domain-allow list. A
non-persistent XSS would involve the user trusting or not trusting the
attacker’s sent link based on the information available (trust link) or using
security of some other kind. Note that links containing JavaScript in the query
string or post body may be legitimate. The link here would be one of either
possible first order security (if the script affects only client-side
functions) or trust (if the script accesses the server, where the user cannot
monitor). Note also that the protections described here are as fine-grained as
generally regarded to be feasible. A first order security link only on the part
of the client, combined with trust links based on the destination domain of
remote-connecting scripts would enable flexible cross-domain access. The
feasibility of this has been doubted (by Mark Russinovich for one) but by no
means disproved.</p>
<p>Further examples could be included (SQLi, encrypted/steganographed traffic in a
workplace)</p>