WifiCIDR Blog (Posts about math)https://blog.wificidr.net/enFri, 22 Sep 2023 17:30:39 GMTNikola (getnikola.com)http://blogs.law.harvard.edu/tech/rssThe Modular Inversehttps://blog.wificidr.net/posts/the-modular-inverse/Daniel Justice<div class="cell border-box-sizing text_cell rendered" id="cell-id=9b40912a"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="The-Modular-Inverse">The Modular Inverse<a class="anchor-link" href="https://blog.wificidr.net/posts/the-modular-inverse/#The-Modular-Inverse">¶</a></h2><p>This is the third post in my short series on the RSA algorithm.
We met most of the cast of characters used in the algorithm in the <a href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/">last post</a>.
Prime numbers, Euclid's algorithm, Bézout's identity, and of course, one of Euler's identities come together to form the foundation of our work.
The last detail is the modular inverse.
Conceptually, the idea is straightforward, but I feel many articles on RSA don't give it the attention it deserves.</p>
<h3 id="Identities">Identities<a class="anchor-link" href="https://blog.wificidr.net/posts/the-modular-inverse/#Identities">¶</a></h3><p>Many operations paired with sets have an element known as the identity element.
This should be a familiar idea even if you haven't seen the formal definition.</p>
<ul>
<li>0 is the identity of the set $\mathbb{R}$ and the operation of addition. 42 + 0 = 42</li>
<li>1 is the identity of the set $\mathbb{R}$ and the operation of multiplication. 42 * 1 = 42</li>
<li>[] is the identity of list concatenation. [40, 41, 42] + [] = [40, 41, 42]</li>
</ul>
<p>Can you think of other examples?
What about the identity of a 3x3 matrix?</p>
$$\begin{bmatrix}
42 & 99 & 2\\
0 & -35 & 12\\
4 & 2 & 8
\end{bmatrix} \cdot
\begin{bmatrix}
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{bmatrix} =
\begin{bmatrix}
42 & 99 & 2\\
0 & -35 & 12\\
4 & 2 & 8
\end{bmatrix}$$<h3 id="Inverses">Inverses<a class="anchor-link" href="https://blog.wificidr.net/posts/the-modular-inverse/#Inverses">¶</a></h3><p>Informally, and inverse simply "undoes" an operation.
Specifically, we are interested in an element that gives us back the identity.</p>
<ul>
<li>$42 + -42 = 0$</li>
<li>$42 \cdot \frac{1}{42} = 1$</li>
</ul>
<p>This works for matrices as well, but keep in mind that not all matrices have an inverse!</p>
$$\begin{bmatrix}
42 & 99 & 2\\
0 & -35 & 12\\
4 & 2 & 8
\end{bmatrix} \cdot
\begin{bmatrix}
\frac{38}{967} & \frac{197}{1934} & \frac{-629}{3868}\\
\frac{-6}{967} & \frac{-41}{967} & \frac{63}{967}\\
\frac{-35}{1934} & \frac{-39}{967} & \frac{735}{3868}
\end{bmatrix} =
\begin{bmatrix}
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{bmatrix}$$<h4 id="Put-the-modular-in-the-inverse">Put the modular in the inverse<a class="anchor-link" href="https://blog.wificidr.net/posts/the-modular-inverse/#Put-the-modular-in-the-inverse">¶</a></h4><p>Consider a multiplication table for $\pmod 9$ (excluding 0).</p>
$$\begin{array}{c|c|c|c|c|c|c|c|c|}
& 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 \\
\hline
1 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 \\
\hline
2 & 2 & 4 & 6 & 8 & 1 & 3 & 5 & 7 \\
\hline
3 & 3 & 6 & 0 & 3 & 6 & 0 & 3 & 6 \\
\hline
4 & 4 & 8 & 3 & 7 & 2 & 6 & 1 & 5 \\
\hline
5 & 5 & 1 & 6 & 2 & 7 & 3 & 8 & 4 \\
\hline
6 & 6 & 3 & 0 & 6 & 3 & 0 & 6 & 3 \\
\hline
7 & 7 & 5 & 3 & 1 & 8 & 6 & 4 & 2 \\
\hline
8 & 8 & 7 & 6 & 5 & 4 & 3 & 2 & 1 \\
\hline
\end{array}$$<p>A quick inspection will reveal that 1 is indeed the identity element for integers and the multiplication operation.
Our intuition is preserved! 😅
As I noted earlier, not all matrices have an inverse.
Likewise, not all integers have a modular inverse.
Inspect the table closely; can you figure out which numbers <strong>do not</strong> have an inverse and why?</p>
<p>The integers < 9 that are coprime with 9 and their <em>modular inverse</em>:</p>
<ul>
<li>$8 * 8 \pmod{9} = 1$, 8 is the modular inverse of of 8 in $\pmod{9}$</li>
<li>$7 * 4 \pmod{9} = 1$, 4 is the modular inverse of of 7 in $\pmod{9}$</li>
<li>$5 * 2 \pmod{9} = 1$, 2 is the modular inverse of of 5 in $\pmod{9}$</li>
<li>$4 * 7 \pmod{9} = 1$, 7 is the modular inverse of of 4 in $\pmod{9}$</li>
<li>$2 * 5 \pmod{9} = 1$, 5 is the modular inverse of of 2 in $\pmod{9}$</li>
<li>$1 * 1 \pmod{9} = 1$, 1 is the modular inverse of of 1 in $\pmod{9}$</li>
</ul>
<h5 id="Seeking-intuition">Seeking intuition<a class="anchor-link" href="https://blog.wificidr.net/posts/the-modular-inverse/#Seeking-intuition">¶</a></h5><p>Given two integers $A$ and $B$, $A$ has a modular inverse in $\pmod{B}$ <em>if and only if A is coprime with B</em>.</p>
<p>Here is how I think about it.
Consider $4 \pmod{9}$.
You can start at <strong>any</strong> number and count by multiples of 4.
In $\pmod{9}$, you are guaranteed to hit 1 at some point!</p>
<ul>
<li>Start with 30</li>
<li>$30 \pmod{9} = 3$</li>
<li>count by 4's</li>
<li>$34 \pmod{9} = 7$</li>
<li>$38 \pmod{9} = 2$</li>
<li>$42 \pmod{9} = 6$</li>
<li>$46 \pmod{9} = 1$</li>
</ul>
<p>Can you think of a way to determine the number of steps needed?
I covered that in the last post, and it involves Euclid's algorithm and Bézout's identity.</p>
<h3 id="Revisiting-B%C3%A9zout-and-Euclid">Revisiting Bézout and Euclid<a class="anchor-link" href="https://blog.wificidr.net/posts/the-modular-inverse/#Revisiting-B%C3%A9zout-and-Euclid">¶</a></h3><p>Bézout's identity states:</p>
<ul>
<li>If $a$ and $b$ are positive integers, then there exist integers $k$ and $l$ such that $\gcd(a,b)=ak+bl$</li>
</ul>
<p>If $a$ and $b$ are chosen so that their $\gcd = 1$, then we want to find some integers $k$ and $l$ where $1=ak+bl$.
This is done using Euclid's algorithm as described in the <a href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/">previous post</a>.</p>
<h3 id="Conclusion">Conclusion<a class="anchor-link" href="https://blog.wificidr.net/posts/the-modular-inverse/#Conclusion">¶</a></h3><p>I hope this brief discussion clears some of the ideas around the modular inverse.
What we are trying to accomplish is given some value, scale it up in a modular space, then be able to get back to where we started.
Kind of sounds like an encryption scheme, doesn't it? 😁
In my next post in this series, I will finally describe the actual RSA algorithm and how all of these pieces fit together.</p>
</div>
</div>
</div>mathrsahttps://blog.wificidr.net/posts/the-modular-inverse/Wed, 20 Sep 2023 19:46:01 GMTPrime numbers, the Extended Euclidean Algorithm, and the GCDhttps://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/Daniel Justice<div class="cell border-box-sizing text_cell rendered" id="cell-id=29ea443c"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Prime-Numbers,-the-Extended-Euclidean-Algorithm,-and-the-GCD">Prime Numbers, the Extended Euclidean Algorithm, and the GCD<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Prime-Numbers,-the-Extended-Euclidean-Algorithm,-and-the-GCD">¶</a></h2><p>This is the second post in a short series on the RSA algorithm.
The first post was an overview of modular arithmetic, and this article will attempt to cover a few more ideas needed before we dig into the weeds of the algorithm.
It may seem like I am jumping around a bit, but I think this highlights the brilliance of Rivest, Shamir, and Adleman.
They were able to compose an algorithm from pieces that may not seem immediately related to one another.</p>
<h3 id="A-note-on-studying">A note on studying<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#A-note-on-studying">¶</a></h3><p>I skipped some important points in the last article hoping to spark a few people's interest.
I will continue with the goal of keeping the writing light, but we do have to work with a few important theorems and definitions along the way.
Some of the best math advice I ever heard or read is that whenever you are faced with a Definition or a Theorem, stop what you are doing and write down some examples!
"The Math Sorcerer" has a great video about self-study on <a href="https://youtu.be/fb_v5Bc8PSk">YouTube</a>.</p>
<h3 id="More-review">More review<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#More-review">¶</a></h3><p>Formally speaking, the Division Algorithm is the basis of modular arithmetic.
This may feel pedantic, but the aim is to build our argument in small, clear steps.</p>
<h4 id="Definition:-Division-Algorithm">Definition: Division Algorithm<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Definition:-Division-Algorithm">¶</a></h4><p>For all integers $a$ and $m$ with $m>0$, there exist unique integers $q$ and $r$ such that
$$a=mq+r$$
where $0≤r<m$ (Cummings 2022).</p>
<p>A bar or "pipe" is used to mean divides.
For example, if you see $3|6$, this means that 3 divides 6.
A bar with a slash across it means doesn't divide.
$3\nmid 7$.
One integer divides another only if the remainder is zero.</p>
<h4 id="Theorem:-Modular-arithmetic">Theorem: Modular arithmetic<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Theorem:-Modular-arithmetic">¶</a></h4><p>For integers $a$,$r$, and $m$, it is said that $a$ is congruent to $r$ modulo $m$, and one writes $a \equiv r \pmod m$, if $m|(a−r)$ (Cummings 2022).</p>
<p>This concept should be familiar if you read my last article.
The new piece here is the idea of congruency.</p>
<h5 id="Equivalent,-but-not-equal">Equivalent, but not equal<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Equivalent,-but-not-equal">¶</a></h5><p>You will see the word "congruent" in almost any article about the mathematical machinery of the RSA algorithm.
We all know that 4 is not equal to 16.
But what about our $\pmod{12}$ number system?
We can exchange them in equations and get the same result.
This is what it means to be congruent.
In a modular system, we say that 4 is congruent to 16 in $\pmod{12}$.
This is a good point to stop and write down some examples of your own!</p>
<h3 id="Prime-numbers">Prime numbers<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Prime-numbers">¶</a></h3><p>I hope you aren't bored yet!
Number theory begins with many of the ideas you were taught about whole numbers as a child.
Prime numbers play a key role in RSA.
What does it mean to be prime?
Prime numbers are the "atoms" of the integers; they compose all other integers.
I have a bag of small wooden cubes that I have used when teaching my kids math.
A prime number is a number that can only be arranged in a line.
Composite numbers can be built as rectangles, squares or 3-dimensional combinations of squares and rectangles.</p>
<h5 id="Homework">Homework<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Homework">¶</a></h5><p>I leave prime factorization and a formal definition of primes as an exercise for the reader.
If you are a programmer, write an implementation of the <a href="https://mathworld.wolfram.com/SieveofEratosthenes.html">Sieve of Eratosthenes</a> in your favorite language.
Want an old-school challenge?
Write it in <a href="https://n8ta.com/projects/awk_intermediate.html"><code>awk</code></a>.</p>
<h3 id="GCD">GCD<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#GCD">¶</a></h3><h4 id="Theorem:-Greatest-common-divisor">Theorem: Greatest common divisor<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Theorem:-Greatest-common-divisor">¶</a></h4><p>Let $a$ and $b$ be integers. If $c|a$ and $c|b$ ,then $c$ is said to be a common divisor of $a$ and $b$.
The greatest common divisor of $a$ and $b$ is the largest integer $d$ such that $d|a$ and $d|b$.
This number is denoted $gcd(a,b)$ (Cummings 2022).</p>
<p>This should be more grade school maths.</p>
<ul>
<li>$\gcd(20, 24) = 4$</li>
<li>$\gcd(28, 35) = 7$</li>
</ul>
<h4 id="Co-prime-numbers">Co-prime numbers<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Co-prime-numbers">¶</a></h4><p>Two numbers are called "co-prime" if they only have 1 as a common divisor.
In other words, their GCD = 1.
Some examples of co-prime pairs:</p>
<ul>
<li>7, 13</li>
<li>24, 71</li>
<li>112341234120042, 112341234120043</li>
</ul>
<p>Take a look at that last one and notice that they differ by 1.
Take a moment and convince yourself that any two integers with a difference of 1 are co-prime.
Are you convinced?</p>
<h5 id="Homework">Homework<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Homework">¶</a></h5><p>Use this fact to explore one of the earliest proofs that there are an infinite number of primes!</p>
<h3 id="Euler's-totient-function">Euler's totient function<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Euler's-totient-function">¶</a></h3><p><img alt="Euler" src="https://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Leonhard_Euler_2.jpg/384px-Leonhard_Euler_2.jpg" title="Euler"></p>
<p>This guy shows up everywhere in math, doesn't he?!</p>
<p>Euler's totient function is also known as the Euler $\phi$-function.
Don't let the names spook you away; the concept is straightforward.</p>
<p>We are interested in how many numbers are co-prime with a number.
This calls for examples.
Think of the function as the length of a list.</p>
<ul>
<li>$\phi(7) = 6$ because 1, 2, 3, 4, 5, 6 are co-prime with 7; $\gcd(7, 1..6) = 1$.</li>
</ul>
<p>$\phi$ of any prime number is equal to $p - 1$ by the definition of a prime number (you looked that up, earlier, right?).</p>
<ul>
<li>$\phi(11) = 10$</li>
<li>$\phi(65537) = 65536$, AKA $(2^{16} + 1)$ and $2^{16}$ for you programmers.</li>
</ul>
<p>Composite numbers are a little trickier.
We have to perform a prime factorization and eliminate numbers with <strong>any</strong> common factors $\ne$ 1 (because 1 is a divisor of all integers).</p>
<ul>
<li>$\phi(24) = 8$ because 1, 5, 7, 11, 13, 17, 19, 23 are co-prime with 24.</li>
</ul>
<p>Again, 8 is the length of this list.</p>
<p>This topic will be new to many of you without much college-level math, even former engineering students.
It is not horribly complicated at first sight, but my brain exploded when I saw it used in other formulas...
Spoiler alert, we will see it used in another formula!</p>
<h4 id="Prime-factorization">Prime factorization<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Prime-factorization">¶</a></h4><p>The strength of RSA depends solely on the ability to factor large numbers.
Quantum computers could theoretically provide a huge increase in our ability to factor these numbers, and efforts are underway by governments, academic institutions, and corporations to find replacement algorithms that are "post-quantum" safe.
If you want to know more about this, do a search for Peter Shor's algorithm.
The NIST (USA) has already chosen one key exchange suite called Kyber.
As of this writing, it has not been standardized, but companies such as <a href="https://security.googleblog.com/2023/08/toward-quantum-resilient-security-keys.html">Google</a> and <a href="https://cloudflare.net/news/news-details/2023/Cloudflare-Democratizes-Post-Quantum-Cryptography-By-Delivering-It-For-Free-By-Default/default.aspx">Cloudflare</a> are fast at work deploying it across their networks (articles linked in their names).
We don't have scalable quantum computers today, but there is a real threat that encrypted conversations recorded on the internet now could be decrypted in a decade or two.</p>
<h3 id="B%C3%A9zout's-identity">Bézout's identity<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#B%C3%A9zout's-identity">¶</a></h3><p><img alt="Étienne Bézout" src="https://mathshistory.st-andrews.ac.uk/Biographies/Bezout/Bezout_2.jpeg"></p>
<p>We need one more theorem before I wrap up with the Euclidean algorithm.
<strong>This theorem is very important to the inner machinery of RSA</strong>, so make sure you play with it and understand how it works.</p>
<h4 id="Theorem:-B%C3%A9zout's-identity">Theorem: Bézout's identity<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Theorem:-B%C3%A9zout's-identity">¶</a></h4><p>If $a$ and $b$ are positive integers, then there exist integers $k$ and $l$ such that $\gcd(a,b) =ak+bl$ (Cummings 2022).</p>
<p>Note that $k$ and $l$ aren't necessarily positive.
Given $\gcd(28, 35) = 7$, how do we find $k$ and $l$?
In this case, simple inspection tells us that $k=-1$ and $l=1$, thus</p>
$$\gcd(28, 35)=(-1)(28) + (1)(35) = 7$$<p>Do you think $k$ and $l$ are unique?
Existence and uniqueness questions appear frequently in the study of mathematics.</p>
<p>7 and 3 are prime, and $\gcd(7, 3) = 1$.</p>
$$\gcd(7, 3) = 1 = (-2)(3) + (1)(7)$$<p>Before going much further, I want to introduce Cayley tables.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" id="cell-id=5184351e"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h5 id="Cayley-tables">Cayley tables<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Cayley-tables">¶</a></h5><p>Recall addition and multiplication tables from grade school.
We can do the same thing in modular arithmetic systems.
Using the $\pmod{3}$ system, here is the addition table:</p>
<div style="text-align: center"> $a + b \pmod{3}$ </div>
<pre><code> | -7 | -6 | -5 | -4 | -3 | -2 | -1 | 0 | 1 | 2 | 3 | 4 |
+----+----+----+----+----+----+----+----+----+----+----+----+----+
-7 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 |
-6 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 |
-5 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 |
-4 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 |
-3 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 |
-2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 |
-1 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 |
0 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 |
1 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 |
2 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 |
3 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 |
4 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 |
</code></pre>
<p>And here is the multiplication table:</p>
<pre><code> | -7 | -6 | -5 | -4 | -3 | -2 | -1 | 0 | 1 | 2 | 3 | 4 |
+----+----+----+----+----+----+----+----+----+----+----+----+----+
-7 | 1 | 0 | 2 | 1 | 0 | 2 | 1 | 0 | 2 | 1 | 0 | 2 |
-6 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-5 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 |
-4 | 1 | 0 | 2 | 1 | 0 | 2 | 1 | 0 | 2 | 1 | 0 | 2 |
-3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-2 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 |
-1 | 1 | 0 | 2 | 1 | 0 | 2 | 1 | 0 | 2 | 1 | 0 | 2 |
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 |
2 | 1 | 0 | 2 | 1 | 0 | 2 | 1 | 0 | 2 | 1 | 0 | 2 |
3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
4 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 | 2 | 0 | 1 |
</code></pre>
<p>These can be quite useful in your study of the modular world, and they will make another appearance in the next aritcle.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" id="cell-id=ec77023c"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Back-to-B%C3%A9zout">Back to Bézout<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Back-to-B%C3%A9zout">¶</a></h4><p>Do you notice any patterns in the previous tables?
It may not be immediately obvious that the tables are helpful, so I will provide an additional clue.
Consider a "clock" in $\pmod{3}$.</p>
<p><img alt="mod3" src="https://blog.wificidr.net/images/mod3.png"></p>
<p>Now take a tape with all the integers (positive and negative) written on it.
Center is at zero and wrap it around both ways.
Write this down on a piece of paper; do you see the result?
Pay special attention to where <em>multiples</em> of 7 land on the $\pmod{3}$ circle.</p>
<p>Again, Bézout tells us that there is a $k$ and $l$ that satisfies $\gcd(7, 3) =7k+3l$.
Multiples of 7 give us $k$, and corresponding multiple of 3 give us $l$.
I say corresponding because not every muliple of 3 will do.
Notice that it takes two "wraps" of our tape to hit a multiple of 7.</p>
<p><img alt="numline" src="https://blog.wificidr.net/images/numline.png"></p>
<p>Consider the sequence of differences between multiples of 7 and its nearest neighbor.
This results in the sequence</p>
$$[9−7, 15−14, 21−21, 30−28, 36−35, 42−42, 51−59, 57−56, . . .]$$<p>or</p>
$$[2, 1, 0, 2, 1, 0, 2, 1, . . .]$$<p>The pattern should be clear, and one knows it repeats because of the properties of modular arithmetic.</p>
<p>I hope this is starting to make a bit of sense now.
The symmetry of modular systems is fascinating to me, and I hope you have a better intuition of Bézout's identity.
This is a critical Theorem, and I feel like it gets overlooked in a lot of math books and articles.</p>
<h3 id="Extended-Euclidean-Algorithm">Extended Euclidean Algorithm<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Extended-Euclidean-Algorithm">¶</a></h3><p>I am not going to spend much time on this point because the method well-documented in books and across the web.
The mechanics of it should start to feel familiar by this point.
If not, create some examples of your own and work a couple problems (you should do it regardless!).</p>
<p>The table here follows the procedure <a href="https://www.math.cmu.edu/~bkell/21110-2010s/extended-euclidean.html">described by Brian Kell</a>.</p>
<p>Beginning with $a=93060$ and $b=307$, use the division algorithm to find the values $q$ and $r$.
$a$ and $b$ were chosen such that $\gcd(a,b) =1$.
Back-substitute $a$ and $b$ appropriately, and iterate until the remainder is zero.</p>
<style type="text/css">
tr:nth-child(even) {
background-color: #ECE1DE;
}
.thead { font-weight: bold; }
table, th, td {
border: 1px solid black;
border-collapse: collapse;
padding: 10px;
}
</style>
<table class="tg">
<thead>
<tr>
<td class="thead">c</td>
<td class="thead">d</td>
<td class="thead">q</td>
<td class="thead">r</td>
<td class="thead">out</td>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-0lax">93060</td>
<td class="tg-0lax">307</td>
<td class="tg-0lax">303</td>
<td class="tg-0lax">39</td>
<td class="tg-0lax">39 = 93060 - (303)(307) = a - 303b</td>
</tr>
<tr>
<td class="tg-0lax">307</td>
<td class="tg-0lax">39</td>
<td class="tg-0lax">7</td>
<td class="tg-0lax">34</td>
<td class="tg-0lax">34= 307 - 397 = b - 7(a - 303b) = -7a + 2122b</td>
</tr>
<tr>
<td class="tg-0lax">39</td>
<td class="tg-0lax">34</td>
<td class="tg-0lax">1</td>
<td class="tg-0lax">5</td>
<td class="tg-0lax">5 = 39 - 34 = (a - 303b) - (-7a + 2122b) = 8a - 2425b</td>
</tr>
<tr>
<td class="tg-0lax">34</td>
<td class="tg-0lax">5</td>
<td class="tg-0lax">6</td>
<td class="tg-0lax">4</td>
<td class="tg-0lax">4 = 34 - 65 = (-7a + 2122b) - 6(8a - 2425b)</td>
</tr>
<tr>
<td class="tg-0lax">5</td>
<td class="tg-0lax">4</td>
<td class="tg-0lax">1</td>
<td class="tg-0lax">1</td>
<td class="tg-0lax">1 = 5 - 4 = (8a - 2425b) - (-55a + 161672b) = 63a - 19097b</td>
</tr>
</tbody>
</table>
<p>The last row gives us the values of $k$ and $l$ that can be plugged into Bézout's identity.</p>
$$\gcd(93060, 307) = 1 = 63(93060) + -19097(307)$$<h3 id="Wrapping-up">Wrapping up<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Wrapping-up">¶</a></h3><p>This article is quite a bit different than the <a href="https://blog.wificidr.net/posts/modular-arithmetic/">last one</a>, but we have to face the nitty-gritty details at some point.
I hope you are starting to see the connections between these three subjects even though you may be wondering where all this is going.
The next article will introduce the star of the show, the modular inverse!
Don't worry, there will be no surprises there if you can understand the ideas in this post.</p>
<h4 id="Resources">Resources<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Resources">¶</a></h4><p>The web is full of great articles, but I stare at a screen all day.
If you are interested in basic number theory and an absolutely incredible introduction to proofs, I highly recommend "Proofs: A Long-Form Mathematics Textbook" by Jay Cummings.
For my technology friends, Kenneth Rosen's "Discrete mathematics and its applications" has earned itself a permanent spot on my desk.
It covers the number theory used here as well as many other topics relevant to programmers.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" id="cell-id=e29b4ff0"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="References">References<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#References">¶</a></h3><ul>
<li>Cummings, J. (2021). Proofs: A Long-Form Mathematics Textbook.</li>
<li>Judson, T. (2021). Abstract algebra: Theory and Applications. Orthogonal Publishing L3c.</li>
<li>Kell, B. (2010). The extended euclidean algorithm. <a href="https://www.math.cmu.edu/~bkell/21110-2010s/extended-euclidean.html">https://www.math.cmu.edu/~bkell/21110-2010s/extended-euclidean.html</a></li>
<li>Rosen, K. (2011). Discrete mathematics and its applications. McGraw-Hill Education.</li>
</ul>
<h3 id="Photos">Photos<a class="anchor-link" href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/#Photos">¶</a></h3><ul>
<li>Euler portrait: <a href="https://commons.wikimedia.org/wiki/File:Leonhard_Euler_2.jpg">https://commons.wikimedia.org/wiki/File:Leonhard_Euler_2.jpg</a></li>
<li>Bézout portrait: <a href="https://mathshistory.st-andrews.ac.uk/Biographies/Bezout/">https://mathshistory.st-andrews.ac.uk/Biographies/Bezout/</a></li>
</ul>
</div>
</div>
</div>mathrsahttps://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/Sat, 12 Aug 2023 17:31:38 GMTModular arithmetichttps://blog.wificidr.net/posts/modular-arithmetic/Daniel Justice<div class="cell border-box-sizing text_cell rendered" id="cell-id=87a5ecbc"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="An-introduction-to-modular-arithmetic">An introduction to modular arithmetic<a class="anchor-link" href="https://blog.wificidr.net/posts/modular-arithmetic/#An-introduction-to-modular-arithmetic">¶</a></h2><p>Most of my tech-savvy peers have heard of the RSA algorithm or RSA certificates.
It is commonly used to generate ssh key pairs, for example.</p>
<div class="highlight"><pre><span></span>ssh-keygen<span class="w"> </span>-t<span class="w"> </span>rsa<span class="w"> </span>-f<span class="w"> </span>mykey
</pre></div>
<p>The command above produces two files, a private and public key.</p>
<div class="highlight"><pre><span></span>ll<span class="w"> </span>mykey*
.rw-------<span class="w"> </span><span class="m">2</span>.6k<span class="w"> </span>djustice<span class="w"> </span><span class="m">7</span><span class="w"> </span>Aug<span class="w"> </span><span class="m">18</span>:25<span class="w"> </span>mykey
.rw-r--r--<span class="w"> </span><span class="m">586</span><span class="w"> </span>djustice<span class="w"> </span><span class="m">7</span><span class="w"> </span>Aug<span class="w"> </span><span class="m">18</span>:25<span class="w"> </span>mykey.pub
</pre></div>
<p>You can share the public key, but the private key must be kept secure (also note the file permission differences).
Public means public, too!
You can see someone's public key on Github by appending <code>.keys</code> to their username.
It may respond with <code>ssh-rsa ...</code> or something else depending on the algorithm used to generate the key (EdDSA is common).</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>https://github.com/some-user.keys
ssh-rsa<span class="w"> </span>lots-o-chars...
</pre></div>
<p>Many of us in the tech world have seen this base64-encoded soup daily, but have you ever peeked under the hood?
I think the RSA algorithm is one of the more tractable subjects in cryptography, and it opens the door to many other ideas in number theory.</p>
<p>The RSA algorithm works because of the properties of prime numbers and modular arithmetic.
I will start with the latter, and I hope to work the former into another post as I develop this topic.
My goal is to spark an interest, not to provide a rigorous discussion.
There is plenty of jargon to discuss, so please try to work through it.
There is a reason for many of these terms, and I will do my best to justify them as we go.</p>
<p>We will only be working with integers in this article, so think of the set of whole numbers from -∞ to ∞.
For example: {..., -3, -2, -1, 0, 1, 2, 3, ...}.
This set is known in mathematical circles as ℤ.
The symbol is used because it represents a precise idea in a compact space.</p>
<h3 id="Clock-arithmetic">Clock arithmetic<a class="anchor-link" href="https://blog.wificidr.net/posts/modular-arithmetic/#Clock-arithmetic">¶</a></h3><p>Some mathematicians don't like the clock metaphor for modular arithmetic, but I think it is a great starting point.
In my own experiences, I have had the most success explaining this subject to other people using clocks, and the comparison doesn't have any sharp edges that will confuse you later on.</p>
<p><img alt="wall clock" src="https://blog.wificidr.net/images/clock.jpg" title="wall clock"></p>
<p>Starting with a 12-hour wall clock, we will create a number system called "the integers modulus 12".
That is quite a bit to write several times in a row, so you will often see ℤ mod 12, or simply "mod 12".
This wouldn't be a techy post without some code!
The modulus function is essentially the remainder of division by some number; in the current case, 12.
Most programming languages perform the operation using the binary operator <code>%</code> (binary means it takes two arguments).</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" id="cell-id=b0b9eeb1">
<div class="input">
<div class="prompt input_prompt">In [1]:</div>
<div class="inner_cell">
<div class="input_area">
<div class="highlight hl-ipython3"><pre><span></span><span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">14</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"The remainder of </span><span class="si">{n}</span><span class="s2"> divided by 12 is </span><span class="si">{r}</span><span class="s2">."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">n</span><span class="o">=</span><span class="n">n</span><span class="p">,</span> <span class="n">r</span><span class="o">=</span><span class="n">n</span> <span class="o">%</span> <span class="mi">12</span><span class="p">))</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>The remainder of 1 divided by 12 is 1.
The remainder of 2 divided by 12 is 2.
The remainder of 3 divided by 12 is 3.
The remainder of 4 divided by 12 is 4.
The remainder of 5 divided by 12 is 5.
The remainder of 6 divided by 12 is 6.
The remainder of 7 divided by 12 is 7.
The remainder of 8 divided by 12 is 8.
The remainder of 9 divided by 12 is 9.
The remainder of 10 divided by 12 is 10.
The remainder of 11 divided by 12 is 11.
The remainder of 12 divided by 12 is 0.
The remainder of 13 divided by 12 is 1.
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" id="cell-id=933a1571"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Those first few lines can trip people up.
Why is 1 the remainder of 1 divided by 12?
It is because <code>12 * 0 + 1 = 1</code>.
Pay close attention to the last few values.
The remainders don't continue to increment without bound, they roll over back to zero!</p>
<p>Okay, I will admit that it probably isn't that exciting.
Most of us should remember these facts from grade school.
Continuing our introduction (or refresher), do you recall that we can perform arithmetic in this number system?
Ask yourself, if the hour hand is on 5 right now, what time will it be in 37 hours?
The answer is</p>
<pre><code>(5 + 37) mod 12 = 42 mod 12 = 6 o'clock
</code></pre>
<p>Here is another way to think of it.
<code>37 mod 12 = 1</code> and <code>5 + 1 = 6</code>.
Is that a coincidence?
It is not!</p>
<p>What about multiplication; does that work in our modular system?
What does <code>5 * 9</code> hours equal?</p>
<pre><code>(5 * 9) mod 12 = 45 mod 12 = 9 o'clock
</code></pre>
<p>This may seem a bit contrived, but there are practical applications.
It is 2 o'clock when you start your delivery run.
You drive 3 hours north to the warehouse, then make 3, 3-hour round-trips to a remote depot and back.
What time do you return to the warehouse?</p>
<pre><code>[(2 + 3) + (3 * 3)] mod 12 = (5 + 9) mod 12 = 14 mod 12 = 2 mod 12 = 2
</code></pre>
<p>This is fairly basic number theory; things you most likely already know.
If we dig a bit deeper, an interesting structure emerges.</p>
<p><img alt="mod12" src="https://blog.wificidr.net/images/mod12.png" title="mod 12"></p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" id="cell-id=17f49147"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Consider the numbers on the ray originating at 4.</p>
<pre><code>{..., -20, -8, 4, 16, 28, 40, ...}
</code></pre>
<p>When performing arithmetic in <code>mod 12</code>, we can substitute any of these numbers with each other and achieve the same result.
Mathematicians call this an <em>equivalence class</em>.
The terminology is necessary because we are trying to describe a precise idea.
Clearly, -20 and 40 are <strong>not</strong> equal to one another.
However, in the <code>mod 12</code> number system, they are equivalent when we perform computations.
This is typically written as <code>[4]</code> where</p>
<pre><code>[4] = {..., -20, -8, 4, 16, 28, 40, ...}
</code></pre>
<p>Each number <code>n</code> in this set is <em>related</em> to ℤ by <code>n mod 12 = 4</code>, or more compactly: <code>{n | n ∈ ℤ, n mod 12 = 4}</code>.</p>
<p>Look at a few more of these equivalence classes:</p>
<pre><code>[0] = {..., -24, -12, 0, 12, 24, 36, ...},
[1] = {..., -23, -11, 1, 13, 25, 37, ...},
[2] = {..., -22, -10, 2, 14, 26, 38, ...},
...
[11] = {..., -13, -1, 11, 23, 35, 47, ...}
</code></pre>
<p>Start at any column and make your way down subtracting 1 each time.
When you get to the bottom, move one column to the right.
Notice anything interesting?
Every single number in ℤ is represented in one of these... partitions.
Division already has a definition in mathematics, so we will use the phrase <em>partition</em> to describe these collections of numbers.
The really neat part is that this system of partitions representing all the integers works in <em>any</em> modular base!
My teenage son who has no use for math even admitted that this is pretty cool, so put that in your pipe and smoke it!</p>
<h3 id="Homework">Homework<a class="anchor-link" href="https://blog.wificidr.net/posts/modular-arithmetic/#Homework">¶</a></h3><p>This wouldn't be an article on math without some homework.
Use your favorite search engine and read about the Caesar cipher.
Write an implementation in your favorite language and see if your friends can break it.
Just don't send it to me; I am terrible at cryptography.
😂
You can strengthen the cipher by using a random permutation of the alphabet, but both ends of the conversation must use the same permutation.
This is still susceptible to frequency attacks, so don't use it to send GPG keys over the internet.</p>
<h3 id="Next-steps">Next steps<a class="anchor-link" href="https://blog.wificidr.net/posts/modular-arithmetic/#Next-steps">¶</a></h3><p>This subject is inspired by the work I did on my research paper to earn my B.S. in Applied Mathematics.
My plan is to translate it in a way that will be consumable by most programmers (really anyone) who have a little bit of mathematics background.</p>
<p>If I don't get hit by a bus, I hope to write:</p>
<ul>
<li><a href="https://blog.wificidr.net/posts/prime-numbers-the-extended-euclidean-algorithm-and-the-gcd/">Prime numbers, the Extended Euclidean Algorithm, and the GCD</a>.</li>
<li><a href="https://blog.wificidr.net/posts/the-modular-inverse/">The Modular Inverse, an attempt to explain it without hand-waving</a>.</li>
<li>How RSA works (the math, not the code).</li>
</ul>
<p>I am going to provide a detailed guide, but it is up to the reader to sit down and draw their own conclusions about how these things work.
Math is not a spectator sport!</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" id="cell-id=d613bccc"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="References">References<a class="anchor-link" href="https://blog.wificidr.net/posts/modular-arithmetic/#References">¶</a></h3><p>Clock photo: <a href="https://commons.wikimedia.org/wiki/File:B_%26_HB_Kent_Pocket_Watch_(52584138758).jpg">https://commons.wikimedia.org/wiki/File:B_%26_HB_Kent_Pocket_Watch_(52584138758).jpg</a></p>
</div>
</div>
</div>mathrsahttps://blog.wificidr.net/posts/modular-arithmetic/Mon, 31 Jul 2023 23:42:56 GMTData exploration with Rust and Polarshttps://blog.wificidr.net/posts/data-exploration-with-rust-and-polars/Daniel Justice<div class="cell border-box-sizing text_cell rendered" id="cell-id=b1db9c03"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Data-analysis-with-Rust-and-Polars">Data analysis with Rust and Polars<a class="anchor-link" href="https://blog.wificidr.net/posts/data-exploration-with-rust-and-polars/#Data-analysis-with-Rust-and-Polars">¶</a></h2><p>Something different, something new!</p>
<p>I suffer from distractions at times, and I chased a squirrel so hard this time that I am surprised I didn't snap my neck!
I am pretty good about going through open tabs and closing things that I know I simply won't get to in a reasonable time.
For things I might want to reference later, I use <a href="https://getpocket.com/">Pocket</a> and tags.
I had a week of no coursework last week, so I spent some time poking through my backlog.
This whole adventure started off with me thinking about a way to come up with some realistic mock data.
Yes, <a href="https://github.com/joke2k/faker"><code>from faker import Faker</code></a> is trivially easy, but my brain was dragging me down a rabbit hole.
Plus Rust.
Rust is cool.</p>
<h3 id="Gray-squirrel,-a-little-bit-of-data">Gray squirrel, a little bit of data<a class="anchor-link" href="https://blog.wificidr.net/posts/data-exploration-with-rust-and-polars/#Gray-squirrel,-a-little-bit-of-data">¶</a></h3><p>I found a list of <a href="https://en.wikipedia.org/wiki/List_of_international_airports_by_country">international airports on Wikipedia</a>.
Hmmm, that isn't formatted too horribly.
Copy-and-paste and you have yourself a tab-separated file (start with "Africa" and copy to the end of the tables).
Not bad!</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>head<span class="w"> </span>IntAirports.txt<span class="w"> </span>
Africa
Northern<span class="w"> </span>Africa
Algeria
Location<span class="w"> </span>Airport<span class="w"> </span>IATA<span class="w"> </span>Code
Adrar<span class="w"> </span>Touat-Cheikh<span class="w"> </span>Sidi<span class="w"> </span>Mohamed<span class="w"> </span>Belkebir<span class="w"> </span>Airport<span class="w"> </span>AZR
Algiers<span class="w"> </span>Houari<span class="w"> </span>Boumediene<span class="w"> </span>Airport<span class="w"> </span>ALG
</pre></div>
<p>Well, not <em>too</em> bad.
This isn't exactly structured data, but we can probably work with it.
The country is always listed before a header row, so let's ignore the continent and region.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" id="cell-id=a925b8e8-1aa4-48e2-a3c7-99592a2c442f">
<div class="input">
<div class="prompt input_prompt">In [ ]:</div>
<div class="inner_cell">
<div class="input_area">
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">csv</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">"IntAirports.txt"</span><span class="p">)</span> <span class="k">as</span> <span class="n">fp</span><span class="p">,</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'IntAirports.csv'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">,</span> <span class="n">newline</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span> <span class="k">as</span> <span class="n">csvfile</span><span class="p">:</span>
<span class="n">fieldnames</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'country'</span><span class="p">,</span> <span class="s1">'city'</span><span class="p">,</span> <span class="s1">'IATA'</span><span class="p">]</span>
<span class="n">writer</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">DictWriter</span><span class="p">(</span><span class="n">csvfile</span><span class="p">,</span> <span class="n">fieldnames</span><span class="o">=</span><span class="n">fieldnames</span><span class="p">)</span>
<span class="n">writer</span><span class="o">.</span><span class="n">writeheader</span><span class="p">()</span>
<span class="n">old</span> <span class="o">=</span> <span class="s2">""</span>
<span class="n">country</span> <span class="o">=</span> <span class="s2">""</span>
<span class="k">for</span> <span class="n">raw</span> <span class="ow">in</span> <span class="n">fp</span><span class="o">.</span><span class="n">readlines</span><span class="p">():</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">raw</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">if</span> <span class="n">line</span> <span class="o">==</span> <span class="s2">"Location Airport IATA Code"</span><span class="p">:</span>
<span class="n">country</span> <span class="o">=</span> <span class="n">old</span>
<span class="k">continue</span>
<span class="n">old</span> <span class="o">=</span> <span class="n">line</span>
<span class="n">parts</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">"</span><span class="se">\t</span><span class="s2">"</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">parts</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">3</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">writer</span><span class="o">.</span><span class="n">writerow</span><span class="p">({</span>
<span class="s2">"country"</span><span class="p">:</span> <span class="n">country</span><span class="p">,</span>
<span class="s2">"city"</span><span class="p">:</span> <span class="n">parts</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span>
<span class="s2">"IATA"</span><span class="p">:</span> <span class="n">parts</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span>
<span class="p">})</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" id="cell-id=7941b056-d470-47f7-b9b6-1b402ed75225"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>What?
Python?!
Heck, yeah!
Use the appropriate tool for the job.
I am poking at some poorly-formatted data, and it is simply easier and faster to do this in Python.</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>head<span class="w"> </span>IntAirports.csv<span class="w"> </span>
country,city,IATA
Algeria,Adrar,AZR
Algeria,Algiers,ALG
Algeria,Annaba,AAE
Algeria,Batna,BLJ
</pre></div>
<p>Cool, that looks usable.
What can we do with this?
What if I want to know population and lattitude/longitude for the cities on this list?
Kaggle has a "World Cities" CSV you can <a href="https://www.kaggle.com/datasets/juanmah/world-cities">download here</a>.</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>head<span class="w"> </span>worldcities.csv<span class="w"> </span>
<span class="s2">"city"</span>,<span class="s2">"city_ascii"</span>,<span class="s2">"lat"</span>,<span class="s2">"lng"</span>,<span class="s2">"country"</span>,<span class="s2">"iso2"</span>,<span class="s2">"iso3"</span>,<span class="s2">"admin_name"</span>,<span class="s2">"capital"</span>,<span class="s2">"population"</span>,<span class="s2">"id"</span>
<span class="s2">"Tokyo"</span>,<span class="s2">"Tokyo"</span>,<span class="s2">"35.6897"</span>,<span class="s2">"139.6922"</span>,<span class="s2">"Japan"</span>,<span class="s2">"JP"</span>,<span class="s2">"JPN"</span>,<span class="s2">"Tōkyō"</span>,<span class="s2">"primary"</span>,<span class="s2">"37732000"</span>,<span class="s2">"1392685764"</span>
<span class="s2">"Jakarta"</span>,<span class="s2">"Jakarta"</span>,<span class="s2">"-6.1750"</span>,<span class="s2">"106.8275"</span>,<span class="s2">"Indonesia"</span>,<span class="s2">"ID"</span>,<span class="s2">"IDN"</span>,<span class="s2">"Jakarta"</span>,<span class="s2">"primary"</span>,<span class="s2">"33756000"</span>,<span class="s2">"1360771077"</span>
<span class="s2">"Delhi"</span>,<span class="s2">"Delhi"</span>,<span class="s2">"28.6100"</span>,<span class="s2">"77.2300"</span>,<span class="s2">"India"</span>,<span class="s2">"IN"</span>,<span class="s2">"IND"</span>,<span class="s2">"Delhi"</span>,<span class="s2">"admin"</span>,<span class="s2">"32226000"</span>,<span class="s2">"1356872604"</span>
</pre></div>
<h3 id="Fox-squirrel,-data-exploration">Fox squirrel, data exploration<a class="anchor-link" href="https://blog.wificidr.net/posts/data-exploration-with-rust-and-polars/#Fox-squirrel,-data-exploration">¶</a></h3><p>The first step of data analysis is data exploration.
<a href="https://pandas.pydata.org/">Pandas</a> and <a href="https://jupyter.org/">Jupyter notebooks </a> are industry standard tools, but I am digging into <a href="https://www.pola.rs/">Polars</a> today.
There is an excellent series of blog posts about using <a href="https://github.com/wiseaidev/rust-data-analysis/tree/main">Rust for data analysis</a> that you should read if you are interested in digging deeper.
The <a href="https://pola-rs.github.io/polars-book/">Polars user guide</a> is invaluable as well.</p>
<h4 id="Flying-squirrel,-a-note-on-notebooks">Flying squirrel, a note on notebooks<a class="anchor-link" href="https://blog.wificidr.net/posts/data-exploration-with-rust-and-polars/#Flying-squirrel,-a-note-on-notebooks">¶</a></h4><p>I started off using the Rust kernel in Jupyter-lab.
My first impression is that I don't like it much.
Rust is a compiled language, and using it in a notebooks feels clunky and slow because it has to recompile all the time.
When it comes to data exploration, I would use Python and Polars in a Jupyter notebook.
Figure out what your data looks like and what you want your pipeline to do.
Once you have this, it is pretty trivial to write the production implementation in plain Rust.</p>
<h3 id="Arctic-ground-squirrel,-Polars">Arctic ground squirrel, Polars<a class="anchor-link" href="https://blog.wificidr.net/posts/data-exploration-with-rust-and-polars/#Arctic-ground-squirrel,-Polars">¶</a></h3><p>The complete code listing is posted at the end.</p>
<p>Rust is a strongly-typed language, and even Python has to be given a data type when your CSV is full of quoted cells.
I start out by loading the data and declaring the schema.</p>
<div class="highlight"><pre><span></span><span class="k">fn</span> <span class="nf">get_cities</span><span class="p">()</span><span class="w"> </span>-> <span class="nc">DataFrame</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">fields</span>: <span class="nb">Vec</span><span class="o"><</span><span class="n">Field</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">vec!</span><span class="p">[</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"city_ascii"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"lat"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Float32</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"lng"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Float32</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"country"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"iso2"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"iso3"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"admin_name"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"capital"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"population"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Float32</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"id"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Int64</span><span class="p">),</span>
<span class="w"> </span><span class="p">];</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">schema</span>: <span class="nc">Schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Schema</span>::<span class="n">from_iter</span><span class="p">(</span><span class="n">fields</span><span class="p">);</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">cities_file_path</span>: <span class="kp">&</span><span class="nc">Path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Path</span>::<span class="n">new</span><span class="p">(</span><span class="s">"../worldcities.csv"</span><span class="p">);</span>
<span class="w"> </span><span class="n">read_data_frame_from_csv_with_schema</span><span class="p">(</span><span class="n">cities_file_path</span><span class="p">,</span><span class="w"> </span><span class="n">schema</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
<p>This whole project is a software spike, so please forgive the gratuitous use of <code>unwrap</code> and <code>expect</code>.
Polars has a very similar interface to Pandas, so we can load up two different dataframes and compare them just like you would query SQL tables.</p>
<div class="highlight"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">df_cities</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">get_cities</span><span class="p">().</span><span class="n">sort</span><span class="p">([</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="s">"country"</span><span class="p">],</span><span class="w"> </span><span class="kc">true</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">df_airports</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">get_airports</span><span class="p">().</span><span class="n">sort</span><span class="p">([</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="s">"country"</span><span class="p">],</span><span class="w"> </span><span class="kc">true</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">joined</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">df_cities</span>
<span class="w"> </span><span class="p">.</span><span class="n">join</span><span class="p">(</span>
<span class="w"> </span><span class="o">&</span><span class="n">df_airports</span><span class="p">,</span>
<span class="w"> </span><span class="p">[</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="s">"country"</span><span class="p">],</span>
<span class="w"> </span><span class="p">[</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="s">"country"</span><span class="p">],</span>
<span class="w"> </span><span class="n">JoinType</span>::<span class="n">Left</span><span class="p">,</span>
<span class="w"> </span><span class="nb">None</span><span class="p">,</span>
<span class="w"> </span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span>
</pre></div>
<p>I know there are many more cities in the Kaggle data set than the Wikipedia set, so I need to filter null values.
After that, I sort the remaining table by population.</p>
<div class="highlight"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">mask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">joined</span><span class="p">.</span><span class="n">column</span><span class="p">(</span><span class="s">"IATA"</span><span class="p">).</span><span class="n">expect</span><span class="p">(</span><span class="s">"not null"</span><span class="p">).</span><span class="n">is_not_null</span><span class="p">();</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">filtered</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">joined</span>
<span class="w"> </span><span class="p">.</span><span class="n">filter</span><span class="p">(</span><span class="o">&</span><span class="n">mask</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="n">sort</span><span class="p">([</span><span class="s">"population"</span><span class="p">],</span><span class="w"> </span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span>
</pre></div>
<p>And now we can take a peek at the data.</p>
<div class="highlight"><pre><span></span><span class="fm">println!</span><span class="p">(</span>
<span class="w"> </span><span class="s">"{}"</span><span class="p">,</span>
<span class="w"> </span><span class="n">filtered</span>
<span class="w"> </span><span class="p">.</span><span class="n">select</span><span class="p">([</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="s">"country"</span><span class="p">,</span><span class="w"> </span><span class="s">"population"</span><span class="p">,</span><span class="w"> </span><span class="s">"IATA"</span><span class="p">])</span>
<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="n">head</span><span class="p">(</span><span class="nb">Some</span><span class="p">(</span><span class="mi">5</span><span class="p">))</span>
<span class="w"> </span><span class="p">);</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="go">┌───────────┬─────────────┬────────────┬──────┐</span>
<span class="go">│ city ┆ country ┆ population ┆ IATA │</span>
<span class="go">│ --- ┆ --- ┆ --- ┆ --- │</span>
<span class="go">│ str ┆ str ┆ f32 ┆ str │</span>
<span class="go">╞═══════════╪═════════════╪════════════╪══════╡</span>
<span class="go">│ Tokyo ┆ Japan ┆ 3.7732e7 ┆ HND │</span>
<span class="go">│ Jakarta ┆ Indonesia ┆ 3.3756e7 ┆ HLP │</span>
<span class="go">│ Guangzhou ┆ China ┆ 2.694e7 ┆ CAN │</span>
<span class="go">│ Mumbai ┆ India ┆ 2.4973e7 ┆ BOM │</span>
<span class="go">│ Manila ┆ Philippines ┆ 2.4922e7 ┆ MNL │</span>
<span class="go">└───────────┴─────────────┴────────────┴──────┘</span>
</pre></div>
<p>This looks reasonable, and today I learned that Jakarta is a huge city!
Lastly, I write out the data to a new CSV.</p>
<div class="highlight"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">File</span>::<span class="n">create</span><span class="p">(</span><span class="s">"cities-with-airports.csv"</span><span class="p">).</span><span class="n">expect</span><span class="p">(</span><span class="s">"could not create file"</span><span class="p">);</span>
<span class="w"> </span><span class="n">CsvWriter</span>::<span class="n">new</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">file</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">has_header</span><span class="p">(</span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">with_delimiter</span><span class="p">(</span><span class="sc">b','</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">finish</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">filtered</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">expect</span><span class="p">(</span><span class="s">"failed to write output"</span><span class="p">);</span>
</pre></div>
<h3 id="Abert's-squirrel,-takeaways">Abert's squirrel, takeaways<a class="anchor-link" href="https://blog.wificidr.net/posts/data-exploration-with-rust-and-polars/#Abert's-squirrel,-takeaways">¶</a></h3><p>Where did this all start?
Honestly, I already forgot, but it was an excuse to dig into a new library and write some code.
Rust is wicked fast, but the learning curve is infamous.
The Polars library has a great API, and I found it intuitive and easy to use.</p>
<p>TL;DR;</p>
<ul>
<li>Do data exploration in Jupyter+Python</li>
<li>Write your production pipeline in Rust</li>
<li>Profit!</li>
</ul>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" id="cell-id=f47833f6-0819-4c7c-a79e-09de63781a9c"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Full-source">Full source<a class="anchor-link" href="https://blog.wificidr.net/posts/data-exploration-with-rust-and-polars/#Full-source">¶</a></h3><div class="highlight"><pre><span></span><span class="k">use</span><span class="w"> </span><span class="n">polars</span>::<span class="n">prelude</span>::<span class="p">{</span>
<span class="w"> </span><span class="n">CsvReader</span><span class="p">,</span><span class="w"> </span><span class="n">CsvWriter</span><span class="p">,</span><span class="w"> </span><span class="n">DataFrame</span><span class="p">,</span><span class="w"> </span><span class="n">DataFrameJoinOps</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span><span class="p">,</span><span class="w"> </span><span class="n">Field</span><span class="p">,</span><span class="w"> </span><span class="n">JoinType</span><span class="p">,</span><span class="w"> </span><span class="n">Schema</span><span class="p">,</span>
<span class="w"> </span><span class="n">SerReader</span><span class="p">,</span><span class="w"> </span><span class="n">SerWriter</span><span class="p">,</span>
<span class="p">};</span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">fs</span>::<span class="n">File</span><span class="p">;</span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">path</span>::<span class="n">Path</span><span class="p">;</span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">sync</span>::<span class="n">Arc</span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">read_data_frame_from_csv_with_schema</span><span class="p">(</span><span class="n">csv_file_path</span>: <span class="kp">&</span><span class="nc">Path</span><span class="p">,</span><span class="w"> </span><span class="n">schema</span>: <span class="nc">Schema</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">DataFrame</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">CsvReader</span>::<span class="n">from_path</span><span class="p">(</span><span class="n">csv_file_path</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">expect</span><span class="p">(</span><span class="s">"Cannot open file."</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">has_header</span><span class="p">(</span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">with_schema</span><span class="p">(</span><span class="n">Arc</span>::<span class="n">new</span><span class="p">(</span><span class="n">schema</span><span class="p">))</span>
<span class="w"> </span><span class="p">.</span><span class="n">finish</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">get_cities</span><span class="p">()</span><span class="w"> </span>-> <span class="nc">DataFrame</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">fields</span>: <span class="nb">Vec</span><span class="o"><</span><span class="n">Field</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">vec!</span><span class="p">[</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"city_ascii"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"lat"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Float32</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"lng"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Float32</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"country"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"iso2"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"iso3"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"admin_name"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"capital"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"population"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Float32</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"id"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Int64</span><span class="p">),</span>
<span class="w"> </span><span class="p">];</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">schema</span>: <span class="nc">Schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Schema</span>::<span class="n">from_iter</span><span class="p">(</span><span class="n">fields</span><span class="p">);</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">cities_file_path</span>: <span class="kp">&</span><span class="nc">Path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Path</span>::<span class="n">new</span><span class="p">(</span><span class="s">"../worldcities.csv"</span><span class="p">);</span>
<span class="w"> </span><span class="n">read_data_frame_from_csv_with_schema</span><span class="p">(</span><span class="n">cities_file_path</span><span class="p">,</span><span class="w"> </span><span class="n">schema</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">get_airports</span><span class="p">()</span><span class="w"> </span>-> <span class="nc">DataFrame</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">fields</span>: <span class="nb">Vec</span><span class="o"><</span><span class="n">Field</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">vec!</span><span class="p">[</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"country"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="n">Field</span>::<span class="n">new</span><span class="p">(</span><span class="s">"IATA"</span><span class="p">,</span><span class="w"> </span><span class="n">DataType</span>::<span class="n">Utf8</span><span class="p">),</span>
<span class="w"> </span><span class="p">];</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">schema</span>: <span class="nc">Schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Schema</span>::<span class="n">from_iter</span><span class="p">(</span><span class="n">fields</span><span class="p">);</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">airports_file_path</span>: <span class="kp">&</span><span class="nc">Path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Path</span>::<span class="n">new</span><span class="p">(</span><span class="s">"../IntAirports.csv"</span><span class="p">);</span>
<span class="w"> </span><span class="n">read_data_frame_from_csv_with_schema</span><span class="p">(</span><span class="n">airports_file_path</span><span class="p">,</span><span class="w"> </span><span class="n">schema</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">df_cities</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">get_cities</span><span class="p">().</span><span class="n">sort</span><span class="p">([</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="s">"country"</span><span class="p">],</span><span class="w"> </span><span class="kc">true</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">df_airports</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">get_airports</span><span class="p">().</span><span class="n">sort</span><span class="p">([</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="s">"country"</span><span class="p">],</span><span class="w"> </span><span class="kc">true</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">joined</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">df_cities</span>
<span class="w"> </span><span class="p">.</span><span class="n">join</span><span class="p">(</span>
<span class="w"> </span><span class="o">&</span><span class="n">df_airports</span><span class="p">,</span>
<span class="w"> </span><span class="p">[</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="s">"country"</span><span class="p">],</span>
<span class="w"> </span><span class="p">[</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="s">"country"</span><span class="p">],</span>
<span class="w"> </span><span class="n">JoinType</span>::<span class="n">Left</span><span class="p">,</span>
<span class="w"> </span><span class="nb">None</span><span class="p">,</span>
<span class="w"> </span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">mask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">joined</span><span class="p">.</span><span class="n">column</span><span class="p">(</span><span class="s">"IATA"</span><span class="p">).</span><span class="n">expect</span><span class="p">(</span><span class="s">"not null"</span><span class="p">).</span><span class="n">is_not_null</span><span class="p">();</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">filtered</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">joined</span>
<span class="w"> </span><span class="p">.</span><span class="n">filter</span><span class="p">(</span><span class="o">&</span><span class="n">mask</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="n">sort</span><span class="p">([</span><span class="s">"population"</span><span class="p">],</span><span class="w"> </span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span>
<span class="w"> </span><span class="s">"{}"</span><span class="p">,</span>
<span class="w"> </span><span class="n">filtered</span>
<span class="w"> </span><span class="p">.</span><span class="n">select</span><span class="p">([</span><span class="s">"city"</span><span class="p">,</span><span class="w"> </span><span class="s">"country"</span><span class="p">,</span><span class="w"> </span><span class="s">"population"</span><span class="p">,</span><span class="w"> </span><span class="s">"IATA"</span><span class="p">])</span>
<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="n">head</span><span class="p">(</span><span class="nb">Some</span><span class="p">(</span><span class="mi">5</span><span class="p">))</span>
<span class="w"> </span><span class="p">);</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">File</span>::<span class="n">create</span><span class="p">(</span><span class="s">"cities-with-airports.csv"</span><span class="p">).</span><span class="n">expect</span><span class="p">(</span><span class="s">"could not create file"</span><span class="p">);</span>
<span class="w"> </span><span class="n">CsvWriter</span>::<span class="n">new</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">file</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">has_header</span><span class="p">(</span><span class="kc">true</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">with_delimiter</span><span class="p">(</span><span class="sc">b','</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">finish</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">filtered</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">expect</span><span class="p">(</span><span class="s">"failed to write output"</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
</div>mathrusthttps://blog.wificidr.net/posts/data-exploration-with-rust-and-polars/Tue, 06 Jun 2023 17:26:54 GMTBasic Counting Ruleshttps://blog.wificidr.net/posts/basic-counting-rules/Daniel Justice<div class="cell border-box-sizing text_cell rendered" id="cell-id=13087553"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Basic-counting-rules">Basic counting rules<a class="anchor-link" href="https://blog.wificidr.net/posts/basic-counting-rules/#Basic-counting-rules">¶</a></h2><p>AKA this is harder than it looks, and I'm not a great teacher some days.</p>
<p>I was helping my youngest (who isn't that young any more) with his math homework the other night, and we were learning about combinations and permutations.
Some of these problems are fun, and some are depressingly difficult to sort out, and most of them make you facepalm once you learn the solution to a wrong answer.
How hard can it be?!
Page 1 of "A First Course in Probability" begins with "The Basic Principle of Counting" (Ross, 2010).
The rules themselves are straightforward.
Most of us are familiar with counting the possible number of license plates given some collection of available letters and numbers.
Things get trickier, though, when we mix and match and when we consider dependent and independent probabilities.</p>
<h3 id="A-bag-of-balls,-queue-up-AC%E2%9A%A1%EF%B8%8FDC">A bag of balls, queue up AC⚡️DC<a class="anchor-link" href="https://blog.wificidr.net/posts/basic-counting-rules/#A-bag-of-balls,-queue-up-AC%E2%9A%A1%EF%B8%8FDC">¶</a></h3><p>My son was working with a problem involving a bag of colored balls.
There were some green, some red, and one gold; a total of ten balls in a bag.
I don't recall what the original problem was because I saw a squirrel and chased it.
I have had more than one semester of statistics classes, this can't be that hard!
I blurted out the first question that came to mind, "What is the probability of pulling out a gold ball if you draw a ball out ten times?".</p>
<p>So far, so good.</p>
<p>We immediately discussed how important it is to quantify the problem.
Details matter!
We can't tell same-colored balls apart, so this isn't a permutation problem (order doesn't matter).
What if we don't replace the balls?
The probability of pulling a gold ball in ten trials is 100%, easy!
🎉
I am really interested in the situation with replacement, though.
I thought the answer would be super easy, and I grabbed a calculator and started scratching my chin.
"Oh, that's not right."
"Hrm, neither is that."
"Oh, oh, I see, but why???"
His mother immediately scolded me for chasing a squirrel and they returned to their homework while I tried to sort out the problem in my own pea brain.</p>
<p>The chances of pulling a gold ball out of the bag of ten balls is 10% for any trial.
We can all agree that this is an independent event, considered on its own.
If you take my initial, naïve approach, you end up with in a dead end.
By the rules of independent trials, you simply multiply the probabilities together.</p>
$$ 0.1^{10} = 1.0\times10^{-10} $$<p>Oh, snap, that is a really small number.</p>
<p>Even worse, it gets smaller the more trials we do; that can't be right!
Astute readers (or anyone with more brain cells than me) will quickly spot the solution, but is it clear <em>why</em>?
Maybe this problem is a consequence of the subtlety of language.
At any rate, we have to consider that if we pull out a gold ball, the game is over.
It doesn't matter if that is on the first or the tenth trial.
This problem is a really good example of when it is more useful to consider the <em>compliment</em> of the probability.</p>
$$ p + q = 1 $$<p>Where <code>p</code> is the probability of our event and <code>q</code> is the compliment.
Another way to ask our burning question is, "What is the likelihood of <strong>not</strong> drawing a gold ball in ten trials?"
This is straightforward, and our independent trial formula works just fine.</p>
$$ 0.9^{10} \approx 0.35 $$<p>This should calm our intuition.
It gets smaller with more trials as one would expect.</p>
<p>Why doesn't it work the other way around? (keep reading!)</p>
<p>I'll admit; I didn't earn a great score in probability theory.
I may be showing my ignorance, but I have a soft spot for the struggle my own child is experiencing at the moment.</p>
<h3 id="Another-look">Another look<a class="anchor-link" href="https://blog.wificidr.net/posts/basic-counting-rules/#Another-look">¶</a></h3><p>Consider it this way, "What is the probability of drawing a gold ball on the first trial?"
Easy-peasy, 10%.
"What is the probability of pulling out a gold ball in two trails?"
A ball, any, it doesn't matter, we just want a gold ball.
There is a ten percent chance on the first draw, and a success there eliminates the need to draw a second time.
Think about the space of all possible outcomes; the cardinality is $10^2$ for two trials.
There is one happy outcome and 9 sad ones on the first draw.
The pattern repeats <em>for each possible outcome of the first trial</em>.
Out of 100 possibilities in the second trial, 19 of them contain a gold ball.</p>
<p>The probability of drawing a gold ball in two trials is:</p>
<ul>
<li>the probability of drawing a gold ball on the first trial <em>plus</em></li>
<li>the probability of <strong>not</strong> drawing a gold ball <strong>in the first trial</strong> <em>multiplied</em><ul>
<li>by the probability of drawing a gold ball during an independent, second trial.</li>
</ul>
</li>
</ul>
$$ 0.1 + (0.9 * 0.1) = 0.19 $$<p>The probability of drawing a gold ball in three trials is:</p>
<ul>
<li>the probability of drawing a gold ball on the first trial <em>plus</em></li>
<li>the probability of <strong>not</strong> drawing a gold ball <strong>in the first trial</strong> <em>multiplied</em><ul>
<li>by the probability of drawing a gold ball during an independent, second trial <em>plus</em></li>
</ul>
</li>
<li>the probability of <strong>not</strong> drawing a gold ball in the <strong>previous two</strong> trails <em>multiplied</em><ul>
<li>by the probability of not drawing a gold ball during an indpendent, third trial.</li>
</ul>
</li>
</ul>
$$ (0.1 + (0.9 * 0.1)) + (0.9*0.9*0.1) = 0.271 $$<p>This is quickly turning into a mess, but do you see the pattern?
The probability of each trial is <em>dijoint</em> and independent from the previous trials, so we use the addition rule.
The probability of getting a gold ball on this trial is <strong>dependent</strong> on the chances of getting here without a gold ball <em>multiplied</em> by the probability of drawing a gold ball right now (10%).
There is a great article that summarizes this on <a href="https://courses.lumenlearning.com/boundless-statistics/chapter/probability-rules/">Lumen Learning</a>.</p>
<p>Another way to look at it is that 10 percent of the possibilities of any round are ancestors of the first possible gold ball. Add this 10 percent to 9 times the previous number of success, and you have the number of gold paths to this point.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" id="cell-id=9d94cddf">
<div class="input">
<div class="prompt input_prompt">In [17]:</div>
<div class="inner_cell">
<div class="input_area">
<div class="highlight hl-ipython3"><pre><span></span><span class="n">acc</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">10</span><span class="p">):</span>
<span class="n">acc</span> <span class="o">=</span> <span class="p">(</span><span class="n">acc</span> <span class="o">*</span> <span class="mi">9</span><span class="p">)</span> <span class="o">+</span> <span class="mi">10</span><span class="o">**</span><span class="n">i</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">acc</span> <span class="o">/</span> <span class="mi">10</span><span class="o">**</span><span class="p">(</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>0.19
0.271
0.3439
0.40951
0.468559
0.5217031
0.56953279
0.612579511
0.6513215599
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" id="cell-id=1b4a94e1"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We can extrapolate that out to ten trials, but isn't it a lot easier to subtract one from the compliment?!</p>
$$ 1 - 0.9^{10} \approx 0.65 $$<p>There is a 65% chance of drawing a gold ball in ten trials with replacement.</p>
<p>I hope this shows how a simple problem can become overly complex if you approach it in a suboptimal way.
How often do we do this when writing code?</p>
<h3 id="Takeaways">Takeaways<a class="anchor-link" href="https://blog.wificidr.net/posts/basic-counting-rules/#Takeaways">¶</a></h3><ul>
<li>Don't get too creative and deviate away from the learning material when a student is involved unless the material is really fresh in your mind.</li>
<li>I love playing with numbers, but probability problems require a lot of careful consideration. Be especially considerate of students who may not be native speakers. Word problems are evil enough for those of use who grew up speaking English.</li>
<li>Draw pictures! I have a notebook littered with tree diagrams from my own probability class.</li>
<li>It's okay to make mistakes in front of your kids. But don't give up; be a model of resourcefulness and calm determination (ok, ok, maybe I struggle with the calm part 😜).</li>
</ul>
<h3 id="References">References<a class="anchor-link" href="https://blog.wificidr.net/posts/basic-counting-rules/#References">¶</a></h3><ul>
<li>Ross, S. (2009). A First Course in Probability (8th Edition) (8th ed.). Pearson Prentice Hall.</li>
<li><a href="https://courses.lumenlearning.com/boundless-statistics/chapter/probability-rules/">https://courses.lumenlearning.com/boundless-statistics/chapter/probability-rules/</a></li>
</ul>
</div>
</div>
</div>mathhttps://blog.wificidr.net/posts/basic-counting-rules/Mon, 04 Apr 2022 20:58:00 GMTMonoids for Pythonistashttps://blog.wificidr.net/posts/monoids-for-pythonistas/Daniel Justice<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Monoids-for-Pythonistas">Monoids for Pythonistas<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Monoids-for-Pythonistas">¶</a></h2><p><a href="https://bartoszmilewski.com/">Bartosz Milewsky</a> has a great series of articles on category theory as it applies to programmers.
I'm not going to lie, the material is pretty dense which is par for the course for a lot of mathematics subjects (maybe it's worth mentioning that Bartosz has a PhD in Theoretical Physics 😬).
However, many of the core concepts are reasonably simple abstractions, and programmers know all about abstractions!
The language can be as big a hurdle as reading mathematics symbols, but these are the tools mathematicians use to convey ideas in an elegant and concise way.</p>
<p>I promise that you know what monoids are - you use them every day as a programmer!
I am going to flip the script and try to work my way to the definition instead of the other way around.
Hopefully, with a bit of patience, we can uncover the power of these ideas and see how they apply to our day-to-day jobs!</p>
<p><strong>Disclaimer</strong> This is not meant to be a mathematically rigorous discussion. Many small details are overlooked in favor of a high-level understanding. However, please make me aware of any egregious offenses!</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Sets">Sets<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Sets">¶</a></h3><p>As a programmer, you <em>should</em> know what a set is - an unordered collection of unique items of the same type (and if not, maybe a blog post for another day!).</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">my_pets</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"Lamby"</span><span class="p">,</span> <span class="s2">"Dozer"</span><span class="p">,</span> <span class="s2">"Onyx"</span><span class="p">,</span> <span class="s2">"BlackHen"</span><span class="p">,</span> <span class="s2">"RedHen"</span><span class="p">,</span> <span class="s2">"NoisyRooster"</span><span class="p">}</span>
<span class="o">>>></span> <span class="n">my_pets</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s2">"BuffHen"</span><span class="p">)</span>
<span class="o">>>></span> <span class="s2">"Frank"</span> <span class="ow">in</span> <span class="n">my_pets</span>
<span class="kc">False</span>
<span class="o">>>></span> <span class="n">my_pets</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s2">"BuffHen"</span><span class="p">)</span> <span class="c1"># `add` is idempotent and won't create a duplicate</span>
<span class="o">>>></span> <span class="n">my_pets</span> <span class="c1"># Notice that the order isn't consistent</span>
<span class="p">{</span><span class="s1">'NoisyRooster'</span><span class="p">,</span> <span class="s1">'RedHen'</span><span class="p">,</span> <span class="s1">'BuffHen'</span><span class="p">,</span> <span class="s1">'BlackHen'</span><span class="p">,</span> <span class="s1">'Lamby'</span><span class="p">,</span> <span class="s1">'Onyx'</span><span class="p">,</span> <span class="s1">'Dozer'</span><span class="p">}</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="n">pets_son_feeds</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"Shayna"</span><span class="p">,</span> <span class="s2">"Lamby"</span><span class="p">}</span>
<span class="o">>>></span> <span class="n">pets_son_feeds</span><span class="o">.</span><span class="n">intersection</span><span class="p">(</span><span class="n">my_pets</span><span class="p">)</span>
<span class="p">{</span><span class="s1">'Lamby'</span><span class="p">}</span>
<span class="o">>>></span>
</pre></div>
<p>The integers form a set, $\mathbb{I} = \{-1,0,1,2,3,4...\}$.
Rational numbers form a set, $\mathbb{Q} = \{-99.42,0,1.1,1.2,1.3...\}$.
Character strings even form a set, $\{\text{"a", "b", "c", "foo", "cow", "Bob", ...}\}$.
For the purposes of this article, a "set" is a mathematical set, not constrained by physical memory limits of computers.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Symbols">Symbols<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Symbols">¶</a></h3><p>Symbols are the syntax of mathematics.</p>
<h4 id="$%5Cforall$">$\forall$<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#$%5Cforall$">¶</a></h4><p>Means "for all". Given a set, "<em>for every element</em> in the set." In the definition of a monoid, we will use this to assert a property of the items in the set. It is somewhat analogous to a <code>for</code> loop over all the items.</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">odds</span> <span class="o">=</span> <span class="p">{</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">9</span><span class="p">}</span>
<span class="o">>>></span> <span class="k">for</span> <span class="n">odd</span> <span class="ow">in</span> <span class="n">odds</span><span class="p">:</span> <span class="c1"># ∀ in the set...</span>
<span class="o">...</span> <span class="k">assert</span> <span class="n">odd</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">1</span> <span class="c1"># Assert that an odd divided by 2 has a remainder of 1</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="c1"># Success!</span>
</pre></div>
<h4 id="$%5Cin$">$\in$<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#$%5Cin$">¶</a></h4><p>means "in" or "an element of". 1 is <em>an element of</em> the set of integers.</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">some_primes</span> <span class="o">=</span> <span class="p">{</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">11</span><span class="p">}</span>
<span class="o">>>></span> <span class="mi">7</span> <span class="ow">in</span> <span class="n">some_primes</span> <span class="c1"># 7 ∈ some_primes == True</span>
<span class="kc">True</span>
<span class="o">>>></span>
</pre></div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="More-terminology">More terminology<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#More-terminology">¶</a></h3><h4 id="Binary-operation">Binary operation<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Binary-operation">¶</a></h4><p>An operation that works on two (hence binary) values; addition, multiplication, division, etc. Most of the Python numeric <a href="https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types">dunder methods</a> are binary operations.</p>
<h4 id="Associative">Associative<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Associative">¶</a></h4><p>If an operation is associative, the order of <em>composition</em> doesn't matter.
This property is <strong>very</strong> important as we'll see a bit later.</p>
<ul>
<li>Addition <strong>is</strong> associative. 1 + (2 + 3) is the same as (1 + 2) + 3.</li>
<li>String concatenation <strong>is</strong> associative. "A" + ("B" + "C") == ("A" + "B") + "C"<ul>
<li>Unlike numeric addition where 1 + 2 == 2 + 1, string concatenation is <strong>not</strong> <em>commutative</em>! Monoids only depend on the associative property. (See <a href="https://blogs.ncl.ac.uk/andreymokhov/united-monoids/">Andrey Mokhov's article</a> for a fascinating deep-dive into this subject)</li>
</ul>
</li>
<li>Subtraction is <strong>not</strong> associative as order of composition matters. 1 - (2 - 3) is <strong>not equal</strong> to (1 - 2) - 3.</li>
</ul>
<h4 id="Closed">Closed<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Closed">¶</a></h4><p>For a given operation, the input and output types must match for the operation to be closed.</p>
<ul>
<li>The minimum of two integers maps back into the set of integers.</li>
</ul>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">min_</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">min</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
</pre></div>
<ul>
<li>Multiplication of two integers stays within the set of integers.</li>
</ul>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="mi">42</span> <span class="o">*</span> <span class="mi">7</span><span class="p">)</span>
<span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">int</span><span class="s1">'></span>
<span class="o">>>></span> <span class="n">a</span> <span class="o">=</span> <span class="mi">42</span>
<span class="o">>>></span> <span class="n">b</span> <span class="o">=</span> <span class="mi">7</span>
<span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">int</span><span class="s1">'></span>
<span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
<span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">int</span><span class="s1">'></span>
<span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="n">a</span> <span class="o">*</span> <span class="n">b</span><span class="p">)</span>
<span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">int</span><span class="s1">'></span>
<span class="o">>>></span>
</pre></div>
<ul>
<li>Floating point division of two integers returns a float, which is outside the set of integers. The <em>input</em> and <em>output</em> types don't match. This is <strong>not</strong> a closed operation.</li>
</ul>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="mi">42</span> <span class="o">/</span> <span class="mi">33</span><span class="p">)</span>
<span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">float</span><span class="s1">'></span>
<span class="o">>>></span>
</pre></div>
<ul>
<li>The <code>chr</code> function in Python takes an integer and returns a <code>str</code>. This operation is <strong>not</strong> closed for the same reason; the output is of a different type than the input.</li>
</ul>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="nb">chr</span><span class="p">(</span><span class="mi">51</span><span class="p">))</span>
<span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">str</span><span class="s1">'></span>
<span class="o">>>></span>
</pre></div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Axioms">Axioms<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Axioms">¶</a></h3><p>We're almost there! We need to mention identity. The importance of identity is a bit of a rabbit hole, so I'll leave it as an exercise for the reader to dig deeper (abstract algebra rings and rngs).</p>
<p>An identity element is an element that does not change the output when paired with any other input.</p>
<h4 id="Identity">Identity<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Identity">¶</a></h4><ul>
<li>Numeric addition: 0 is the identity (works with reals, integers, naturals, etc).</li>
</ul>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="mi">42</span> <span class="o">+</span> <span class="mi">0</span>
<span class="mi">42</span>
<span class="o">>>></span>
</pre></div>
<ul>
<li>Numeric multiplication: 1 is the identity (gotcha! The identity is dependent on the <em>operation</em> and the <em>type</em>).</li>
</ul>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="mi">42</span> <span class="o">*</span> <span class="mi">1</span>
<span class="mi">42</span>
<span class="o">>>></span>
</pre></div>
<ul>
<li>List concatenation: the empty list is the identity</li>
</ul>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="o">+</span> <span class="p">[]</span>
<span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
<span class="o">>>></span>
</pre></div>
<p>The reason we need this is a little subtle.
We are using a binary operation to make a computation, but what do we do when we only have one value to work with?
Many of the use cases of monoids are recursive in nature, so what do you do when you reduce a list to one element?
This is fairly representative of the problem (or imagine if <code>functools.reduce</code> didn't take a default argument).</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="nb">list</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="p">,</span> <span class="p">[</span><span class="mi">42</span><span class="p">]))</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s2">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="ne">TypeError</span><span class="p">:</span> <span class="o"><</span><span class="k">lambda</span><span class="o">></span><span class="p">()</span> <span class="n">missing</span> <span class="mi">1</span> <span class="n">required</span> <span class="n">positional</span> <span class="n">argument</span><span class="p">:</span> <span class="s1">'y'</span>
<span class="o">>>></span>
</pre></div>
<p>We can use the identity!
This allows the developer to safely perform a binary operation on a single element without changing the output.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Definition">Definition<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Definition">¶</a></h3><p>Hopefully, we now have the tools to unpack this definition...</p>
<p>According to Eric Weisstein on Wolfram, a monoid is "a set that is closed under an associative binary operation and has an identity element $I \in S$ such that $ \forall a \in S, Ia=aI=a$".</p>
<p>I don't like that definition.
What the heck does $Ia=aI=a$ mean?
It <em>looks</em> like multiplication, but that is one of many possible binary operations.
Here's my rephrasing:<br>
A monoid consists of a set, $S$, that is closed under an associative binary operation, $f$, and has an identity element $I \in S$ such that $ \forall a \in S, f(I, a)=f(a, I)=a$.</p>
<p>A monoid has three components.</p>
<ul>
<li>The set $S$, often called the "carrier set". Programmers know this as a type (int, float, double, etc).</li>
<li>$I$ is the identity element, which we identified in the previous section</li>
<li>A binary operation. Addition, multiplication, list concatenation, etc.</li>
</ul>
<p>I said in the intro that you already know what a monoid is.
Here are some examples:</p>
<ul>
<li>Integers, addition, 0</li>
<li>Reals, multiplication, 1</li>
<li>Lists, concatenation, [ ]</li>
</ul>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Why-it-matters">Why it matters<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Why-it-matters">¶</a></h3><p>At this point, you might be thinking about whipping me with a large stick for describing simple arithmetic in such obtuse and abstract terms.
It matters, I promise!
These definitions allow us to precisely identify objects with certain properties that support certain operations.
Enough math, why does this matter to programmers?!</p>
<h4 id="Divide">Divide<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Divide">¶</a></h4><p>Remember how I said that associative property would be important later on?
Consider the following items... they're all equivalent.</p>
<pre><code>1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10
(1 + 2) + (3 + 4 + 5 + 6 + 7 + 8 + 9 + 10)
(1 + (2 + (3 + (4 + (5 + (6 + (7 + (8 + (9 + 10)))))))))
(((((((((1 + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) + 10)
</code></pre>
<p>Here's another example.</p>
<pre><code>"a" + "b" + "c" + "d" + "e" + "f" + "g" + "h" + "i" + "j"
"ab" + "cd" + "ef" + "gh" + "ij"
"abcd" + "efgh" + "ij"
"abcdefgh" + "ij"
"abcdefghij"
</code></pre>
<p>I can write those down on a set of index cards and pass the pairs to two people at a time (think two CPU cores).
As long as I track the order of the results, I can recursively continue this process in any order of composition until the task is done.
This process succeeds even if the workers have different computational capacities.
The answer is the same as if one poor individual toiled away at the tasks independently.</p>
<h4 id="More-cores,-more-computers">More cores, more computers<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#More-cores,-more-computers">¶</a></h4><p>If the list is really long or the binary operation is non-trivial, we can <strong>safely</strong> and easily divide this tasks amongst different cores - or even different computers.
Notice that we are talking about <em>CPU</em>-bound tasks, not <em>IO</em>-bound tasks (multiple cores do little to alleviate the latter).
Python doesn't have the ability to do this natively, but other languages do.</p>
<h5 id="But-be-careful">But be careful<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#But-be-careful">¶</a></h5><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="kn">import</span> <span class="n">Pool</span><span class="p">,</span> <span class="n">cpu_count</span>
<span class="kn">import</span> <span class="nn">timeit</span>
<span class="k">def</span> <span class="nf">read_data</span><span class="p">(</span><span class="n">fname</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Read Wiki tables"""</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">fname</span><span class="p">)</span> <span class="k">as</span> <span class="n">fp</span><span class="p">:</span>
<span class="n">items</span> <span class="o">=</span> <span class="n">fp</span><span class="o">.</span><span class="n">readlines</span><span class="p">()</span>
<span class="k">return</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">item</span><span class="p">:</span> <span class="nb">int</span><span class="p">(</span><span class="n">item</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">" "</span><span class="p">)[</span><span class="mi">3</span><span class="p">]),</span> <span class="n">items</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">do_it</span><span class="p">(</span><span class="n">fname</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Read data and perform calculation"""</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">read_data</span><span class="p">(</span><span class="n">fname</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">sum</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">multi_proc_demo</span><span class="p">()</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
<span class="n">N_PROC</span> <span class="o">=</span> <span class="n">cpu_count</span><span class="p">()</span>
<span class="c1"># Start the processes</span>
<span class="k">with</span> <span class="n">Pool</span><span class="p">(</span><span class="n">processes</span><span class="o">=</span><span class="n">N_PROC</span><span class="p">)</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span>
<span class="n">results</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">fnames</span> <span class="o">=</span> <span class="p">[</span>
<span class="s2">"data/pagecounts-20090601-000000"</span><span class="p">,</span>
<span class="s2">"data/pagecounts-20090601-010000"</span><span class="p">,</span>
<span class="s2">"data/pagecounts-20090601-020000"</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">fname</span> <span class="ow">in</span> <span class="n">fnames</span><span class="p">:</span>
<span class="n">results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">do_it</span><span class="p">,</span> <span class="n">kwds</span><span class="o">=</span><span class="p">{</span><span class="s1">'fname'</span><span class="p">:</span> <span class="n">fname</span><span class="p">}))</span>
<span class="nb">sum</span><span class="p">([</span><span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span> <span class="k">for</span> <span class="n">result</span> <span class="ow">in</span> <span class="n">results</span><span class="p">])</span>
<span class="k">def</span> <span class="nf">single_proc_demo</span><span class="p">()</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
<span class="c1"># cat data/pagecounts* >> all.txt</span>
<span class="nb">sum</span><span class="p">(</span><span class="n">read_data</span><span class="p">(</span><span class="s2">"data/all.txt"</span><span class="p">))</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"Multi-proc: "</span><span class="p">,</span> <span class="n">timeit</span><span class="o">.</span><span class="n">Timer</span><span class="p">(</span><span class="s2">"multi_proc_demo()"</span><span class="p">,</span> <span class="nb">globals</span><span class="o">=</span><span class="nb">globals</span><span class="p">())</span><span class="o">.</span><span class="n">repeat</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"Single-proc: "</span><span class="p">,</span> <span class="n">timeit</span><span class="o">.</span><span class="n">Timer</span><span class="p">(</span><span class="s2">"single_proc_demo()"</span><span class="p">,</span> <span class="nb">globals</span><span class="o">=</span><span class="nb">globals</span><span class="p">())</span><span class="o">.</span><span class="n">repeat</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
</pre></div>
<div class="highlight"><pre><span></span>python<span class="w"> </span>wiki.py
Multi-proc:<span class="w"> </span><span class="o">[</span><span class="m">28</span>.17182195998612,<span class="w"> </span><span class="m">28</span>.688616012994316,<span class="w"> </span><span class="m">28</span>.539513622003142,<span class="w"> </span><span class="m">28</span>.077732990990626,<span class="w"> </span><span class="m">28</span>.294970447997912<span class="o">]</span>
Single-proc:<span class="w"> </span><span class="o">[</span><span class="m">75</span>.17859069499536,<span class="w"> </span><span class="m">73</span>.89702447700256,<span class="w"> </span><span class="m">74</span>.0759316909971,<span class="w"> </span><span class="m">74</span>.918352623994,<span class="w"> </span><span class="m">74</span>.7120841670112<span class="o">]</span>
</pre></div>
<p>The example above uses a publicly-available dataset from Wikipedia (linked in the References).
I spent way too much time fighting <code>multiprocessing</code>, but I learned a lot along the way.
Always RTFM!
There are several caveats to the library, and it is essential to read the <a href="https://docs.python.org/3/library/multiprocessing.html#programming-guidelines">Programming Guidelines</a>.
One of the issues is how Python copies memory to child processes.
This serialization overhead is huge, so it is much better for your child processes to retrieve their own data.
When in doubt, measure (the <code>timeit</code> library is great for this)!
What does this have to do with monoids?</p>
<p>Since addition is a moniod, we can safely split the work up and aggregate the results.
It is also commutative which means that we don't even have to track the order of the results.
Not all operations are commutative, though!
If you are multiplying a bunch of matrices, you can still batch the computation, but you <em>must</em> track the order of the results.</p>
<h5 id="The-right-tool-for-the-right-job">The right tool for the right job<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#The-right-tool-for-the-right-job">¶</a></h5><p>Know your use case.
I am an avid Pythonista, but sometimes alternatives might be the better choice.
Gloang doesn't read <code>all.txt</code> any faster, but it performs the summation in milliseconds instead of seconds!
Julia as native support for <a href="https://docs.julialang.org/en/v1/manual/parallel-computing/">multi-core processing</a>, and there are other languages that do as well.</p>
<h4 id="And-conquer">And conquer<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#And-conquer">¶</a></h4><p>This divide-and-conquer concept is critical to well-known large-scale algorithms like <a href="https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html">MapReduce</a>.</p>
<p>It doesn't mean we can't do it ourselves, though!
Libraries like <code>multiprocessing</code> allow Python developers to delegate tasks among different cores or processors.
Keep in mind that just like delegating tasks to friends, there is a computational overhead to splitting up tasks.
It is important to understand that libraries like <a href="https://numpy.org/"><code>Numpy</code></a> don't <a href="https://scipy.github.io/old-wiki/pages/ParallelProgramming">get you around the single-core limit</a> even though it is implemented in C.
Do your research!
Know your data and the problem domain before you start concatenating small strings across processes! 😜</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="BLS-data">BLS data<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#BLS-data">¶</a></h4><p>Let's look at some data freely available from the Bureau of Labor Statistics.
I downloaded the flat files from the link in the references and created some <code>dataclasses</code> to support our operations.
This is a quick-and-dirty module.
I'm not doing any type checking or validation in an effort to minimize boilerplate and get to the essence of the post.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [1]:</div>
<div class="inner_cell">
<div class="input_area">
<div class="highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">partial</span><span class="p">,</span> <span class="n">reduce</span>
<span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">asdict</span>
<span class="nd">@dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">BLSRecord</span><span class="p">:</span>
<span class="w"> </span><span class="sd">"""BLS table entry"""</span>
<span class="n">series_id</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">year</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">period</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">value</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">footnote_codes</span><span class="p">:</span> <span class="nb">str</span>
<span class="nd">@dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">BLSSeries</span><span class="p">:</span>
<span class="w"> </span><span class="sd">"""BLS series entry"""</span>
<span class="n">series_id</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">area_code</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">item_code</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">series_title</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">footnote_codes</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">begin_year</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">begin_period</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">end_year</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">end_period</span><span class="p">:</span> <span class="nb">str</span>
<span class="k">def</span> <span class="nf">read_with_header</span><span class="p">(</span><span class="n">fname</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">rec_type</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Read BLS tables with header row"""</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">fname</span><span class="p">)</span> <span class="k">as</span> <span class="n">fp</span><span class="p">:</span>
<span class="n">items</span> <span class="o">=</span> <span class="n">fp</span><span class="o">.</span><span class="n">readlines</span><span class="p">()</span>
<span class="c1"># Skip the header row, split, strip, and instantiate the dataclass</span>
<span class="k">return</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">item</span><span class="p">:</span> <span class="n">rec_type</span><span class="p">(</span><span class="o">*</span><span class="nb">map</span><span class="p">(</span><span class="nb">str</span><span class="o">.</span><span class="n">strip</span><span class="p">,</span> <span class="n">item</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">"</span><span class="se">\t</span><span class="s2">"</span><span class="p">))),</span> <span class="n">items</span><span class="p">[</span><span class="mi">1</span><span class="p">::])</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Composition">Composition<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Composition">¶</a></h4><p>When dealing with business logic, we are often faced with brittle code when requirements change and that massive <code>if</code>/<code>else</code> block just keeps growing.
Even worse is when you realize that similar logic is needed in multiple parts of the code and it's unclear how to "glue" the pieces together.</p>
<p>Here is a much less trivial example.
One of the key axioms is that we are dealing with <strong>binary operations</strong> that map some type to the same type.
Does this type have to be a number or a string?
Not at all!
Let's reconsider <code>sum</code> on integers; it takes two integers and returns an integer.
Replace integer with "numeric type", and we can still play ball.
Can we go further?
Here is the formula for standard deviation:</p>
$$\sigma = \sqrt{\frac{\sum_{i=1}^N (x_i - \bar{x})^2}{N-1}}$$<p>Look at the summation $\sum_{i=1}^N (x_i - \bar{x})^2$ and break it down a little.
Couldn't we also say $\sum_{i=1}^N func()$?
In other words, instead of passing some numeric type to <code>add</code>, I want to pass a <em>function</em> that <strong>returns</strong> a numeric type to <code>add</code>.
This is a perfectly valid construction!
The cool thing is that we can <em>compose</em> functions that return numeric types with the <code>add</code> binary operation.</p>
<p>We can use this idea with boolean operations to compose data queries instead of using an endless string of <code>if X and Y and Z</code> where <code>X</code>, <code>Y</code>, and <code>Z</code> tend to get lengthy in my experience.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [24]:</div>
<div class="inner_cell">
<div class="input_area">
<div class="highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">is_coffee_lookup</span><span class="p">(</span><span class="n">rec</span><span class="p">:</span> <span class="n">BLSRecord</span><span class="p">,</span> <span class="n">lookup</span><span class="p">)</span> <span class="o">-></span> <span class="nb">bool</span><span class="p">:</span>
<span class="k">return</span> <span class="n">series_data</span><span class="p">[</span><span class="n">rec</span><span class="o">.</span><span class="n">series_id</span><span class="p">]</span><span class="o">.</span><span class="n">item_code</span> <span class="o">==</span> <span class="s2">"717311"</span>
<span class="k">def</span> <span class="nf">is_january</span><span class="p">(</span><span class="n">rec</span><span class="p">:</span> <span class="n">BLSRecord</span><span class="p">)</span> <span class="o">-></span> <span class="nb">bool</span><span class="p">:</span>
<span class="k">return</span> <span class="n">rec</span><span class="o">.</span><span class="n">period</span> <span class="o">==</span> <span class="s2">"M01"</span>
<span class="k">def</span> <span class="nf">is_not_1995</span><span class="p">(</span><span class="n">rec</span><span class="p">:</span> <span class="n">BLSRecord</span><span class="p">)</span> <span class="o">-></span> <span class="nb">bool</span><span class="p">:</span>
<span class="k">return</span> <span class="n">rec</span><span class="o">.</span><span class="n">year</span> <span class="o">!=</span> <span class="s2">"1995"</span>
<span class="k">def</span> <span class="nf">intersect_identity</span><span class="p">(</span><span class="n">rec</span><span class="p">:</span> <span class="n">BLSRecord</span><span class="p">)</span> <span class="o">-></span> <span class="nb">bool</span><span class="p">:</span>
<span class="c1"># The "natural element" for intersections is True</span>
<span class="k">return</span> <span class="kc">True</span>
<span class="k">def</span> <span class="nf">intersect</span><span class="p">(</span><span class="n">fst_fn</span><span class="p">,</span> <span class="n">snd_fn</span><span class="p">):</span>
<span class="k">return</span> <span class="k">lambda</span> <span class="n">rec</span><span class="p">:</span> <span class="n">fst_fn</span><span class="p">(</span><span class="n">rec</span><span class="p">)</span> <span class="ow">and</span> <span class="n">snd_fn</span><span class="p">(</span><span class="n">rec</span><span class="p">)</span>
<span class="c1"># Create a lookup table for series entries</span>
<span class="n">series_data</span> <span class="o">=</span> <span class="p">{</span><span class="n">row</span><span class="o">.</span><span class="n">series_id</span><span class="p">:</span> <span class="n">row</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">read_with_header</span><span class="p">(</span><span class="s2">"data/ap.series"</span><span class="p">,</span> <span class="n">BLSSeries</span><span class="p">)}</span>
<span class="c1"># Inject our state</span>
<span class="n">is_coffee</span> <span class="o">=</span> <span class="n">partial</span><span class="p">(</span><span class="n">is_coffee_lookup</span><span class="p">,</span> <span class="n">lookup</span><span class="o">=</span><span class="n">series_data</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"All records: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">price_data</span><span class="p">)</span><span class="si">:</span><span class="s2">,</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="c1"># Filter for entries that are coffee</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Coffee: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="n">is_coffee</span><span class="p">,</span><span class="w"> </span><span class="n">price_data</span><span class="p">)))</span><span class="si">:</span><span class="s2">,</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="c1"># Filter for entries that are in January</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"January: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="n">is_january</span><span class="p">,</span><span class="w"> </span><span class="n">price_data</span><span class="p">)))</span><span class="si">:</span><span class="s2">,</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="c1"># Compose and profit!</span>
<span class="n">coffee_and_january</span> <span class="o">=</span> <span class="n">reduce</span><span class="p">(</span><span class="n">intersect</span><span class="p">,</span> <span class="p">[</span><span class="n">is_coffee</span><span class="p">,</span> <span class="n">is_january</span><span class="p">],</span> <span class="n">intersect_identity</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Coffee and January: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="n">coffee_and_january</span><span class="p">,</span><span class="w"> </span><span class="n">price_data</span><span class="p">)))</span><span class="si">:</span><span class="s2">,</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="c1"># We can compose any number of functions that are valid Moniods!</span>
<span class="n">coffee_and_january_not_1995</span> <span class="o">=</span> <span class="n">reduce</span><span class="p">(</span><span class="n">intersect</span><span class="p">,</span> <span class="p">[</span><span class="n">is_not_1995</span><span class="p">,</span> <span class="n">is_coffee</span><span class="p">,</span> <span class="n">is_january</span><span class="p">],</span> <span class="n">intersect_identity</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Coffee and January and not 1995: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="n">coffee_and_january_not_1995</span><span class="p">,</span><span class="w"> </span><span class="n">price_data</span><span class="p">)))</span><span class="si">:</span><span class="s2">,</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>All records: 177,894
Coffee: 786
January: 15,042
Coffee and January: 66
Coffee and January and not 1995: 62
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h4 id="Conclusion">Conclusion<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Conclusion">¶</a></h4><p>I hope I didn't chase too many rabbits; there are plenty to chase on this topic!
I also hope that this has sparked some ideas and will help you identify common patterns in software that will make your coding a bit more efficient and clean.</p>
<h3 id="References">References<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#References">¶</a></h3><ul>
<li>Weisstein, Eric W. "Monoid." From MathWorld--A Wolfram Web Resource. <a href="https://mathworld.wolfram.com/Monoid.html">https://mathworld.wolfram.com/Monoid.html</a></li>
<li><a href="https://en.wikipedia.org/wiki/Monoid">https://en.wikipedia.org/wiki/Monoid</a></li>
<li><a href="https://wiki.haskell.org/Monoid">https://wiki.haskell.org/Monoid</a></li>
<li><a href="https://github.com/dry-python/functional-jargon-python">https://github.com/dry-python/functional-jargon-python</a></li>
<li><a href="https://planetmath.org/examplesofnoncommutativeoperations">https://planetmath.org/examplesofnoncommutativeoperations</a></li>
<li><a href="https://blogs.ncl.ac.uk/andreymokhov/united-monoids/">https://blogs.ncl.ac.uk/andreymokhov/united-monoids/</a></li>
</ul>
<h3 id="Data">Data<a class="anchor-link" href="https://blog.wificidr.net/posts/monoids-for-pythonistas/#Data">¶</a></h3><ul>
<li><a href="https://download.bls.gov/pub/time.series/ap">https://download.bls.gov/pub/time.series/ap</a></li>
<li><a href="https://archive.org/details/wikipedia_visitor_stats_200906">https://archive.org/details/wikipedia_visitor_stats_200906</a></li>
</ul>
</div>
</div>
</div>category-theorymathpythonhttps://blog.wificidr.net/posts/monoids-for-pythonistas/Mon, 05 Oct 2020 12:30:10 GMTCircles and Parabolashttps://blog.wificidr.net/posts/parabola/circles-and-parabolas/Daniel Justice<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="A-few-notes-on-circles-and-tangents">A few notes on circles and tangents<a class="anchor-link" href="https://blog.wificidr.net/posts/parabola/circles-and-parabolas/#A-few-notes-on-circles-and-tangents">¶</a></h2><p>I have been <em>very</em> slowly making my way through Paul Lockhart's "Measurement". It is a fantastic book, and I highly recommend it for anyone who enjoys math as a hobby or pastime. One of the questions posed is to find the radius of the smaller, nested circle given the radius of the two larger circles (all tangent to each other and the line below them).</p>
<p><img alt="" src="https://upload.wikimedia.org/wikipedia/commons/thumb/5/54/Sangaku_Circles.JPG/514px-Sangaku_Circles.JPG">
<small>Wikipedia</small></p>
<p>I struggled with this problem for much longer than I care to admit, and I struggled with whether or not to even write this post. I'm not terribly clever with writing, or math for that matter, but I hope others who are slogging their way through work or school will find a bit of encouragment by the end.</p>
<h3 id="Lessons-learned">Lessons learned<a class="anchor-link" href="https://blog.wificidr.net/posts/parabola/circles-and-parabolas/#Lessons-learned">¶</a></h3><p>The internet is at most of our fingertips, and it is much too easy to search for the answers to all of our questions. I think much of my own personal struggle in mathematics has been due to the fact that I am lazy. It's true. I'd much rather watch <a href="https://www.youtube.com/channel/UC1_uAIS3r8Vu6JjXWvastJg">Mathologer</a> perform magic in front of my eyes than actually <strong>do</strong> math. However, just like exercising, finding my way to the end usually provides the motivation to continue the next time around. I hope through continued practice and discovery that I will build up the skills and tools to make this a little less painful as I attempt other problems. Think of watching Danny MacAskill doing amazing tricks on his bicycle. What we don't see are the endless hours of <strong>practice</strong>, and <strong>repeated failure</strong> that led to amazing moments. This is true of every profession, but I seem to need to remind myself of it daily!</p>
<h3 id="Two-circles-tangent-to-a-line">Two circles tangent to a line<a class="anchor-link" href="https://blog.wificidr.net/posts/parabola/circles-and-parabolas/#Two-circles-tangent-to-a-line">¶</a></h3><p>The way I attacked this problem may be the most naive thing you have ever seen, but I made some personal discoveries along the way that I haven't seen in the textbooks at my disposal. Let's start with two circles tangent to each other and a horizontal line. Grab a piece of scrap paper and draw a few examples. Fix the radius of one circle, and vary the radius of the other. I have a stencil with various sizes of circles on it. I found it much easier to use for this problem than simply relying on a compass. The traditional approach to this problem is to use triangles and a few applications of the Pythagorean theorum. Don't forget my disclaimer about a naive approach!</p>
<p>Do you notice a pattern drawn by the centers of the tangential circles? They follow a quadratic curve (parabola) away from the fixed circle. After a little poking around in an algebra book, I quickly realized that given a fixed circle, the centers of the tangential circles all lie on a parabola where the fixed circle sits at the focus! Check out the Desmos link below for a demo.</p>
<p><a href="https://www.desmos.com/calculator/kgnhqs8b5r">Two circles tangent to one another</a><br>
Move the slider around to see what happens.</p>
<p>If we think about the <a href="https://www.purplemath.com/modules/parabola.htm">definition of a parabola</a>, this pattern makes sense. The vertex sits midway between the focus and the directrix. The directrix is a fixed line, and a "shortest line" between a point on the parabola and the directrix always hits the directrix at a 90-degree angle. The focus is a different case as it is a fixed point, not a line. This means a "shortest line" from the focus to a point on the parabola is a matter of connecting the dots at whatever angle they make.</p>
<p>What if we identify all the "fixed" parts of this setup? The distance between the focus and the vertex is the same as the distance between the vertex and the directrix. If we draw a line through the vertex parallel to the directrix, we've identified all the points this fixed distance from the directrix. What about the focus? Let's do the same thing. All the points at this fixed distance around the focus forms a circle lying in the "cup" of the parabola... and tangent to it. You should be noticing the same pattern here that we saw earlier. The reason is pretty neat (at least to my pea brain). By identifying a set of fixed points, we are left with the difference between the circle around the focus and the line through the vertex - and tangent to both!</p>
<p><img alt="" src="https://blog.wificidr.net/images/parabola-cir.png"></p>
<p>Let's use the previous image as an example. I have identified all of our "fixed points" (the black circle around the focus and the purple line along the x-axis) that are the focal distance from either the focus or the directrix. If we consider the point (4, 4), it's "shortest line" to the directrix is straight down and interesecting the directrix at a perpedicular angle. It is also a distance of 4 from the purple line through the vertex (and a distance of 5 to the directrix). What about the "shortest line" between the focus (0, 1) and (4, 4)? Well, the horizontal distance is 4, and the vertical distance is 3. This makes a 3-4-5 triangle, so it is 5 units away from the focus. That means it is also only 4 units from the circle around the focus. This means we can sweep a line of length 4 around the point (4, 4) and get a circle that intersects both the circle around the focus <strong>and</strong> the line through the vertex. Cool, right?!</p>
<p>To solve this problem I noticed that what is true for one circle is also true for the other. The nested, inner cicle lies at the intersection of the parabolas formed by the two bigger circles. The algebra gets a little messy since you have to locate one of the parabolas in terms of the other, but the math does work out. I'm going to present an interactive example instead of a mess of LaTex since the geometry is really the point of this post.</p>
<p><a href="https://www.desmos.com/calculator/77xvzr94v1">The solution</a><br>
<img alt="" src="https://blog.wificidr.net/images/parabola-cir-demo.png"><br>
Move the slider to see the solution adjust to changing conditions.</p>
<h3 id="Why-does-it-matter?">Why does it matter?<a class="anchor-link" href="https://blog.wificidr.net/posts/parabola/circles-and-parabolas/#Why-does-it-matter?">¶</a></h3><p>It really doesn't at the end of the day, but I learned something from it even if it was a really long-winded way around to a much simpler problem. It got me to poking around with parabolas quite a bit even though that wasn't even the point of the question... or was it? That is the beauty of mathematics to me. The breadth of material is vast to say the least, but interconnected in interesting and beautiful ways!</p>
</div>
</div>
</div>mathhttps://blog.wificidr.net/posts/parabola/circles-and-parabolas/Wed, 30 Jan 2019 01:04:39 GMTProject Euler #6https://blog.wificidr.net/posts/project-euler-6/project-euler-6/Daniel Justice<div class="cell border-box-sizing text_cell rendered"><div class="prompt input_prompt">
</div><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Project-Euler-problem-6">Project Euler problem 6<a class="anchor-link" href="https://blog.wificidr.net/posts/project-euler-6/project-euler-6/#Project-Euler-problem-6">¶</a></h3><p>I stumbled across Project Euler (at <a href="https://www.hackerrank.com/contests/projecteuler/challenges/euler006">HackerRank</a> or at <a href="https://projecteuler.net/">Project Euler</a>) a while back, and it has been a low item on my to-do list for some time. It can be quite frustrating at first, but I find it rewarding each time I unlock a little piece of the mathematical puzzle. Of course you can cheat, but that completely defeats the purpose and harms no one but yourself. I hope I can demonstrate why with this example.</p>
<ul>
<li>On a side note, HackerRank typically penalizes you for brute force methods. They force you to really think about the problems.</li>
</ul>
<p><a href="https://projecteuler.net/problem=6">Problem 6</a> asks, "Find the difference between the sum of the squares of the first one hundred natural numbers and the square of the sum." In other words, what is
$$ \big|\big(\sum_1^n n\big)^2 - \big(\sum_1^n n^2\big)\big| $$</p>
<p>I was already familiar with <a href="http://mathandmultimedia.com/2010/09/15/sum-first-n-positive-integers/">Gauss's method</a> of summing the first series of natural numbers. What I did not know how to do was to sum their squares. The brute force method will yield a result, but HackerRank will penalize you with a timeout if you go that route. I chose to dig deeper...</p>
<h4 id="A-row-of-boxes">A row of boxes<a class="anchor-link" href="https://blog.wificidr.net/posts/project-euler-6/project-euler-6/#A-row-of-boxes">¶</a></h4><p>Nothing beats a little chair time with pencil and paper. I suspected a strong link between geometry and the answer to this problem, so I started drawing squares on my graph paper.</p>
<p><img alt="" src="https://blog.wificidr.net/images/boxes.png"><!-- .element height="50%" width="50%" --></p>
<p>If you are familiar with <a href="https://www.mathsisfun.com/algebra/triangular-numbers.html">triangle numbers</a>, you might notice the pattern of the right edge of each square. It is the series 1, 3, 6, 10, 15... It is really fascinating to me how these things relate to one another. The formula to find the nth triangle number is the same formula to find the sum of a series of natural numbers!<br>
$$ \sum_1^n n=\frac{n(n+1)}{2} $$</p>
<p>Therefore, to find the area of the enclosing rectangle, we simply multiply the sum from above by our 'n'. Now, we need to account for all those little strips of triangle numbers that are left over on top of the squares. I have to admit, I spent quite a bit of time on this one and simply got stuck. I knew I needed to find another sum, but this time the series is the sum of the triangle numbers themselves. I needed to find a formula for the series 1, 4, 10, 20, 35...</p>
<h4 id="Tetrahedral-Numbers">Tetrahedral Numbers<a class="anchor-link" href="https://blog.wificidr.net/posts/project-euler-6/project-euler-6/#Tetrahedral-Numbers">¶</a></h4><p>In an attempt to not ruin the learning process, I did an internet search for the mystery series. I immediately came across this article on <a href="https://en.wikipedia.org/wiki/Tetrahedral_number">tetrahedral numbers</a>. This was my extra quantity! To find the sum of a series of squares, subtract the (n - 1)th tetrahedral number from the area of the rectangle that overlays the boxes.</p>
<p>The tetrahedral numbers are easily visualized if you think of a triangular tower of spheres as in this image:<br>
<img alt="" src="https://upload.wikimedia.org/wikipedia/commons/8/87/Pyramid_of_35_spheres_animation_original.gif"></p>
<p><small>By Blotwell [GFDL (http://www.gnu.org/copyleft/fdl.html), CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/) or GPL (http://www.gnu.org/licenses/gpl.html)], from Wikimedia Commons</small></p>
<p>If you start taking apart the tower, the spheres perfectly fill in the gaps in the stack of rectangles above. If you get really bored, you can validate this like I did with a bunch of blocks and marbles. These items are readily available if you have kids. One last bit worth noting is that we only need (n - 1) tetrahedral numbers because we don't need to fill in the last row.</p>
<p>Finally, the sum of a series of squares can be found by:
$$ \sum_1^n n^2=n * \frac{n(n+1)}{2} - \frac{(n - 1)(n)(n+1)}{6} $$</p>
$$ =\frac{n^3+n^2}{2} - \frac{(n^2-n)(n+1)}{6} $$$$ =\frac{n^3+n^2}{2} - \frac{n^3+n^2-n^2-n}{6} $$$$ =\frac{3(n^3+n^2)}{6} - \frac{n^3-n}{6} $$$$ =\frac{3n^3+3n^2-n^3+n}{6} $$$$ \sum_1^n n^2=\frac{2n^3+3n^2+n}{6} $$<h4 id="Why-does-this-matter?">Why does this matter?<a class="anchor-link" href="https://blog.wificidr.net/posts/project-euler-6/project-euler-6/#Why-does-this-matter?">¶</a></h4><p>Not to sound overly cliché, but it is easy, often too easy, to find answers to all of our questions at our fingertips. For example, if you want to cut right to the chase, you could have a look at <a href="http://www.mathblog.dk/project-euler-problem-6/">this answer</a>. Be sure to take a look at the <a href="https://trans4mind.com/personal_development/mathematics/series/sumNaturalSquares.htm">proof</a> as well. Both posts are respectable, but I think they hide the intuition in the formulas.</p>
<p>In my search for an answer I learned something new, and I also saw the problem in a new light not illustrated at either of the previous links. Pardon my drawing skills, but imagine the previous image as a stack of cubes, all laid on top of one another pushed to one side. Here's a top-down view of what I'm describing:</p>
<p><img alt="" src="https://blog.wificidr.net/images/stacked_boxes.png"><!-- .element height="50%" width="50%" --></p>
<p>If you imagine the sum of squares as rows of stacked boxes, you can clearly see the relationship between the sum and the tetrahedral numbers from earlier (the missing space to be subtracted)!</p>
<p>I hope I have encouraged you to dig a little deeper the next time you run across a tricky math problem and to look for a deeper understanding of the fundamental forces that shape it.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [ ]:</div>
<div class="inner_cell">
<div class="input_area">
<div class="highlight hl-ipython3"><pre><span></span>
</pre></div>
</div>
</div>
</div>
</div>mathpythonhttps://blog.wificidr.net/posts/project-euler-6/project-euler-6/Tue, 23 Jan 2018 02:35:26 GMT