Functional Programming in Python

A small foray into functional programming in Python¶

This is an annotation of my own journey into functional programming. My goal is to write my thoughts down and share my experiences, not to make a technical argument about one set of technologies/techniques or another.

I have been studying category theory and functional programming as I continue along the path of trying to become a better developer. I don't have a degree in mathematics, nor do I have a hard-core computer science background, so many of these concepts fall into my personal bucket of "non-trivial" despite what the egg-heads on Quora say about it. People like Bartosz Milewski are beating the drum that we're doing it all wrong and using the wrong abstractions.

Queue up Raymond Hettinger beating the podium and yelling "There must be a better way!"...

I don't want to mislead my tiny pool of readers; I'm not here to weigh in on that discussion. As Hillel Wayne says, we should exercise caution in the absence of empirical evidence.

One thing that I do find interesting and potentially very useful is the application of monads with I/O and Optional (maybe) data types. I have found myself increasingly frustrated with Python's type system and error-handling, and I have been seeking "the better way". Is the better way another language?

They all suck
Most all of them kept around nil/None

I don't believe another language is the answer, but more on that later. Is it even possible to have a functional programming discussion without mentioning Haskell? Let's not risk it and go ahead and offer our sacrifice to the FP gods. I want to get my IP address from a web service.

Get my IP in Haskell¶

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE RecordWildCards #-}

module Main where

import           Control.Exception (try)
import           Data.Aeson
import qualified Data.Text as Text
import           Network.HTTP.Simple

data IpAddrData = IpAddrData { ip :: Text.Text
                              , ip_decimal :: Int
                              , city :: Text.Text
                              , country :: Text.Text
                              } deriving (Show)

instance FromJSON IpAddrData where
  parseJSON (Object v) = IpAddrData <$> v .: "ip" <*> v .: "ip_decimal" <*> v .: "city" <*> v .: "country"

instance ToJSON IpAddrData where
  toJSON IpAddrData {..} = object ["ip" .= ip, "ip_decimal" .= ip_decimal, "city" .= city, "country" .= country]

formatCityCountry :: IpAddrData -> Text.Text
formatCityCountry a = Text.concat [city a, ", ", country a]

main :: IO ()
main =
    try (httpJSON "https://whatsmyip.ovh/json")
    >>= \eresponse -> case eresponse of
        Left e -> print (e :: HttpException)
        Right r -> print (formatCityCountry (getResponseBody r :: IpAddrData))

Put down the Kool-Aid, folks¶

First, I want to dispell some myths. I have to admit that I had some pretty naïve views of Haskell when I started messing with it. My ignorance is probably shining through here, but I'm only human. You see that try bit in the Haskell code? Haskell might be "pure", but it isn't magic! We still have to deal with runtime exceptions. Second, it's typed and checked at compile time, right? Our program should be "correct"!

stack build  --ghc-options="-XOverloadedStrings -XRecordWildCards" && echo $?
0

Drum roll...

stack exec fp-exe
fp-exe: JSONParseException Request {
  host                 = "whatsmyip.ovh"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept","application/json")]
  path                 = "/ip"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}, responseVersion = HTTP/1.1, responseHeaders = [("Server","nginx"),("Date","Sat, 30 May 2020 22:59:44 GMT"),("Content-Type","text/plain; charset=utf-8"),("Content-Length","16"),("Connection","keep-alive"),("Strict-Transport-Security","max-age=31536000")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}) (ParseError {errorContexts = [], errorMessage = "endOfInput", errorPosition = 1:8 (7)})

What happened?! I snuck in there and pointed it to a different endpoint that doesn't return JSON data, so the parser died. And yes, I must admit that I purposefully ommitted -Wall -Werror... But those still would not have saved us - the error is in the construction of the program, not the language. Is there a way around that? Yes, of course there is, but we're not here to deep-dive into Haskell - what happend to a Python article?

Functional Python¶

There are a lot of cool tools in the functools library. Many Pythonistas are also aware of the map, reduce, and filter methods taught in many Python FP articles. This is barely the tip of the iceberg. Some very smart folks at Dry-labs have brought monads to Python. They're not the first to try it, but it is one of the most complete libraries I have seen.

Get my IP in Python¶

from pydantic import BaseModel

import requests

from returns.context import RequiresContextIOResultE
from returns.functions import tap
from returns.io import IOResultE, impure_safe

import typing as t


class OVHIpAddr(BaseModel):
    ip: str
    ip_decimal: int
    country: str
    city: str


def get_ip_addr(url: str) -> RequiresContextIOResultE[t.Any, OVHIpAddr]:
    """Get our IP address from a service"""

    @impure_safe
    def inner(session: t.Any) -> OVHIpAddr:
        resp = requests.get(url)
        return OVHIpAddr(**resp.json())

    return RequiresContextIOResultE(inner)


def format_city_country(data: OVHIpAddr) -> str:
    """Format the city and country"""
    return f"{data.city}, {data.country}"


if __name__ == "__main__":
    session = requests.Session()
    (
        get_ip_addr("https://whatsmyip.ovh/json")
        .map(format_city_country)
        .map(tap(print))(session)
    )

Still broken¶

Finally, we get to the meat of my rambling. The Python code above has just about the exact same failure modes as the Haskell snippet before, and the fix is the same in both cases. We need to deal with the possibility that the JSON parser poops the bed. If you actually run the code above, you'll notice that it doesn't die on bad JSON input. That is because the parsing is wrapped up in impure_safe... but we violated single responsibility in the process. Does that matter? What if we add print(resp.not_an_attr)? The exception is swallowed by the wrapper, and we spent 10 minutes trying to track down a silent error with little to no feedback from the code. TL;DR; the language didn't save us. FP didn't save us.

Wait a sec, your design sucks¶

Absolutely, I agree. And in fairness to FP, Either and Maybe are elegant solutions to a tangled chain of risky function calls. The techniques have to be wielded in an intelligent manner to produce meaningful results!

Functional my Python¶

So let's apply FP to all of our Python codes! Unless you are familiar with dry-python's returns package, that last chunk of code is probably a lot of wat. This isn't a tutorial on monads, but the code above basically wraps an I/O-bound call in a container that can be composed with other functions. returns enables us to practice a concept called Railway-oriented programming. Before we rewrite all of our Python repositories, let's consider...

The good¶

returns has great documentation. It is still an early-stage project and is under rapid development, and I am impressed with their commitment to testing and documentation. They are very responsive to issues and questions. Maybe has a great use case when dealing with unpredictable dictionaries or class instances. Some libraries use a bit of magic to create dynamic objects from JSON responses, and nested key/property lookups can be a nightmare of subtle errors.

attr = monsters.get("monster", {}).get("arm").get("fingers", {}).count
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'get'
>>>

Dang it, I forgot to pass in a default argument!

>>> monster_mash().map(lambda mash: mash.get("monster")).map(lambda monster: monster.get("arm")).map(lambda arm: arm.get("fingers")).map(lambda fingers: getattr(fingers, "count"))
<returns.maybe._Nothing object at 0x7fe06938a370>
>>>

See! Isn't that better! We now have to explicitly deal with a Just value or Nothing.

Why are you looking at me so skeptically? Why are you pointing and shaking your head? Is it just me, or do you feel like we just hammered a big square block through a round hole? A coworker asked me to look at a script that was failing on him, and it looked like a great opportunity to go way down the Python/FP rabbit hole. You can see for yourself how it turned out.

The bad¶

Honestly, where to start? Python isn't a functional language and doesn't have very elegant mechanisms for handling monadic constructs. I went through a phase where I thought "Pythonic" was a great way to code. Don't get me wrong, I still try to follow those principles when and where it makes sense. I think they are good guidelines for people starting off on their development journey, but the guard rails turn into barriers in some instances. On the flip side, if you crash through the barriers and chart your own course, the results can be messy. I do believe that libraries like returns have a place in our toolbox, and I am excited to follow the project as it evolves - and how it deals with being that square peg.

The ugly¶

This is an important point, and a good example of what makes dynamic languages both awesome and horrifying. We can use frameworks to write code that is almost unreadable to our fellow Python developers. To add insult to injury, that block was formatted with black, so have fun trying to put comments in sane places so others can follow what the heck is going on!

And type annotations in Python. 80% of the time, Mypy works all the time. Another feature that I find awesome and infuriating in alternating 15-minute blocks! Here is a story about functional programming and type annotations in Python.

A team built a road through a winding mountain pass and a long valley below. You can drive a lot of different types of vehicles on this road, some small and fast, others large and slow. As time goes on, people realize that accidents often happen near the sharp turns and cliffs. Some more people came along and decided to add guard rails in certain parts of the road, but the road wasn't originally designed for them. Unfortunately, this means that most of the guard rails ended up along the straight and wide parts of the road.

Libraries like returns will help in the general case, but the thorniest corner cases will always be the hardest to solve. FP libraries and type annotations cannot solve the foot-guns that are built into the language. Python doesn't have "compile-time" checks baked in, and there is nothing stopping a desperate developer on a short timeline from accessing unsafe_obj._inner_value.

<\rant>

In fairness, the road described earlier is still under active development, and those guard rails are getting better as time goes on.

Dict annotations 🐿️¶

A note about ♥♥Pydantic♥♥... Go through all of your typed Python code and replace every. single. instance. of Dict[str, Dict[int, Dict[Optional[omg...]]]] with something that inherits from BaseModel. You're welcome; your hair might start growing back. For the love of sanity at least use data classes or attrs or NamedTuple. Don't worry about dependencies, our Python containers are YUGE!

Never return a bare tuple from a method. It seems simple, oh it's obvious what that does, but be a good teammate and don't make them guess.

Is it Python... or me?¶

Remember the languages question from the opening? They all suck! Python was my ticket to success about ten years ago and has been an integral part of my career growth; I would be a fool to say otherwise. Just look at how many of these cool network automation projects are based on the language. Designing a "correct" program is really hard. Even Haskell, that "pure" language that is supposed to save us from our stupidity, has a really hard time dealing assertions about natural numbers. Look and see how many tutorials copy and paste the same safeDiv method to demonstrate the Maybe monad. C'mon, it's just a pesky little zero in the middle of infinity! Apparently, dealing with partial functions isn't easy after all.

The right tool for the right job¶

Learning different styles and techniques has made me a better developer. We can take these concepts and apply them in different ways in several different languages. Just [favorite search engine] "monads in [some-programming-language]", and you will see some really interesting links.

Python is an awesome language. I still enjoy using it daily to keep food on the table, but I am branching out more. I may have poked at Haskell a bit, but I am trying to learn it and be productive with its tools. It is a fascinating language that will flip your OOP brain on its lid, but it's not the end-all, be-all!

Some may disagree, but I find it really hard to build small Python containers.

341M    /usr/local/lib/python3.7/

So maybe Python isn't the best choice for that new microservice that you know is going to have several dependencies. Or maybe your requirements are well-defined and you can keep the scope slim getting away with a distroless container. Maybe you're inheriting a project written in a language designed by mean people, and you just have to roll with it. Who knows, but you have to make a choice with few clear-cut options.

I am loathe to wade into language wars. I have seen really cool, massively functional projects in languages that appear to be horrible mismatches for the task at hand. Are four thousand issues a language problem? Their main competitor isn't doing much better. Are those really language issues? Nah, VS Code has them both beat.

What can I say, software is hard.

It's all about design¶

As I said, this post is all about me getting my thoughts written down. I really don't have much of a point to make other than I'm trying to do my part to avoid the software apocalypse. Design is critical to writing correct programs, but is it "what it's all about"?

A former manager reminds me pretty often that "it" is about people. I spend most of my time writing tools that other people consume; I rarely "eat my own dog food". It is too easy to get bogged down in the technical minutae and day-to-day fire drills. To me, writing good software is about being a better teammate. I am embarassed every time someone brings me a bug, and even more so if it took two or three tries to fix it. Test coverage won't save me here, nor will the hottest trends. I don't know what the answer is, but I'm trying to do better.

My preferred guardrails¶

So what is in my toolbox? How do I try to stay sane and craft good code? Glad you asked!

For linting, Black is it. Agree with your team on a line legth, and autoformat. I don't pass CI on bad formatting - just do it.

Pylama - great baseline sanity checks and lots of plugins.
Pytest - test all the things.
Mypy - check as many types as you can...
pytype - and infer the rest when you can't.
CodeCov - Keep that number going up!
faker - Super convenient random inputs

And what's on my radar¶

I really need to review hypothesis and figure out where I can be using it.

TLA+ and formal methods. I have Hillel's book; I need to force myself to spend the time to finish it.

Conclusion¶

Make it a discussion! Feel free to drop me a line on LinkedIn or harass me in an issue!