Deliberate Software

With deliberate practice come consistent results

Haskell Is the Dark Souls of Programming

HUMOROUS POST AHEAD. Please don’t hit me, Haskell does a great job of that already.

solaire

I decided to start the next version of my safety score posts. This time, however, I decided to do it in Haskell. I love Haskell for the same reasons I love Dark Souls. Fantastic and inscrutable lore, a great combat type system, a cliff-wall difficulty curve, and unending punishment.

I want to collect some statistics from the GitHub API. Watch as I retrace my steps attempting the Tomb of the Dread HTTPS GET Request.

Step One - Stack (aka Pride Comes Before The Fall)

I download stack and start a project:

1
2
3
4
> cd /home/jack/programming && stack new github-stats && cd github-stats
Downloading template "new-template" to create project "github-stats" in github-stats/ ...
 ......
All done.

So far so good. Does it work?

1
2
3
4
5
  > stack build && stack exec -- github-stats-exe 
   github-stats-0.1.0.0: configure
   ..... 
   Registering github-stats-0.1.0.0...
   someFunc

Awww yisss. This is going to be so easy!

Step Two - HTTPS GET Request (aka The Fall After The Pride)

giants

Now I need to query the GitHub API. Not my first time to the rodeo, I generate a personal access token from GitHub and copy it to a local file. What query should I run first? How about the count for all ASM tetris repositories? Poking around the docs comes up with:

1
2
3
GET https://api.github.com/search/repositories?q=tetris+language:assembly&sort=stars&order=desc
User-Agent: steveshogren
Authorization: token PUT_TOKEN_HERE

{.. “total_count”: 354}

Easy life. Now how do you GET a resource in Haskell? Ah, Network.HTTP! I copy the front page sample into src/Lib.hs

1
2
3
4
5
6
7
8
9
module Lib
    ( someFunc
    ) where

x = simpleHTTP (getRequest "https://www.github.com/") >>= fmap (take 100) . getResponseBody

someFunc :: IO ()
someFunc =
   print x

So simple! This is why laugh at my NodeJS loving friends! What a bunch of cretins.

1
2
3
4
5
> stack build
src/Lib.hs:5:5: Not in scope: simpleHTTP
src/Lib.hs:5:17: Not in scope: getRequest
src/Lib.hs:5:77: Not in scope: getResponseBody
Compilation failed.

Doesn’t compile. Durp, hackage is a package library, I need to add this to my cabal. What is the name of the package? HTTP-4000? HTTP-4000.3.2? Nothing in hackage seems to indicate what goes into the cabal file. I discover it is just HTTP through trial and error. I update my cabal file… in all three build-depends…?

1
2
  build-depends:       base >= 4.7 && < 5
                       , HTTP

Hrm, same error.

1
2
3
4
5
> stack build
src/Lib.hs:5:5: Not in scope: simpleHTTP
src/Lib.hs:5:17: Not in scope: getRequest
src/Lib.hs:5:77: Not in scope: getResponseBody
Compilation failed.

Oh, durp, I’d need an import. (WHY ISN’T THIS IN THE CODE SAMPLE?!) Also, print doesn’t work, I need putStrLn.

1
2
3
4
5
6
import Network.HTTP

x = simpleHTTP (getRequest "https://www.github.com/") >>= fmap (take 100) . getResponseBody

someFunc :: IO ()
someFunc = x >>= putStrLn

Here goes!!!

1
2
 > stack build && stack exec -- github-stats-exe
github-stats-exe: user error (https not supported)

Wat. Further inspection of the docs shows a line WAAY DOWN in paragraph 5.

NOTE: This package only supports HTTP;

giants

When playing Dark Soulsprogramming Haskell, sometimes the best move is to run away. I search again. haskell https request returns “http-conduit” as the best choice. After adding http-conduit to my cabal, I come up with this beast without any surprises:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
query :: IO String
query = do
    initReq <- parseUrl "https://api.github.com/search/repositories"
    let r = initReq
                   { method = "GET"
                    , requestHeaders = [(hUserAgent, "steveshogren")
                                      , (hAuthorization, "token PUT_TOKEN_HERE")]}
    let request = setQueryString [("q", Just "tetris+language:assembly")
                                 ,("order", Just "desc")
                                 ,("sort", Just "stars")] r
    manager <- newManager tlsManagerSettings
    res <- httpLbs request manager
    return . show . responseBody $ res

someFunc :: IO ()
someFunc = do
   query >>= putStrLn

Huzzah! Results! I’m getting back a monster string of json data.

“\”{\\“total_count\\”:66, ….}\"

solaire

Step Three - Parsing JSON

Time to parse this mega JSON string. Aeson seems to be the biggest contender. To use Aeson and get the total_count value from the return, I needed the following additions:

1
2
3
4
5
6
7
8
9
10
11
12
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}

import GHC.Generics
import Data.Aeson

data ResultCount = ResultCount {
  total_count :: Int }
  deriving (Generic, Show)

instance ToJSON ResultCount
instance FromJSON ResultCount

ResultCount allows me to use decode from aeson instead of show to parse the “total_count” from the JSON response into an Int. Sure enough, it does!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}
module Lib
    ( someFunc
    ) where

import Control.Monad
import Network
import Network.HTTP.Conduit
import Network.HTTP.Types.Header
import GHC.Generics
import Data.Aeson

data ResultCount = ResultCount {
  total_count :: Int }
  deriving (Generic, Show)

instance ToJSON ResultCount
instance FromJSON ResultCount

query :: IO (Maybe Int)
query = do
    initReq <- parseUrl "https://api.github.com/search/repositories"
    let r = initReq
                   { method = "GET"
                    , requestHeaders = [(hUserAgent, "steveshogren")
                                      , (hAuthorization, "token PUT_TOKEN_HERE")]}
    let request = setQueryString [("q", Just "tetris+language:assembly")
                                 ,("order", Just "desc")
                                 ,("sort", Just "stars")] r
    manager <- newManager tlsManagerSettings
    res <- httpLbs request manager
    return . liftM total_count . decode . responseBody $ res

someFunc :: IO ()
someFunc = query >>= print

Puts out: Just 66. Success! Wait. 66 isn’t the same count I got when running from the browser. Check again. Sure enough, browser comes up with a totally different count.

solaire

Maybe the query request isn’t correct? Adding a print request on line 31 after building the request shows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Request {
  host                 = "api.github.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("User-Agent","steveshogren"),("Authorization","token PUT_TOKEN_HERE")]
  path                 = "/search/repositories"
  queryString          = "?q=tetris%2Blanguage%3Aassembly&order=desc&sort=stars"
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = Just (-3425)
  requestVersion       = HTTP/1.1
}

The queryString isn’t right! ?q=tetris%2Blanguage%3Aassembly&order=desc&sort=stars It encoded my + and :! After an hour of reading through docs and researching URL encoding specs, it dawns on me. + is an encoded whitespace.

No face-palm gif could ever represent the shear magnitude of my current emotions… You’ll have to use your imagination

I change my query to ("q", Just "tetris language:assembly") and the right count comes back! Just 354

I finally have something that correctly fetches a count of repositories from GitHub and parses it into an Int. After over four hours of Dark SoulsHaskell punishment, we deserve to enjoy a bonfire!

solaire

Edit: Bonus Round!

Thanks to Chris Allen and /u/JeanParker for pointing me towards wreq, which weirdly didn’t come up when I looked around for libs yesterday. Yep, it was 6th on the Google when searching for haskell https get. Network.HTTP is the top three results, and that doesn’t even do https.

¯\(ツ)

Armed with their helpful suggestions, I knocked this out this morning.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import Network.Wreq
import Control.Lens
import Data.Aeson
import Data.Aeson.Lens
import qualified Data.Text as T
import qualified Data.ByteString.Char8 as BS

opts :: String -> String -> Options
opts lang token = defaults & param "q" .~ [T.pack $ "tetris language:" ++ lang]
                        & param "order" .~ ["desc"]
                        & param "sort" .~ ["stars"]
                        & header "Authorization" .~ [BS.pack $ "token " ++ token]

query lang = do
    token <- readFile "token"
    r <- getWith (opts lang token) "https://api.github.com/search/repositories"
    return $ r ^? responseBody . key "total_count" . _Number

MUCH better. This includes reading my token from file called “token” so I don’t accidentally commit it. Also includes building up the different query options based on inputs, which was the next step. Thanks y'all.

solaire

Pixel gifs sourced from zedotagger on deviantart, thanks zedotagger!

Development Disappointment Disorder

“We’ve never had a successful release”

You just finished this really hard feature. The whole thing was worse than anyone realized. Not only that, but the feature wasn’t clearly explained, so you lost time churning on the actual requirements. Despite all the confusion, iteration, and technical challenges, you managed to get it working! You look back, savoring how much you have learned and grown.

You show it off to the product owner. He barely seems to hear you. His shoulders slump in disappointment.

“Great, but we are still four weeks behind.”

Your team is infected with Development Disappointment Disorder.

It looks different in every team. The manager who sets unreasonable deadlines then demands overtime. The project manager who gets angry at every little thing. Developers who feel they need to point fingers to shift the blame. The boss who is never happy no matter what is achieved. The team that feels they have never had a successful release.

Unreasonable Expectations

Development Disappointment Disorder is caused by unreasonable expectations. Someone thinks, hopes, or wishes they can get 100 units of productivity from a team and codebase that only can sustain 30-40 units. They want the impossible, and no amount of cajoling, pressuring, yelling, or passive aggressive comments will change reality.

Productivity is not completely a people problem. Every team has an upper limit to what they can produce in a system. The human mind has boundaries. Very real limits exist given the team’s size and existing codebases. While new technical tools and libraries enable more productive teams, these changes often are hindered by an existing codebase. The team with a multi-million line C# codebase is not going to get much value from the productivity gains possible with Haskell’s type system.

You cannot rush software development without incurring a drop in quality, stability, or maintainability. The work is complex and difficult: every expert in our field agrees with adages like “adding developers to a late project makes it later.”

Suggestions

  • For the technical staff: How accurate are your estimates? How consistent is your throughput? Do you regularly under-estimate your features? Do developers often say, “oh, that’s only…”? Are some types of features “always late”?

    You must learn to be blameless in this situation, and that means striving to give as accurate an estimate as you can with what you have. Throwing estimates out without much thought only makes things worse. How long did a similar feature take last time? If you regularly have inaccurate estimates in a certain area of the system, put extra care into those estimates, working to provide the best estimate you can.

  • For the business: There is only so many units of productivity that fit into a given time frame.You need to assess every feature and estimate, and consider the risk with each.

    Acting disappointed, angry, or passive aggressive will not get you more features, it will only demotivate the technical staff. You will get more consistency, because they will consistently work slower. You will get more hours, but each hour will see a massive drop in valuable work.

Acting disappointed, angry, or passive aggressive will not get you more features

If you are unhappy with the work produced, you need to consider why. Were you told it would be sooner? Did you promise that to someone? What changed since then? If the feature isn’t actually needed, why did you pick it? If spending twice as much made it not worthwhile, was it really a good choice?

What would happen if you didn’t promise when the features would be done? Either way, your promise doesn’t change when it will be done, it just adds risk to your credibility. Often the only thing to be gained by giving out estimates and time-lines is risk. Unless the customer is truly blocked on your estimate, simply communicate what you are doing, not when you hope it will be done.

Often the only thing to be gained by telling customers an estimate is risk

  • For the team: Celebrate your successes. Abolish the notion of a “failed release”. Build up a culture that finds little victories throughout the release.

With these tools you can fight Development Disappointment Disorder. You can start to celebrate what you do accomplish. A team that is energized and motivated will accomplish more. The team that celebrates their work will strive for more.

Designing a Compassionate Interview for a High Performing Individual

finding

After my post Interview Humiliation, a number of people have asked me how I interview compassionately. I strive to make my interviews as stress-free and respectful as possible while still rendering a yes/no at the end.

Any good interview process needs to start with goals:

  • Respect the candidate’s time
  • Make the candidate comfortable
  • Under-skilled candidates should feel no shame
  • Both passing and failing candidates should want to tell their friends to apply
  • We should render the same decision if the interview is repeated multiple times (with different exercises each time)
  • The candidate should know the process in advance and not be able to earn a pass through short term “cramming”
  • The candidate should feel we are all working collaboratively, instead of us against them

The Ideal Candidate

We want to hire for high-level skills. Fast learning, strategic thinking, good design, and emotional intelligence are more important than rote memorization. We care more about “ability to grow” than “current skills”. Our job does not involve whiteboard coding, puzzle questions, or anything that can be discovered with a decent IDE, so those are left out.

We care a lot about “no jerks”. We want our workplace to be fun, relaxing, and supportive. We want candidates who are comfortable being wrong and corrected regularly, and who can correct others politely.

We do not care about any question that could be easily be answered by Google. If an internet search can easily answer the question, it is pointless to care if the human in front of me happens to know it. Given the search revolution of the last decade, the value of memorized facts falls to almost zero. Facts as a proxy for actual job skill probably used to be accurate until the internet allowed every interview question to become public record.

Scoring

I hope someone one day invents a perfect productivity score! For now, counting bugs, features, correct answers, etc, are all proxies that poorly represent actual productivity. If those actually worked, why don’t we use them for raises, bonuses, and reviews? Because. They. Do. Not. Work.

Counting bugs, features, correct answers, etc, are all proxies that poorly represent actual productivity

A well-designed rubric can work. A rubric is a way of measuring the “un-measurable”. You probably have seen them in year-end reviews. A grid with categories on one side, and a 1-5 score with a paragraph explanation for each. We only score what has to be explained with a description from the interviewers.

“How was the candidate’s communication skills?”

“They misunderstood me only a few times, and I only had trouble understanding them once or twice. The watchers all felt like the communication was pretty good. They were definitely above average: ¾.”

“How about design skills?”

“They were able to clearly design every abstraction. They deeply understood passing functions as values, see how this takes a function for polymorphism? We prodded a bit, and they were able to give a couple other ways to achieve the same behavior. We all agreed it was best the way it was. All said, they did most of the heavy lifting for this whole project, and it is superbly designed: 4/4”

Our Process

To “level-set” our expectations, we had several of our developers at different levels perform the interview live in front of the team. We made them use various languages and exercises they had never seen before. This “level-setting” helped us to see how known good developers perform when way out of their element and uncomfortable.

All interviews are assigned to developers randomly, and all are expected to participate. The candidate should have two developers around at all time, to prevent bias by any individual. One pairs while the others rotate watching silently. The watchers are expected to stay quiet, to prevent confusion.

Before the first interview, the candidate is sent something very much like this document. We think there is no way someone could “fake us out” on any criteria. The things we look for take years of practice to cultivate, so “cramming” should not help much (if at all). The one exception to that is if they have never heard of Test Driven Development, we do recommend they Google it for a few minutes, as prior knowledge helps smooth the phone screen considerably.

Phone Screen

We pair program over the internet using a shared programming environment called Cloud9. We (currently) have projects set up for Java, JavaScript, Python, Ruby, and C#. The candidate may choose their desired language. We then pair program using Test Driven Development to build out a well defined exercise. The candidate may entirely verbally “navigate”, while one of our programmers entirely “drives” and types out the code. This is acceptable.

If the candidate has never seen any of those languages, Google or asking the interviewer is totally fair game, and does not count against them. If the candidate has completely no idea, the interviewer is expected to politely complete the task while keeping the candidate engaged.

The phone screen is under an hour, with 30 minutes for the pair programming. The remaining time is for questions about our office. We let them drive the second half of the interview, asking us questions about what our team and process is like. We attempt to answer clearly and honestly, both good and bad.

In-Person

The in-person interview is from 9:30 to 3:00. We are not thrilled with how much time it takes and would like to shorten it, but so far have no superior alternatives.

At the start of the day, we spend 30 minutes chatting and asking questions back and forth. Our questions are meant to tell us about the candidate’s background and “programming philosophy”: books read, favorite languages, interesting projects they’ve worked on, etc. We allow them as many questions as they would like.

The rest of the day is very much like a “typical day at work”, but with programming exercises instead of production code. We pair program on the exercises, following the same flow from the phone screen. The exercises have no “tricks”, they are reasonably straightforward and don’t require any prior domain or CS theory knowledge. Throughout the day, the candidate is free to get food, drinks, and run to the restroom.

The developer who is pairing has a primary role of making the candidate feel comfortable and accomplish any effort with a 50/50 split of effort. If the candidate really is struggling, the developer is expected to keep working with the candidate, even to the point of doing the hard parts and giving them the easy parts. A completely unskilled candidate will see their pair doing all the work with a smile and politely asking for “help” with variable names and such.

The developer pair is also expected to give the candidate some time to think if they are lost. The candidate at the end of the interview should feel like they have a complete understanding of what happened. The exercises are designed to be “too much to finish”, and as such finishing the exercises is not a success criteria as much as working well together and writing maintainable code.

We feel this gives us a highly accurate understanding of a candidate’s skill, while still putting them at ease. Many times we’ve seen a terrified candidate light up fifteen minutes into a pairing session while watching their developer pair do most of the work. Something about seeing code written “activates” even the most nervous candidate. All the thoughts of nerves turn into a concrete “oh, wait, I know what he just did, and I’ve got an opinion on that!”

Conclusion

We like this process because it is exactly what our job is like. We pair program for a lot of work, and the interview is meant to give candidates a sample of that. No one should show up for their first day of work and say, “wait, I actually hate pair programming.” By thinking through our goals and an “ideal candidate”, we were able to design a process that meets those goals.

I recommend you take an hour to write out your goals and needs. Then see how your interview process matches. Are you looking for a “human Google”? Does your job involve whiteboard coding? Would telling the candidate the exact process in advance with all the questions let them game the system? Because every single interview question I’ve seen is up on Google to be found and memorized on easy to read lists. How much time does your process take? Could it be shortened? Does it require the candidate to spend a lot of time at home? You might be surprised at what you find!

How Daily Meditation Improved My Technical Leadership

meditation

A couple years ago, a coworker told me a success story about mindful meditation. He explained how his wife took a class that required an hour of daily meditation for a whole month. He reported that she found it uncomfortable, but the effects were astounding. She claimed to feel more relaxed, more focused, and more “herself”. She found new reserves of motivation. She dropped the book she’d been trying to write for years, and started a business. After that business failed, she started another, this time successfully.

Immediately convinced, I started by listening to guided mindfulness meditations on YouTube and phone apps. After a couple weeks of that, I started doing breathing exercises on my own while sitting listening to rain sounds. I made a habit of doing it for about an hour every day on my lunch break. A few months later, I hated it.

For whatever reason, after each session I would feel very uncomfortable. I would be confronted with emotions that made me feel worthless and embarrassed. My mind was bringing forward all these painful memories and feelings. Over time, I started to dread the lunch break.

The Revelation

Then, one day, I had a profound sense of understanding. I realized what was causing these negative emotions. I was in the middle of a major push to guide our team to adopt a new programming language. While I’d led many such initiatives before, I saw that I’d always had the same struggles. My ego and self-worth were so tied into “my” campaigns that I wasn’t able to clearly hear criticism. When people pointed out legitimate concerns or weaknesses, I would get embarrassed and take it as a personal failure.

I could see, from the very beginning, my current “campaign” had been poorly executed. After researching the new technology for a long time, I felt like I saw a place where it could solve some of our needs. Instead of carefully writing out a proposal, I “half-jokingly” brought it up whenever I could. When the listeners didn’t immediately react positively, I quickly retracted the idea, “just kidding”. Over weeks I dropped not-so-subtle hints, “this would be better with…”, or, “we wouldn’t have this problem if…”. Instead of convincing anyone of anything, I just broadcasted to everyone how insecure I felt about the issue. Everyone could clearly see that I cared, but I was too afraid to put myself out there and make a serious suggestion. For months nothing changed, and I felt more and more bitter that “my idea” wasn’t being listened to. In reality, I hadn’t even made a serious suggestion yet!

Eventually, the people around me started to listen and research the new language. They agreed it really did solve a need we had. After a while, enough people got on board that we started using it more. Even then, I held tightly to “my idea”, afraid that it would prove to be a failure. When someone ran into issues, I would get defensive. Even though I tried to keep my emotions in check, I clearly took any criticism of the language as if it was a personal failing, no matter how legitimate. I’d tightly coupled my ego to my ideas, and I always had.

After a few months of meditating, this idea rocked my world. My negative emotions came from a realization that my actions were not in line with my values. I don’t want my self-worth to be tied to a simple tool. No tool is worth it. Every tool fails. I’d sold myself short, lowered my self-worth to that of a tool. This completely blocked my ability to be an effective technical leader.

I couldn’t hear criticism of my ideas without getting defensive, so I couldn’t think critically. I couldn’t help work out solutions to the best of my ability, because that would mean admitting personal failure. Every side comment or momentary struggle felt like a catastrophe.

After realizing this, I felt a wave of relief. My emotional struggle leading this programming language change made sense! And I had an easy way out! I simply had to distance myself from my ideas, separate my self-worth from my tools. The more I meditated, the easier it got.

Present Day

While I still sometimes get the old pang of embarrassment or defensiveness, now I understand where it comes from, and how to move forward. I can let my ideas go. I can consider ideas critically. I can make changes and adjustments if the ideas are not working.

This summer I decided to write up my learning. I wanted to always be able to look back and remember these important lessons. My write-up was the seed for the book Convincing Coworkers. I explain the lessons learned from implementing a half dozen major technical changes, and how I learned to lead without letting my ego cloud my thinking.

If you want to become a more effective leader, I highly recommend a practice of daily meditation. Meditation gives your mind a chance to deeply consider your actions and values. It allows your subconscious to reflect on what and why you act.

Not everyone will struggle from the same issues as me. Every leader has a different set of challenges and situations. Meditation lets your mind analyze your unique weaknesses and strengths, coming up with a tailored perspective. Meditating allows your mind to creatively look for ways to improve itself. While rarely comfortable, this process can have incredible benefits.

Appendix: “I’m uncomfortable with the connotations of meditation”

You may feel uncomfortable with meditation because of its current strong associations with religion. Meditation is not inherently religious. I consider meditation like an exercise routine for the mind. If you are religious, and exercise your body as a form of religious expression, exercise is a religious practice. If you are not religious, and you exercise to improve your health and well-being, exercise is a secular practice. Meditation is just a form of mental exercise. Simple as that.

Meditation is a tool, and the tool wielder determines its use. Don’t let a fear of the connotations of meditation stop you from experiencing improved leadership, creativity, and focus!