Skip to main content

2020 Changes

I've spent today updating everything to do with ntoll.org. Many thanks to my buddy Kushal Das for the invaluable advice on various technical matters.

Here's what I've done:

  1. The website is run using Nikola ~ a Python based static site generator. It was very easy to migrate away from my previous hand-coded solution made with a custom Flask script and HTML.
  2. It's hosted on GitHub Pages which is both free and makes it very easy to manage code and deployment.
  3. I've switched email hosting to Fastmail, who made the whole migration process extraordinarily easy.
  4. I've archived the code for the old website for "safe keeping", even though all the content has moved over to the new site.

Lots of things have changed in addition to my own technical infrastructure.

I've just finished a very busy period of freelance work with two clients - the Freedom of the Press Foundation for whom I contributed Python code to an application for journalists, and NHSDigital for whom I fulfilled the remote role of a code mentor / developer coach (like an agile coach, but much more technical). I was collaborating with a team of Java developers who were learning Python and Django while building APIs for internal customers and users. I also provided Python training, which was very rewarding since I love teaching and the NHSDigital folks were already technically very strong and lots of fun. Both teams were exceptionally talented and friendly. I had a blast and made lots of new friends.

In just over a week I'll be starting a new gig with the Ministry of Justice as a senior Python engineer. I'm really looking forward to this since the work is useful, interesting and challenging. I'm looking forward to learning from my new colleagues and making contributions to our collective effort.

What does this mean for all my other projects (for instance CodeGrades or Mu)..? Ironically, I believe I'll have more rhythm to my day and so my work on these projects will continue in my own time, but at a more predictable pace. Watch this space. :-)

Finally, the current Corona virus pandemic is affecting life in rather unforeseen ways: my daughter and older son have had their summer exams cancelled. Apparently, their results will be based upon their predicted grades. It means my daughter is going to read Mathematics at university with a scholarship (dependent on her achieving her target grades, which she will in light of the current situation) and my son will have the grades needed to go onto study Physics, Maths, Music and Design Technology at "A" level (his current plan is to read engineering at university). My youngest son continues school lessons, completed at home via an online platform.

We've also had to purchase some home-studio equipment (mics, pre-amp, monitor speakers, midi keyboard and related audio software). Mary, my wife, uses this to teach her many instrumental students remotely via video calls. It soon became apparent that laptop or mobile phone mics are not great for cello or piano teaching. Happily, the new equipment is proving to be a great success and Mary is able to continue teaching most students without disruption and from the comfort of our own music room:

Our home studio

It also means I have the equipment available to produce my own (Python) training materials and the kids are having a great time mixing all sorts of different types of music.

Here's hoping you stay safe and remain healthy during this challenging time. Since most of us are in lock-down, don't hesitate to get in touch if you fancy a chat..! Social distancing rules don't mean you have to be anti-social.

:-)

Sponsorship

The majority of the code I write is given away for free for the benefit of others. This is unsustainable so I often think about how such voluntary work could be supported. One solution is related to my CodeGrades idea. But this is a slow burn project that has involved, and continues to demand, a huge amount of time, effort and money before I'll have anything to show for it.

Another solution is simply to ask folks for money.

Having been a musician, I'm used to this feeling. I used to busk with some buddies as a way of trying to make ends meet. However, as my grandmother once pointed out, it's a form of begging. In one sense, this is a tragedy: begging is a last resort because nothing else you do is of value. You're forced to rely on the compassion and generosity of others. Yet in another sense it's like the activity of a mendicant asking for alms for their service or contributions to society. In any case, no matter how you "spin" it, it's a precarious situation to find oneself in ~ be you a musician or free software developer (and I'm both!).

While there are some kind hearted individuals and (even rarer) companies or organisations who are happy to help support such work, most think, "meh, I'll just use the software" (without so much as a thank you). Even worse (and I have personal experience of this) some folks will pass off such voluntary work as their own and reap the rewards for such a lie. This is why I actually earn money-to-live by writing software as a freelance developer (it gives me the flexibility I need to work on my own coding projects).

Therefore, it was with some interest that I noticed GitHub had created what they describe as account sponsorship. "Sponsorship" is an interesting word to use because it makes the begging / mendicant aspect of the situation more palatable. It also cheerfully suggests that the free/open-source software world is full of generous hearted folks or organisations just waiting to give back to the selfless volunteers who make all this code available via their GitHub account.

This is, of course, a fiction. It's also normalising begging as a solution to funding free software volunteers.

But I'm open minded and willing to give anything a go, especially if it means it'll cover my costs. So I've signed up and now you can give me money for my efforts. As of the time of writing, and after tweeting about this to my 3,000+ followers, this is what my account looks like:

I can't help but think that in six months time, it'll still look the same. But at least I can say I gave it a go. Despite my misgivings, I feel it important to give ideas a chance to flower and I'm trying very hard (yet failing) to view this as a mendicant rather than beggar.

So, if you've ever found my work or activities useful, beneficial or supportive, please consider helping me sustain my efforts. You can do so by visiting my "sponsorship" page to find out more about what I do and how to give "alms". The summary is that I focus on education: I develop software that helps folks learn the skills and knowledge they need to imagine, develop and program the stuff they want. If you believe this is important work, sponsor me with a monthly contribution.

Alternatively, and many thanks to the small number of you who have already done this, you can send me a virtual hug-as-a-book by buying me something from my Amazon wish list (none of the items are particularly expensive). Small gestures, such as gifting books or music with a supportive message, can actually mean a lot.

I'll report back in a few months to let you know how I get on. I have to admit, I don't hold out much hope. Best knuckle down on CodeGrades.

Well... what are you waiting for..? Go on... get your wallets out. :-)

(Honestly, I quite understand if you don't. I have the same misgivings you're probably having after reading the sentence above. Such a feeling is the nub of what I want to demonstrate in this blog post.)

CircuitPython 2020

My buddy Scott, one of the wonderful people at Adafruit, recently asked folks in the wider CircuitPython community to share their thoughts about what they'd like to see for the project and community during 2020. This is my contribution to the conversation.

There are three aspects to my hopes for CircuitPython in 2020:

  1. Things I hope will continue,
  2. Developments I hope to happen,
  3. "Moon on a stick" wishes for the future.

I want to start by reaffirming my belief that Adafruit, and those involved with CircuitPython in particular, do inspiring stuff when it comes to fostering an open, welcoming, supportive and collaborative tech community. This is an extraordinary achievement. My own experience in the wider Python community is that it takes a huge amount of effort, patience, compassion and thoughtfulness to grow and sustain such a "scenius" (and, sadly, this can be destroyed very quickly by only a few bad actors). So my wish for continuation in 2020 is that everyone involved in the CircuitPython community grasp opportunities to enlarge positive aspects of ourselves such as mutual appreciation, the sharing of tools and techniques, latent network effects, and a spirit of tolerance, compassion and fun.

What I hope happens this year is a broadening of CircuitPython's horizons. My passion for tech can be summed up by a quote from David Allen, the producer on the original BBC micro from the 1980s -- my first computer. He explained that, "[t]he aim was to democratise computing. We didn't want people to be controlled by it, but to control it." CircuitPython feels to me like it sits very much in the hobbyist / enthusiast "segment" of users who already know something of programming. I'd love CircuitPython to make inroads into beginner related education. This probably involves a focus on learning what teachers would like from a coding platform, honest appraisals of how beginner coders (no matter their age or background) find their first steps with CircuitPython and Adafruit boards, as well as making the results of such findings accessible to all. This final point is important. Learning to code shouldn't just be an option for English speaking, able bodied folk from a cultural background that means examples and cultural assumptions are easily understood. I hope you agree that education should be an inclusive endeavour, and this takes concerted and conscientious effort. I hope the CircuitPython community find the time and space to invest in the (never finished) work such a focus entails.

My "moon on a stick" is a CircuitPython based mobile phone with a touch screen and a simple (PyperCard like?) Python UI that's easily hackable. Think of it as the open, easy-to-assemble, relatively cheap and "hack friendly" version of something like the Light Phone with all the schematics and design assets (e.g. files for 3d printing the case) available under open source licenses so folks can go mod-crazy. Now wouldn't that be a turn up for the books..?

I'll finish by wishing everyone in the CircuitPython a flourishing and happy new year. Here's to wonderful things in 2020.

Let's go..!

:-)

A Classical Playlist of Love

When my Canadian buddies Andrea and Brett visited the UK in the summer, we visited the Royal Albert Hall to hear a classical music concert that was part of the Proms -- the world's largest classical music festival. I think they must have enjoyed themselves because they recently got in touch asking me to suggest a playlist of classical music. What follows is my response along with videos of performances of the pieces I chose.

Rather than put together a playlist of apparently random choices, I decided it would be interesting to assemble a set of pieces related to a particular feeling. In this case, I chose love.

Why love?

Because love is complicated, tragic, happy, joyful, funny, raunchy and many more aspects too numerous to list here. The western classical tradition has around 900 years of musical reactions, settings and descriptions of love -- plenty of opportunity to reveal a breadth and depth of music. So, I'm going to associate each piece with some aspect of love, provide a short commentary and embedded a video of a performance of the selected piece.

I want to draw your attention to my sharing videos of performances. Ideally, you'd listen to these pieces in person at a concert. Only in such a live situation can you really experience the fleeting moment of the performance, executed by highly trained musicians and feel that sense of being in a particular unique moment, together with all the others in attendance. The videos are a less than perfect replacement, yet still have the potential to move you.

For convenience I've put them into a playlist on YouTube.

Since this is classical music, it may sound strange to modern ears not familiar with styles and mannerisms from hundreds of years ago or it uses sung or instrumental techniques that sound odd because they're not used much in contemporary popular music. If classical music isn't your thing, consider this an invitation to explore a new musical world -- a world that rewards repeated engagement, an open mind and long term listening. This is music that makes demands of the listener, but will reveal depths one cannot even imagine until you experience them for yourself. For those of you who are already immersed in classical music you'll find none of my choices particularly challenging. In fact, you may even roll your eyes and think, "oh, not that old chestnut again". Please remember, this is an introductory playlist, but know that I'm open to suggestions.

I sincerely hope you are moved by the pieces I've selected.

Tender Love

Gustav Mahler - Adagietto from Symphony No. 5

This is a musical representation of Mahler's love for his wife Alma. Their friend Wilem Engelberg recollects, "In place of a letter, he sent her this in manuscript form, not adding a further word. She understood and wrote to him telling him to come!!! They both told me this."

Alma later revealed Gustav left her the following poem with the score.

In which way I love you, my sunbeam,
I cannot tell you with words.
Only my longing, my love and my bliss
can I with anguish declare.

Tragic Love

Pyotr Ilyich Tchaikovsky - Fantasy Overture to Romeo and Juliet

Tchaikovsky was well acquainted with tragic love: he was a gay man in 19th century Russia (a time and place full of prejudice towards someone such as himself).

Such tragedy and pain as well as an overflowing sense of passion are skilfully captured in this piece or "musical impressions" loosely assembled to follow the story of Romeo and Juliet. Listen out for the overwhelming "love theme" which has become a musical cliché for inevitably doomed lovers.

Romantic Love

Sergei Rachmaninoff - Piano Concerto No.2

Rachmaninoff, another Russian, is widely considered one of the greatest pianists of all time, as well as one of the greatest composers for the piano. This piano concerto is full of yearning melodies, gushing outpourings of emotion, tender moments and joyful exuberance. Just like falling in love!

It was effectively used in the David Lean romance from 1945, "Brief Encounters", thus cementing it as a "romantic" classic.

Love Denied

Puccini - O Mio Babbino Caro

This aria, from Puccini's opera "Gianni Schicchi", comes at the moment where the daughter of Gianni tells him that she's fallen in love, wants to get married, and if he won't let her go to her true love, she'll throw herself off the Ponte Vecchio bridge in Florence.

If you imagine this sounds like a musical tantrum in the making, get ready with the hankies and prepare for something absolutely not tantrum like at all.

The soprano singer in the video, Montserrat Caballé, is simply stunning. Her charisma, presence, musicianship and vocal control is awe inspiring. Yes, she's the one who sang with Freddie Mercury on his single "Barcelona".

Platonic Love

Brahms - Intermezzo Op. 118 No. 2

Brahms wrote this beautiful piece for his friend Clara Schumann (Clara is another musician considered one of the greatest pianists of all time and a wonderful composer in her own right). I describe it as "Platonic" because Clara was married to Brahms's mentor and friend Robert Schumann, who eventually succumbed to mental health issues. Brahms and Clara were undoubtedly close and shared a deep love for each other, but it was only ever a love between dear friends. Imagine how you'd feel if you had been gifted such a piece of music.

Fantastical Love

Berlioz - Symphonie Fantastique

This is sex, drugs and classical music with an added dose of musical story telling.

The video is of the concert Andrea, Brett and I attended. I won't say any more, since the performers do a fantastic job of explaining what's going on. It's quite a spectacle (and watch out for the mirror balls).

Love Transformed

Schoenberg - Verklärte Nacht

Schoenberg wrote this piece to closely follow the structure of the poem upon which it is based, to the extent that the different sections in the musical score map directly to lines in the poem. The poem, by Richard Dehmel, whose title is translated as "Transfigured Night", tells of two lovers walking in the woods. The woman opens up about a terrible burden she carries. Her lover is full of compassion, love and support which transforms her burden into a shared aspect of their life together. I'm being necessarily vague here so I don't spoil the poem for you... just read it as translated here and then listen to the piece. My favourite lines (and part of the piece) are:

Just see how brightly the universe is gleaming!
There’s a glow around everything;
You are floating with me on a cold ocean,
But a special warmth flickers
From you into me, from me into you.

At this point the music sounds as if it's gleaming..! One other thing you may hear, the violin (a high instrument) is prominent in the sections representing the woman speaking, whereas the cello (a low instrument) is prominent in the sections representing the man speaking.

Love's Dream

Fauré - Après un rêve

This is simply a song about a dream of the poet's beloved. The words of the poem (and their English translation) can be found here.

Erotic Love

Wagner - Prelude and Liebestod from 'Tristan Und Isolde'

Where do I start with this one..? This is erotic music. I don't mean humping or "having a shag". Rather, it's a musical version of a passionate embrace (between the protagonists, Tristan and Isolde).

In music theory we use the term "climax" to describe the point at which all the tension in the music is released. There's a reason we call it a climax. This piece has a famously ecstatic climax that takes a while to build, after lots of tender pauses, gradually intensifying musical moments, soaring strings and throbbing horns (no pun intended).

I'm sure you'll hear what I mean.

Orgasmic Love

Orff - Dulcissime (from Carmina Burana)

Just watch and listen. No further explanation needed.

Marriage

Mendelssohn - Wedding March from A Midsummer Night's Dream

You'll hear this in pretty much every church wedding you ever attend - and quite rightly, it's very uplifting, celebratory and joyful music. Just what you need for such an occasion.

A Moment of Love

Vaughan Williams - Silent Noon

Dante Gabriel Rossetti's poem "Silent Noon" recollects an intimate shared moment of love that was both fleeting and intense. Vaughan Williams music beautifully complements the nostalgic, thoughtful and timeless nature of the moment in question.

Your hands lie open in the long fresh grass,--
The finger-points look through like rosy blooms:
Your eyes smile peace. The pasture gleams and glooms
'Neath billowing skies that scatter and amass.
All round our nest, far as the eye can pass,
Are golden kingcup-fields with silver edge
Where the cow-parsley skirts the hawthorn-hedge.
'Tis visible silence, still as the hour-glass.
Deep in the sun-searched growths the dragon-fly
Hangs like a blue thread loosened from the sky:--
So this wing'd hour is dropt to us from above.
Oh! clasp we to our hearts, for deathless dower,
This close-companioned inarticulate hour
When twofold silence was the song of love.

Love's Yearning

Gluck - Che farò senza Euridice

The story of Orfeo and Euridice is popular among composers. The hero, Orpheus, is a master musician who can tame nature with his music. His beloved Euridice dies and is taken to the Greek underworld. Missing his lover, Orpheus travels into the underworld and charms Hades, the god of the dead, with his music. As a result, Hades promises that Euridice can live and return to the world of the living so long as Orpheus doesn't look back to see if she follows him on the way home. Inevitably, Orpheus looks back and Euridice is, once again, struck dead. It's at this point in the opera that we hear the following aria for counter-tenor (the highest natural voice for a man to sing). The title means, "What shall I do without Euridice?".

What will I do without Euridice
Where will I go without my wonderful one.
Euridice, oh God, answer
I am entirely your loyal one.
Euridice! Ah, it doesn´t give me
any help, any hope
neither this world, neither heaven.

I'll leave you to work out what happens next.

Love's Pain

Strozzi - Che si può fare

Barbara Strozzi was a remarkable woman. Born in Venice in 1619, she was one of the most well known musicians of her age. It is claimed she had more music in print than any other composer of the era. This success was of her own doing since she had no support from the usual sources of the church or nobility.

I imagine that if Bridget Jones were from Baroque era Venice, she'd be listening to this instead of Céline Dion's "All By Myself".

What can I do?
The stars have no pity and work against me;
If heaven will give me no gesture
Of peace for my pain,
What can I do?

What can I say?
The heavens are raining disasters on me;
If Love will not grant me a moment of breath,
to relieve all my suffering,
What can I say?

Love of Home

Smetana - Vltava (The Moldau) from Má vlast

This is the musical antidote to nationalism.

The Czech composer Smetana uses music to represent the river Moldau as it runs through his country in a celebration of the landscape, activity, folk music, dance and industry of the people of the Czech republic.

It works well as a musical metaphor, starting with the trickle of notes from the opening flutes and building up to a flowing melody surrounded by melodic eddys and currents in the strings. The river music keeps returning throughout the piece as different musical interludes interject to describe some aspect of where the river flows.

Love of Nature

Beethoven - Pastoral Symphony

This piece is an old friend of mine. Forget the Disney version from the animated film "Fantasia" -- this is Beethoven musically representing a visit to the country and the emotional impact that has on him.

In the latter part of the work he paints a picture of a country dance interrupted by a storm which he concludes with a hymn to nature. Beethoven wasn't conventionally religious, but this is a certainly a spiritual reaction to nature.

Love of Humanity

Beethoven - Choral Symphony

Beethoven's 9th is full of different emotions and a real journey of the soul in musical form. But the part I want to particularly highlight is the finale, the "Ode to Joy".

Beethoven was famously misanthropic and that's how the finale starts with the rather annoyed sounding cellos and double basses. But once the "joy" theme is found the instruments try their hardest to make something of it. Yet Beethoven's genius is to interrupt the finale at this point, and start again with human voices. Schiller's words are a celebration of brotherhood, togetherness and humanity.

Yes, it really does have the line, "here's a giant kiss for all" sung by the massed choir. Thank you Ludwig.

Parental Love

Shostakovich - Piano Concerto No.2 (Second Movement)

Dimitri Shostakovich wrote this concerto for his son Maxim. Maxim used it as an audition piece for music conservatoire when he was a teenager. This second movement is both sad yet hopeful, as a parent lets go and reflects on the child leaving and growing into adulthood. The music full of love and sadness. Perhaps if the orchestra is Dimitri, then Maxim is the piano.

Love Ending

Strauss - "Im Abendrot" from Four Last Songs

This is a musical and poetic meditation on two lovers coming to the end of their long life together.

Through sorrow and joy
we have gone hand in hand;
we are both at rest from our wanderings
now above the quiet land.

Around us, the valleys bow,
the air already darkens.
Only two larks soar
musingly into the haze.

Come close, and let them flutter,
soon it will be time to sleep -
so that we don't get lost
in this solitude.

O vast, tranquil peace,
so deep in the afterglow!
How weary we are of wandering--
Is this perhaps death?

Love Lost

Elgar - Cello Concerto

My wife, Mary, once played this in a concerto competition (and won).

Elgar wrote the piece soon after his wife, Alice, passed away. The cello is, of course, Alice and the music Elgar creates reflects all the different feelings he has now that she is gone.

This very old performance, by Jacqueline du Pré on cello and Daniel Barenboim conducting, is poignant. They were married but Jackie's playing succumbed to multiple sclerosis soon after this recording was made. The MS eventually killed her around a decade or so later.

Testing CircuitPython Modules

MicroPython, a project by my buddy Damien George, is a complete reimplementation of the Python programming language for microcontrollers and embedded systems. Put simply, it's Python for extremely small computers (and I once wrote a book about it). CircuitPython, a friendly fork of MicroPython, is sponsored by the fantastic folks at Adafruit, a company who make playful and easy-to-use "hackable" technology while promoting a welcoming, friendly and diverse community of makers. They are led by legendary founder and electrical engineer extraordinaire, Limor "ladyada" Fried. CircuitPython is MicroPython for Adafruit's line of boards and with a consistent API for accessing the numerous bits of cool hardware you can use with such devices.

I was privileged to recently complete a block of work for Adafruit: I've written a CircuitPython module called adafruit_radio that makes it easy for Bluetooth enabled Adafruit boards to communicate with each other. The API is a simplified version of the work myself and a number of other volunteers did on the equivalent functionality for the BBC micro:bit (here's a tutorial I wrote for the micro:bit radio, to make electronic "fireflies" blink at each other over Bluetooth).

In the new Adafruit module, sending a message is as simple as:

from adafruit_radio import Radio
r = Radio()
r.send("Hello")

Receiving requires a loop (to allow the device to keep checking for messages), but is equally as simple:

from adafruit_radio import Radio
r = Radio()

while True:
    message = r.receive()
    if message:
        # Do something useful with the message.
        print(message)

The best model for thinking about this module is that of kids' walkie-talkies. Each radio is tuned to a certain channel upon which it broadcasts and receives. If you send a message, anyone else listening in on that channel and within range will receive it. You'll also receive messages broadcast on that channel from others within range. This is an ideal network topology because it's both familiar yet capable since other, more specialised, network topologies can be built on top of it. There is potential for users to grow from a simple "walkie-talkie" model of networking to something more sophisticated and of their own devising.

Kids on walkie-talkies

The channels in the module are numbered between 0-255 and can be set when creating a Radio object (the default channel number is 42 in honour of Douglas Adams, who saw so much humane potential in new technology yet mischievously warned of the polluting effect of technology for technology's sake via the inept work of the Sirius Cybernetics Corporation):

from adafruit_radio import Radio
r = Radio(channel=123)

Alternatively, you can change channel at any time via the Radio object's configure method (continuing the example above):

r.configure(channel=7)

Finally, in addition to sending strings of characters it's also possible to send and receive arbitrary bytes:

from adafruit_radio import Radio
r = Radio()
r.send_bytes(b"Hello")

Receiving bytes gives you the full context of the message expressed as a Python tuple consisting of three values: the raw bytes, the RSSI reading (i.e. strength of the signal, from 0 [max] to -255 [min]), and a timestamp representing the number (and fraction of) seconds since the device powered up, as demonstrated in the following REPL based example:

>>> from adafruit_radio import Radio
>>> r = Radio()
>>> r.receive_full()
(b'Hello', -40, 3245.155408037)

That's it! You can find the code in this GitHub repository. Special mention and thanks must be made to Adafruit's Scott Shawcroft and Dan Halbert who wrote the Bluetooth API I used. Because of their work I was able to create my module with less than a hundred lines of Python (at time of writing). Thanks chaps!

Armed with this context, I'm going to explain why and how I created a comprehensive test suite for the adafruit_radio module.

I'd summarise my approach to testing as follows:

  • Testing is fundamental for creating and maintaining well designed software. My experience is that it tends to produce code that is both simple and easy to understand because nobody wants to write complicated tests to exercise equally complicated code. The process of exercising code through tests forces a developer to think carefully about how their code is written while having the added benefit of demonstrating it works as expected. Simplicity and ease of understanding are desirable properties because such code is easier to read, correct and maintain and also indicates the author's own clarity of thought (a good sign). I also believe it helpful and friendly to write code that's easy for others to read and understand (I often imagine my code will be read by an intelligent beginner coder, because it forces me to explain and address my own assumptions about my code).
  • I personally don't practice strict test-driven development, where one writes tests before writing the implementation. I prefer to explore, improvise, extemporise and play at first. I often try various different approaches to the task in hand and regularly re-draft. I'll often seek advice and comments from collaborators, colleagues and potential users of my code as soon as possible. Therefore, my highest priority when I start a new project is making my code simple enough so that it is very easy to change. Often this step in the coding process is called a "spike".
  • Only when a project settles on a certain architecture, implementation or foundation of code do I add tests. I think of this as a sort of "hardening" process. When I'm happy with an approach I'll often re-draft the exploratory code I've already written, alongside writing tests to exercise that specific piece of code. I aim for, and often achieve, 100% test coverage (every line of my code is exercised in some way by a test). This process allows me to get a feel for how my API works from the point of view of a fellow coder encountering it for the first time.
  • The hardening has another effect: I've baked in an expectation for how the code should behave via the tests. I also make sure my tests are commented in such a way that a meaningful intention behind the test is revealed. They're also a useful source of information for future users and/or maintainers of my code. Finally, and perhaps most importantly, they help manage change.
  • No useful software is ever finished simply because the universe changes (and so must software). Despite our best efforts, software is often complicated and it's easy to forget something or not realise how a change in one part of the code may break another apparently unrelated part. Having a suite of tests to check all the aspects of a codebase helps future maintainers make changes with confidence.
  • Being pragmatic, I sometimes don't follow the playful explorations outlined above. If I'm dealing with a well defined or mature protocol (for example), I'll quickly settle on an approach, usually based upon research into how other folks have solved the same problem, and proceed by writing tests based upon the protocol in order to measure the completeness, accuracy and progress of my resulting implementation.

The problem for developers writing for CircuitPython is that such code is usually to be run on microcontrollers with certain hardware capabilities. Yet such code is invariably written on conventional computers running operating systems such as Windows, OSX or Linux. The only way to know your code works is to try it on the target device. This is, to say the least, laborious compared to having the benefits of running an extensive test suite in a matter of seconds.

Having written my small and simple adafruit_radio module I found myself missing the benefits of a comprehensive test suite. After asking around, I found most Python modules for CircuitPython don't have a test suite and there hadn't been much (if any) exploration for how to address this. Scott suggested I add what I thought best in terms of testing to my module.

I wanted my solution to meet the following requirements:

  • It works with existing Python testing tools so non-specialist Python developers feel at home and can bring their existing skills to CircuitPython with little effort.
  • It runs on a conventional computer, because that's where developers do the majority of their work. (I'm constantly running and re-running my test-suite as I make changes to code.)
  • It handles the problem of using CircuitPython-only modules in a test-suite run on a conventional computer with standard Python.

I decided to focus on using an existing, well known and mature Python testing tool called PyTest (the creation of another friend of mine, the extraordinarily prolific and very thoughtful Holger Krekel). The advantage of using PyTest is that it has a large number of plug-ins that are both useful and familiar to a large number of Python programmers. One such plug-in I use a lot is the Pytest coverage tool, which makes it easy to spot areas of a code base that are not exercised by a test suite. PyTest also has a well defined mechanisms to extend it to work in specialist testing situations (such as our own CircuitPython based context).

As far as I was concerned, using PyTest met the first two of my self-imposed requirements. I was left with the final problem of dealing with CircuitPython only modules that wouldn't work on a conventional computer.

This is where I need to introduce the concept of "mocking" in a test suite (as in "mocked up" rather than poking fun). Mocking is a way to replace parts of your system under test with "mocked up" objects and then make assertions about how such objects have been used. Thanks to another friend of mine (the huge hearted, funny and rather hairy Michael Foord), mocking is built right into Python's standard library.

My idea was simple: automatically mock away those parts of CircuitPython that don't work on a conventional computer. Such objects give the developer a way to check and ensure the module under test is working as expected with CircuitPython. It's even possible to specify how such mock objects should behave under certain conditions (such as when they may be the source of some sort of data which the module under test will use). However, this aspect of mocking should be used with great care -- more on which later.

If a developer creates a conftest.py file in their test suite PyTest will import it before running any of the test code and use various functions found therein to configure the test suite. For example, if PyTest finds a function called pytest_runtest_setup in the conftest.py file, then this function will always be called immediately prior to any test function. Just what we need!

My conftest.py file is very short and simply mocks away named aspects of CircuitPython which cannot be run on a conventional computer immediately before any test is run, via the aforementioned pytest_runtest_setup convention.

It means I can write conventional looking PyTest based unit tests like the following:

def test_radio_receive_full_no_messages(radio):
    """
    If no messages are detected by receive_full then it returns None.
    """
    radio.ble.start_scan.return_value = []
    assert radio.receive_full() is None
    radio.ble.start_scan.assert_called_once_with(
        adafruit_radio.AdafruitRadio, minimum_rssi=-255, timeout=1
    )
    radio.ble.stop_scan.assert_called_once_with()

Some things you should know: radio.ble is a mocked away part of CircuitPython. As a result, on the first line of my test function, I've been able to tell the mock that the result of calling the start_scan method is an empty list. Then I can assert that the method I want to test (the radio.receive_full method returns None in this context. Furthermore, I'm able to check in the final two statements of the function that the start_scan method was called with the expected arguments, and that an additional stop_scan method was called too.

When I run the test suite, I see something like this:

$ pytest --cov-report term-missing --cov=adafruit_radio tests/
============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-5.3.0, py-1.8.0, pluggy-0.13.0
rootdir: /home/ntoll/src/adafruit_radio
plugins: cov-2.8.1
collected 12 items                                                             

tests/test_adafruit_radio.py ............                                [100%]

----------- coverage: platform linux, python 3.7.5-final-0 -----------
Name                Stmts   Miss  Cover   Missing
-------------------------------------------------
adafruit_radio.py      61      0   100%


============================== 12 passed in 0.09s ==============================

Note all the feedback about code coverage..! Neat, huh?

A mock object, by its very nature, is a mock-up of something else... it's not the real thing..! So what happens when the real thing (that has been mocked-up in the test suite) changes? For instance let's imagine that the result of a call to start_scan is no longer an empty list, but something else. The tests will still pass because the mocked-up object doesn't reflect the real object, yet when the module under test is used on a real device with the changed version of CircuitPython then it won't work correctly.

This is obviously not a good situation and why I mention mocks should be used with great care and attention.

The most obvious solution is for the developer in charge of the test suite to be careful and check API updates in the release notes of the modules being mocked away. However, this becomes a burden if the test suite mocks away a huge number of modules. It's also open to human error.

There are several ways to mitigate this problem, but because it's early days I've not been able to investigate these potential solutions properly. The best I can do at this point in time is shrug my shoulders, say things are under construction and invite folks to dive in and help. Our community would certainly be enriched by such collaborations.

In conclusion, I'm quite pleased with this first step in equipping CircuitPython modules with comprehensive test suites. Yet there's still plenty to do -- most notably, ways to address the problems mentioned with mocking.

As always, comments, constructive criticism and ideas expressed in a friendly, collaborative and supportive manner are most welcome.

Over to you... :-)