15 July

Why embedded systems like the raspberry pi should not allow swapping to an SDcard

SDcard and Flash memory errors

SD card corruption and eventual failure is common on embedded systems like the raspberry pi. I've realize there is a lack of consistent and reliable information on this. This post describes the problem and provides solutions.

This post has 3 parts: description of the problem, practical implications, and tricks to disable swap on the raspberry pi and related linux devices. People who just want a solution can skip to part 3.


In our lab we have a range of systems from expensive underwater robotics to small embedded IoT devices all of which use SD card or flash memory. There are modes of operation that are acceptable on a normal desktop or laptop that will gradually degrade and eventually destroy an embedded device after a few thousand cycles of writing to a given cell. Since the damage is gradual, it can easy to overlook the cause during setup and testing, and just blame it on "bad memory."

The root issue is that solid state memory wears out with successive changes to the contents, and this happens on a per-memory-cell basis. Modern SD and flash memory have sophisticated internal algorithms for "wear levelling" that try to reduce the impact and extend the life of the chips, but these cannot eliminate the problem totally, since it's intrinsic to the physics of the devices. Hard disks, by the way, can also wear out, but not generally as a function of the read-write behaviour. The key idea behind wear levelling it to change the physical location used for frequently-changing data, to spread the use.
[ Technical aside: As of 2011, 25nm MLC NAND lasted for about 3,000 write cycles. As of 2018 some sources suggest that single level cell NAND (SLC) can last for 10,000 cycles ]

For normal use, only a small amount of the device is written at a time and if this is moved around the device it can be many years before the flash memory storage wears out. Without wear levelling it's easy to write a malicious program that "burns through" a specific cell in just a few minutes (although OS cacheing policies also try to preclude this).

Note that there are differences between expensive SD cards and cheap ones. These differences include both the robustness of the memory elements themselves, their speed and also the nature of the wear levelling (e.g static vs dynamic).

Practical implications

Normal file IO will take many years to damage a modern (wear levelled) flash device. Swapping, however, can be much more destructive. The issue is that swapping is a mechanism used by the operating system to compensate for insufficient RAM (memory) and is depends on writing large amounts of data to "disk", and sometimes doing it a lot. This heavy usage could damage a flash device like an SD card in just a few months.

Note that swapping to a "slow device" will also make the system run slow, but on the other hand in can preclude a hard crash that could occur if the system runs out of memory.

The commands below require root (or prefix each command with sudo).

Turning off swapping

dphys-swapfile swapoff
dphys-swapfile uninstall

and also (to make it permanent):

update-rc.d dphys-swapfile remove

Turn off swapping temporarily

The following two lines work (or npt) depending on the kind of linxu distribution you have (i.e does it use systemd):

swapoff -a

systemctl disable dphys-swapfile

Totally disable the swapping mechanism for keeps

apt-get remove dphys-swapfile

Aside from swapping, you can also make the entire filesystem read-only that preventing any writes (but also limiting the kinds of things you can do).

By Gregory Dudek at | Leave a comment |    
24 June


The International Conference on Robotics and Automation took place in Brisbane, Australia this year. Despite some concerns that the regional robotics community in Australia was smaller than in other regions, the attendance at the conference exceeded any prior year. This is, no doubt, a reflection of how robotics (as well as AI and vision) are growing in significance, impact and relevance.

The industrial trade show that accompanies the academic tracks of the conference was only a small event that had only limited appeal. It has not grown to a major phenomenon with many companies of all kinds participating. This include companies that address leading edge robotics technologies, as well as organizations that use relatively small-scale (yet important) industrial automation. Some of the attendees seeks to advertise and sell technologies, while others focus on attracting attention to better hire new employees or maybe even build partnerships.

With over 1,000 research presentations, it's hard to summarize the diverse kinds of work being done. Field robotics got a fair bit of attention, which aligns well with some of the challenges in the Australian environment, as well as Canada. Rob Brooks gave a great keynote talk which commented on the importance of considered human needs and interactions, while at the same time delivering a stinging critique of classical HRI research. In fact, his critique related to much of the academic community which he felt was overly focussed on obscure or unrealistic quantitative research benchmarks, as opposed to having a greater impact on real-world needs and performance challenges. Rod has defined much of his career by railing against contemporary research trends, which is a great way to get attention. I feel his critique had some truth in it, yet also was a bit too hyperbolic since conducting scientifically valid quantifiable research is critical to ongoing progress, especially as the field matures. Doing stuff that really works in practice is critical, which was part of Rob's message, but it's also important not to use hype and public relations as the arbiters of progress. Anybody who doubts that should just reflect on the Theranos scandal.

On aspect of the conference that provoked a mixed emotional reaction from me was the fact that only a very small fraction of papers could get full-length oral presentations. While reviewing standards have increased and papers were fully-reviewed and generally of pretty good quality, there was simply not enough space or time to allow them to be presented orally and poster-style presentations were the order of the day. This seems like a development that is not likely to be reversed and we are planning something similar for ICRA 2019.

By Gregory Dudek at | Leave a comment |    
13 January

Underwater and surface robotics leaning and mapping the ocean

I am at the 2018 marine robotics field trials and things are ramping up nicely. There are a wide selection of cool projects from several universities including York, University of South Carolina, University of Minnesota and McGill University.

Our own lab is looking at a nice application of inverse reinforcement learning in the context of following divers (such as a marine biologist) and taking optimal photo documentation (such as regions of the reef of interest). We are also deploying an underwater camera to takes footage using similar motivating principles implement in a deep learning architecture.

We start our day at 8am with a morning briefing regarding the day's experiments an the previous days results, successes and challenges. Then we work at sea until dark and do development work and fine-tuning (or more) until late at night. It's exhilarating, but also exhausting.

We have another 5 or 6 applications being fielded.

By Gregory Dudek at | Leave a comment |    
10 December

Sometimes Chrome developers are too autocratic for me

Starting with Chrome 64, it seem that Chrome will no longer allow videos to auto-play on web sites that you visit. This is not the first time that Chrome has changed in a manner that does not suit my needs and I wanted to stop it from auto-updating.

Colin Witkamp at Stack Overflow observed that on OS X you can open the Info.plist file inside the app (using "Show Package Contents") and disable auto-update by altering the KSUpdateURL string which provides the URL used to check for updates. Unfortunately, this may stop Chrome from running at all due to code signing (i.e. Chrome can determine that it has been modified). To get around this too, you then need to also delete the folder inside Contents called _CodeSignature.

A cleaner way to disable auto-update is possible using the Mac OS X Terminal to enter
"defaults write com.google.Keystone.Agent checkInterval 0"
which should stop the auto-update process. I chose to do both since I have previously discovered that some applications are very determined to update themselves even when you try and stop them.

By Gregory Dudek at | Read (1) or Leave a comment |    
07 December

Fall is always a busy time in our lab and this year is definitely no exception. We are several different kinds of projects coming to fruition some of them getting ready for testing in the ocean in January. These include some small embedded devices that report data in various ways, as well as some larger oceanborn crafts that will work in collaboration with other vehicles. In addition we have some new theoretical work, a couple of big grants in play, and some exciting you industrial collaborations.

By Gregory Dudek at | Leave a comment |    
09 October

The International Conference on Robots and Systems

The IEEE/RSJ IROS conference recently took place in Vancouver and it was the biggest, and one of the best, meetings in this series of conferences to ever take place. It was also the 30th anniversary of this series of international meetings.
As usual, there were a couple of days of workshops and meetings, and three main days of technical sessions with close to 1000 original research papers being presented.

With ICRA coming to Montreal in 2019 I am paying particular attention to how other robotics meetings are being organized and what the venues are like. I must say that Vancouver really raised the bar to a frightening level (for future organizers) with a beautiful location, an excellent conference banquet, and a generally well-organized meeting overall.

Of course, learning is a key to much of robotics these days. Learning was one of the top keywords associated with published papers this year. Of the five papers from our lab, all five were about the interaction on learning and robotics (one of the papers was actually at ROSCon which preceded IROS).

Big crowd at Dieter's plenary

Although I could only see a small fraction of the talks myself, I got to see some of what the community thought were the best papers by virtue of being on the awards committee. Robot design make something of a resurgence exemplified by the best paper award going to a paper on the design and analysis of the jumping robot Salto-1P (by Haldane, Yim and Fearing). Human robot interaction continues to be a hot topic, a trend I expect to see continue for many years to come. Maja Matric gave a great plenary on her work in this important domain and urged the attendees to focus more deeply on getting robotics to people with profound needs. Among other favorite talks was the plenary by Dieter Fox and a keynote from Oliver Brock, both exceptionally creative and thoughtful souls, and also exceptional speakers -- also great dinner companions and we went out for the best sushi I've had since Tokyo. This was at Sushi Bar Maumi on Bute St. It only seats a total 11 people in groups no larger than 4, and you get to watch the Sushi being prepared at the bar in traditional Japanese style.

Sushi Maumi
Sushi Maumi
(Click to expand)

By Gregory Dudek at | Leave a comment |    

Still working as of 2016

Streaming radio has been a common use of the Internet for many years. At one point, almost all audio streaming had only limited commercial implications. As the use of advertising has become more important as a source of revenue for radio stations, audio streams on the Internet have become entwined with players, typically flash-based, that serve advertising, collect user data, and provide other services. One side effect of this is that it can may it very difficult to debug the set of network connections being used. If, like me, you run a tight firewall, you need to have a sense of where data is coming from, and going, to allow the ports to be opened and closed.

Thus, I needed to figure out how audio streams were being sent from an audio station I wanted to listen to. The audio stream I wanted was being provided from streamtheworld.com and the actual transactions are handled by a rather complex mess of javascript and flash objects. Hers's how to find the streams using wireshark, a fancy network analysis tool.

There are two steps which I elaborate below:
1) Find the call sign for the station. With luck, you can guess this. This will often (but not always) be related to the station's familiar name. For example, for the station Q107 in Toronto the nationally registered call sign is CILQ and the callsign for streamtheworld.com is CILQFM; sensible. On the other hand, the streamtheworld identifier for CBMT CBC Radio One Montreal is CBC_R1_N_MTL_H.

2) Find the playlist (.pls file) or stream ID.

Call sign identification

Firstly, note that wireshark is a very powerful open-source network analyzer (once known as ethereal). It can be downloaded free for most platforms and runs using the X-windows windowing system (that works on Linux [default], Mac [standard but optional] and [with some effort] Windows).

Turn on packet capture in Wireshark and then using your browser on the same computer load the page with the stream you want.

Enter (i.e. specify) a filter as follows (see the picture):
tcp contains "callsign"
that will be used to "listen in" on the data exchange.
This should return an HTTP GET statement of the probable form:
GET /xmlconfig.php?view=xml&callsign=CILQFM&streamid=86
and an associated full URI of the form:
It's the callsign argument you want (in this case, CILQFM, for example).

Getting the callsign using wireshark
Getting the callsign using Wireshark (click to expand)

Play list via wireshark

At this point, you can just go to
to get the playlist, which can be used to see what's really happening.
Note the use of the CILQFM call sign in the URL.

... ...
There's more. Read the whole story on "How to find an audio stream from streamtheworld.com"
By Gregory Dudek at | Read (9) or Leave a comment |    

Earlier entries...