Raspberry Pi in Secure Embedded Systems: WTF? (What? That Fails)

Although I haven’t made much use of the Raspberry Pi as an embedded CPU, I’ve always wanted to make more use of it and I constantly promote the little board as such a tool. The Pi isn’t perfect, but at the price point and hardware capabilities, I’ve always thought it was a great tool for many embedded uses despite a common joke I repeat about the Pi:

“The Pi is great for embedded system except during three different times of execution: during startup, during normal execution, and during shutdown.”

The follow up is this:

During startup, the Pi takes too long to boot. Many embedded system require near-immediate operation (operating within a couple of seconds) and can’t tolerate the 30 seconds to a couple minutes boot time of Pi OS.

During normal operation, the Pi cannot support real-time response, so it is completely unsuitable for those applications requiring real-time operation.

During shutdown, you must explicitly shut down the Pi (i.e., you cannot simply cut power to it without risking damage to the SD card).

Very little can be done about the boot time. Fortunately, not all embedded applications require immediate operation after applying power, so although the Pi isn’t usable for all embedded application, it can still be used for many that can tolerate a slow startup period.

If your application requires real-time response, you’re not going to get it out of Pi OS. Fortunately, not all applications require this type of response time. Furthermore, for many applications that do require real-time response, it’s easy enough attach a small SBC (such as a Teensy 4.x or other Arduino-class device) to handle the real-time portions of your application.

Most embedded systems are shut down by removing power from device. That’s a very bad practice for Linux-based systems (such as Pi OS). Fortunately, there is a hardware solution for the Raspberry Pi 3: https://juice4halt.com. This is a supercapacitor-based “uninterruptable power supply” module that will provide about 30 seconds of power to the Pi when power is removed. This device sends a signal to the Pi that power has been lost so it can shut down in a safe fashion before the super capacitor supply is exhausted.

I actually used a Pi 3 (with a Pi touchscreen display) in a Nuclear Fuel Temperature Monitor (NFTM) device I installed at National Tsing-Hua University (NTHU) in Taiwan. This device monitors the nuclear fuel temperature and SCRAMs the reactor if the fuel temperature exceeds some set point (typically around 500C). FWIW, the FTM contains three main components: an analog trip system (SCRAMming the reactor is always done by analog circuitry, never by software; this makes the regulators much happier), a Weensy 4.1 board (based on the Teensy 4.1 module), and a Raspberry Pi 3. The Raspberry Pi is basically a glorified video display card for the system, providing a user interface and temperature display. The Weensy provides all the real-time control. The analog trip board handles the actual reactor SCRAMs.

Originally, I had planned to have the Raspberry Pi also provide an Ethernet interface to the reactor console. This feature was never needed at NTHU, so I never finished that code and left the Ethernet jack on the Pi disconnected. This turned out to be a very good thing.

Recently, I was involved with another reactor project at Idaho National Labs (INL). On their NRAD reactor (a small TRIGA reactor used for neutron radiography), they wanted to be able to monitor the reactor operation remotely. The NRAD reactor uses a General Atomics digital control console (Ethernet based). They wanted to pass some operational data from the local reactor network to the general INL network. Now, you *DON’T* connect a reactor’s network directly to any outside network. The NRC (or DOE in INL’s case) doesn’t allow this. However, by using some special (uber-firewall) equipment that guarantees a one-way transmission from the reactor’s network to the outside world, it is possible to satisfy regulatory agencies and they will allow this.

So I designed a two-Pi system that included one Pi on the local reactor network that received UDP messages from the reactor’s computer. This “inside Pi” retransmits the UDP packets across the uber-firewall to the “outside Pi” sitting on the INL network (which is highly controlled and not an “open Internet” type network). This Pi makes the data available to INL services (using TCP/IP transmissions).

I developed the software, tested it out on the reactor (without connecting it to the INL network, using a Pi 400 to simulate the INL server), and got it all working nice and pretty. Then the day arrived that we plugged it into the network. All of a sudden I’m getting calls from the INL IT crew: “why is your machine trying to contact Google? Why is it slamming our DNS? We cannot allow this device on our secured network.”

Now I knew that the Pi would try to contact an NTP server to get the time. So I had installed an Adafruit DS3231 RTC on the Pi (and properly installed the drivers; see https://www.adafruit.com/product/4282) to avoid this. Contacting Google? That was new to me. Slamming their DNS? No idea what was going on there. Needless to say, this is not a behavior you want from a system connected to a nuclear reactor (or even on a secured network operated by the Department of Energy).

A couple hours’ spent trying to figure out what was going on didn’t turn up anything. By this point, the INL IT staff was telling me there was no way they were going to allow a Pi running Pi OS on their network, so I abandoned further research on the matter. They suggested installing a version of Ubuntu. However, I wasn’t sure whether this would support the Juice4Pi drivers or even the Pi touchscreen. So I had to abandon the Pi and switch to a small Windows-based PC.

Perhaps, at some point, I’ll look back into what caused this problem. Until that day arrives, however, I’m very reticent about recommending the Pi for embedded applications. I just don’t feel good about machines accessing things on the Internet that I haven’t explicitly asked them to do. This could be something innocuous, such as automatically checking for updates (though, why Google?). However, it should be documented and easy to shut off. Until then, caveat emptor is the key word here.