Blog

rust

Why Rust is a great fit for embedded software

Henk Dieter
Henk Dieter

Rust is nice for a lot of things. At Tweede golf we've been using the language primarily for high-performance web applications. But that's not all Rust can do. Rust can be used to write embedded applications as well. I'd like to help you get an idea of whether Rust is a good fit for programming your next product with. Not only do I want to look at the general pros and cons, compared to the embedded lingua franca: C. I'm going to take you along writing a (very) basic embedded application in Rust as well. Rust for embedded? You be the judge! Rust for embedded. Why? And why not? Good questions, if I may say so myself. Compared to C, Rust is a relatively new language, which has its pros and cons. Like C, Rust can target low-level devices. However, Rust does a lot of things differently from C. Let's start with the things Rust has to offer. Good stuff Safety. That's one of the main selling points for Rust. It basically means that it's hard to accidentally introduce memory safety bugs when writing Rust. Rust makes unsafe operations explicit. It enforces the developer to think about what exactly they are doing. For example, reading uninitialized memory requires an explicit unsafe block. In Rust, using unsafe is like saying to the compiler: 'Even though you can't prove my code is correct, I'm sure I know what I am doing.' And then the developer can create a safe wrapper around that code, which enforces its user to interact with it in a correct manner. This way, the compiler can assure you that all non-unsafe parts are based on only well-thought-out code. Speaking of compile-time guarantees, Rust has a strong type system. Rust's type system is comparable to C's in some ways: it defines structs, enums, and comparable primitive types. One thing Rust adds on top of that is traits. These traits are a means of defining what a type can do. For instance, in embedded systems, traits can describe which CPU pins can be configured as pulse-width modulation (PWM) outputs and how they should be configured. Used in this fashion, traits enable the compiler to dodge common headaches. They are essentially the data sheet, encoded in Rust. The compiler can help you avoid using a peripheral in an unsupported manner. Type generics is another nice feature of Rust. They are a means to tell Rust that a function can take anything it likes, just as long as the type implements certain traits. For example, generics enable a developer to write a function to which any pin can be passed, as long as the pin can be configured as a PWM output. We can write a single function that can control any given PWM pin. Last but not least, Rust is backed by a great community. Many Rustaceans are very willing to help out with solving bugs and other problems. Focusing on helping you improve, most community members stay positive even when asked the most trivial of questions. Not everything is great Rust is not all kittens and rainbows, though. Rust is not as mature as C. Debugging can be quite hard in embedded contexts, and most debug tooling is focused on C or C++ development. Although it's getting better fast, debugging Rust embedded applications can be a bit dodgy sometimes. Rust being a relatively new language, there's not a lot of libraries or examples a developer can use to improve development speed. It means that sometimes, wheels need to be re-invented when developing embedded applications in Rust. And let's not forget to address the elephant in the room. Rust is feared for its most prominent feature: the borrow checker. Simply stated, the borrow checker essentially enforces two rules. The first rule is that if there exists a reference with write access to a certain part of memory, this is the only reference to that part of memory. The second rule is that if shared references to a piece of memory exist, those references cannot be used to write to that piece of memory. Rust's borrow checker is a great tool for avoiding memory safety issues. It means, however, that you can't just go and structure your application as you would in C. Many people, myself included, experienced the steep learning curve that is introduced by 'fighting the borrow checker'. Rust's safety features and generics, among other things, make the compiler do a lot of work it wouldn't have to do without them. Especially for larger projects, this can cause the compile times to go through the roof. Much effort is put into improving this, but as of this moment, it can really become an issue. Nearly every platform can be programmed using C. In Rust, this is not so much the case. Being based on LLVM, Rust only targets platforms LLVM supports. One of the best-supported platforms is ARM Cortex-M. Xtensa (ESP32) or AVR CPU's are no first-class citizens in the LLVM ecosystem, even though they are quite commonly used. Food for thought. I don't expect you to make any decisions on whether your next project should be in Rust, just based on a bunch of pros and cons. We need to go deeper! Rust embedded applications in general Embedded applications written in Rust usually have some things in common. Let's have a look at what those are. Being familiar with the options will help us when we're going to design the firmware. Embedded systems don't have an operating system as your PC does. They don't have as many resources to spare for all the fancy functionality Linux or MacOS provide. Some embedded applications are built on top of a so-called Real-Time Operating System, or RTOS. Being lightweight, they offer certain guarantees about execution timing, as well as drivers for Bluetooth and networking. But Rust can be used on bare-metal systems as well. And when we do, there's no network stack or memory allocator available. We have to tell Rust that we don't want them, as they take up too many resources. This is done by putting the #![no_std] directive on top of the main source code file. With the directive in place, Rust won't include its standard library. We can only use Rust's core functionality. Another thing Rust inserts by default is what's called a 'main-shim'. This shim loads the arguments the application was called with. There's no such thing in an embedded context. The CPU just boots and goes about doing what it is told. To disable insertion of the main-shim, we add the #![no_main] directive. Loading 'er up There are a couple of dependencies, or crates, many embedded applications in Rust use. On important one is embeddedhal. HAL is short for 'Hardware Abstraction Library'. And that's exactly what this crate is. embeddedhal provides traits that can be used to describe all kinds of peripherals and clocks. It enables you to write platform-agnostic drivers, which can run on any CPU Rust supports, and that has the right peripherals available. embedded_hal is the foundation for device-specific hardware abstraction libraries, as we'll see shortly. Applications for ARM Cortex-M-based CPUs often depend on the cortex-m and cortex-m-rt crates. cortexm provides low-level integrations with Cortex-M CPUs. Among other things, it gives us safe methods of interacting with configuration registers. The other crate, cortexmrt, helps us define the correct memory layout parameters. This is necessary in order to place the application at the correct spot in the CPUs memory. cortexm_rt also helps us define the applications' entry point with the #[entry] directive it provides. A very nice tool provided by the Rust Tools team is svd2rust. This tool can read System View Description (SVD) files, which contain information about a CPU's features. For instance, it describes where to put the bytes that need to be sent over UART. svd2rust converts these SVD files to a safe Rust API crate specific to a CPU. This type of crate is called a 'Peripheral access crate' or PAC. The PAC we are going to use is the stm32f3 crate, which provides abstractions to the peripherals of CPUs in the STM32F3xx family. PACs generated with svd2rust contain a singleton Peripherals type, which contains proxies for all peripherals the CPU has. Internally, these proxies are just pointers to certain locations in memory, called registers. They are accompanied by functions with which we can write to or read from these registers, where appropriate. The Peripherals can only be obtained safely once. This means we'll either have to do the initialization in a central spot or, better yet, pass references to the appropriate registers to modules that need access to them. The need for passing references helps rule out conflicting configurations of registers. As the references are subject to the borrow checker's rules, no two pieces of code can alter the configuration of a peripheral while the other still has access to it. Combining the output of svd2rust with the traits from embeddedhal, results in a device-specific HAL crate. These abstractions can be used to set up any platform-agnostic drivers based on embeddedhal. For the STM32F3xx family of CPUs, there's the stm32f3xx_hal HAL crate. Why am I telling you all this? Because these tools are the bedrock of the application we're going to make. Let's have a look, shall we? Let's get cookin'! Before we begin: Setting up a development environment Apart from obtaining the materials, we need to set up the development environment. This blog is not big enough to go into the details. If you want to try the code, please take a look at the Rust Discovery book. My advice is to read chapter 3 thoroughly. Apart from the tools described there, I'm using Visual Studio Code with the Rust Analyzer plugin, which provides me with code completion and other nice things. Step Blink an LED Tutorials always start with a 'Hello World' example. This tutorial is no different. Except it ends there too. The embedded equivalent of 'Hello World' is a blinking LED. Fortunately, the Discovery board comes with a couple of LEDs soldered on. Easy as it may seem, blinking an LED can be done in very complicated manners. For this step, we just want to get some code running on the CPU. We'll take the quick'n'dirty way. When we're done, we'll have some basic pin set-up code, and an endless loop which switches one of the LEDs on and off, doing some busy waiting in between. This is what that looks like: // The nostd and nomain directives you read about. #![no_std] #![no_main] // Import a directive that marks the entry point of the application. use cortexmrt::entry; // Import the embedded_hal trait implementations for the STM32F303. use stm32f3xx_hal::prelude::*; // NOP means No-Op. It's an operation that does nothing. // We use it in the busy waiting loops to notify Rust that it // should not optimize these loops out, as we actually don't want // to do anything for a while. use cortex_m::asm::nop; // A panic handler is run when the application encounters an error // it cannot recover from. The handler defines what it should do // in that case. #[panic_handler] unsafe fn panichandler(info: &core::panic::PanicInfo) -> ! { // On panic, just go do nothing for eternity, // or at least until device reset. loop {} } // This is the main function, or entrypoint of our applicaton. #[entry] fn main() -> ! { // Get a handle to the peripherals. Safe Rust allows only a single instance // of this handle. That way, accidental concurrent access is avoided. let peripherals = stm32f3xx_hal::stm32::Peripherals::take().unwrap(); // Reset and clock control register. Among other things, this register // is for enabling the General Purpose Input/Output peripherals. // We constrain full access to the RCC, allowing access per part instead. // Individual modules can configure individual parts of the RCC // independently from now on. // This gets more important in larger applications. let mut rcc = peripherals.RCC.constrain(); // The compass LEDs are all connected to the GPIO E peripheral. // Splitting the GPIO provides access to each of the individual pins, // so we can configure each of them independently. let mut gpioe = peripherals.GPIOE.split(&mut rcc.ahb); // The Northern LED is connected to pin pe9. To use it four our purpose, // we need to configure it to a push-pull output. // Import the parts of the type that describe pin pe9 when it's in // push-pull output mode. use stm32f3xx_hal::gpio::{gpioe, Output, PushPull}; let mut led: gpioe::PE9> = gpioe .pe9 .intopushpull_output(&mut gpioe.moder, &mut gpioe.otyper); // Loop forever loop { // Wait a couple of cycles for _ in 0..100_000 { nop(); } // Enable the LED led.set_high().unwrap(); // Wait some more for _ in 0..100_000 { nop(); } // Disable the LED led.set_low().unwrap(); } } Now, while Rust is pretty good at guessing which type my variables should be, I have explicitly noted the type of led: gpioe::PE9>. While that might look very scary to you, it actually illustrates what is so nice about the Rust type system in embedded context. Every pin is of its own type: gpioe::PE9. And that type is generic over the pin mode: Output. This type provides the methods sethigh and setlow. But the application won't compile if I try to read the pin's state. It should have been configured as an input for that. What's more, as a driver library writer, I can enforce my users to pass only correctly initialized peripheral references to my functions. Rust will statically check whether this is done correctly. How thoughtful of Rust to just let us know when we're doing something silly! Conclusion Using Rust for small examples like blinking an LED might seem overkill. But it does illustrate nicely how Rust's type system helps in correctly setting up device peripherals. In larger applications, this becomes a much bigger pro. Especially when multiple people are going to be writing it. And when you want to re-use the code. The Rust compiler being the strict teacher that it is, helps avoid notorious memory safety bugs. As debugging and embedded device is a lot harder than inspecting, say, a browser application, this is an essential feature in our context. And ultimately, development speed will probably be higher than it would be with C. Rust is young. It's becoming mature over time, but many things you can do with C cannot be done in Rust yet. But with devices like those from the STM32F3 family, which are pretty well supported by Rust, we think that Rust is a great language to write your embedded applications in.

Making embedded robust with Rust

Lars
Lars

Embedded software has an issue that most software doesn't: It can be very hard to get it patched. Sometimes a device hangs 5 meters high on a street light in the middle of a highway in another country. Sometimes a device is attached to a customer's heart. Sometimes strict validation requirements make changes to the software very expensive. In each case it is important to build software that doesn't fail, even in unpredictable conditions. Recently, we had the opportunity to write the software for a medical device. We had only a few months to write it before it needed to pass IEC 62304 validation. In the final phase of the project, the device was already working but we hadn't spent much time on error handling yet. In order to make our code as robust as possible and have it pass validation, we took the following approach to error handling. Categorizing errors In order to make our software robust, we've already been using the Rust programming language from the start. It has a strict type system that helps avoid many potential errors by checking them before we ever run the code. We still needed to handle hardware errors though. We began by categorizing these errors by their severity and recovery methods. IO errors Some errors are the result of simple operations that are expected to fail occasionally. These are usually operations that involve communication with a hardware device that can be affected by electrical noise. When communication with a peripheral fails, the operation can simply be retried immediately and it will usually succeed. For the case it keeps failing, we give these operations a retry limit as well as a time limit. When those are exceeded, we consider the component to have failed. Component errors Errors that can't be solved by simply retrying the operation are usually handled by resetting and reinitializing the relevant components. This can be expensive, so when you're on limited battery power, it's good to wait a few seconds before reinitializing. This both helps the remaining parts to stabilize and prevents an expensive reset loop from eating the entire battery. If the component is critical for the purpose of the device, the reset should be attempted indefinitely until the power runs out, but if the device can keep functioning without it, it should just give up at some point and leave the component disabled. Fatal errors The severest kinds of errors are the kinds that can only be resolved by power cycling or not at all. In some cases it's best to shut down and wait for the device to be retrieved for repair. In our case, we wanted to keep trying until the battery runs out, since we wouldn't need it for anything else. These kinds of errors usually occur when the software or a peripheral ends up in a bad state. For example, an SD card can switch to inactive mode if initialization fails and there is no way to get it out of that state without power cycling it. The ability to turn the power off for a peripheral cannot be taken for granted in an embedded setting. The hardware usually needs to be designed to include a software controllable power switch, or multiple, one for each peripheral. Without these, recovery from fatal errors may not be possible without human intervention. It can also be useful to turn on the watchdog peripheral, which reboots the device after a timeout. As long as the device keeps passing through its main loop, it will avoid the reboot by regularly bumping the timeout. If on the other hand it gets stuck somewhere and fails to bump the timeout for a while, it will be reset, breaking the cycle. Don't panic In Rust, most errors are reported using result types. This means the result of a function can indicate either success or some kind of error. Rust makes it easy to bubble these errors up to the points where we want to deal with them and it warns us if we ignore them. However, some sources of errors are too common to encode in the type system. Indexing an array is one example. If an index is out of range, instead of returning an error that needs to be handled all throughout the type system, Rust panics. This is a type of error that is not meant to be recoverable. On server applications, the application can usually log an error message before failing, but on embedded systems, there is usually nothing to report to, so all the device can do is reboot. Rebooting can be expensive because every component will need to be reinitialized, so it's best to avoid that. Unfortunately, Rust's type system does not encode whether a function can panic or not. The best way to find sources of panics is to search for the functions that are usually used to cause them. The most common ones are panic, unwrap, assert and expect. When we give our code a robustness pass, we search for these keywords and replace each call with one of the error handling tactics from above. This method won't help with panics caused by indexing though. Currently, the only way to protect against those is careful writing and rigorous testing. We're looking forward to the moment a feature called "const generics" becomes available in Rust, because it extends the type system so that many potential indexing errors can be proven impossible. Reporting errors Thanks to our efforts to attempt recovery from every error, we don't expect the devices to brick themselves over something minor. Nonetheless, we would still like to investigate any errors they do encounter. Our devices are offline, so they can't just send us a crash report. Instead, they log their sensor data to an SD card which is read out near the end of its life cycle. Ideally, we'd write entire stack traces to the SD card. In practice, we're on a limited power budget and SD cards use a lot of power, so we don't want to write too much. We opted to give every location in our code that can fail its own unique error tag from a single large enum. These errors only cost us 3 bytes to write. Using a single error type throughout our code base also limits the amount of error conversion code we need to write. In some cases a unique location in the source code still doesn't give us enough information. Usually, there is a high-level operation that failed and a low-level cause. In these cases, we simply log two errors in a row, making a mini stack trace of 6 bytes. We also need to watch out that we don't report 6 bytes of errors every millisecond whenever recovery fails. To solve that, we limit the number of errors that can be written in a given time period. In order to not miss out completely on errors after the limit, we keep track of flags for the peripherals and severities that those errors occured at. Concluding remarks By combining cheap yet sufficiently detailed error logging with appropriate recovery tactics, we've built a robust product that is unlikely to fail and easy to debug if it does. By relying on unwrap initially, we made it easy for ourselves to find most error sources. If you have any thoughts about this approach or would like our help developing your project, get in touch with me or Hugo. \[1\] Const generics have already been in the works for more than two years. https://github.com/rust-lang/rust/issues/44580

Implementing Lempel-Ziv Jaccard Distance (LZJD) in Rust

Henk Dieter
Henk Dieter

One of our clients helps companies in becoming GDPR-compliant. A goal is to recognize sensitive pieces of user data in a big pile of registrations, receipts, emails, and transcripts, and mark them to be checked out later. As more and more data is collected by companies, finding and eliminating sensitive data becomes harder and harder, to the point where it is no longer possible for mere human employees to keep up without assistance. A crate I’ve been working on, LZJD-rs, implements an algorithm that can be used to automate this process. It is my first serious attempt at an open source contribution in Rust, and an interesting one at that. LZJD To detect similar documents containing sensitive information, one can use the Lempel-Ziv Jaccard Distance or LZJD algorithm proposed by Edward Raff and Charles Nicholas 1]. LZJD is a means of calculating edit distance between binary sequences. It is based on [Normalized Compression Distance (NCD), which is used in malware classification software. NCD essentially is quite a simple algorithm. In NCD, two binary sequences are first compressed apart from each other. The total of the sizes of the compressed sequences is then compared to the size of the compression output of the two binary sequences concatenated. As compression eliminates duplications in sequences, the greater the difference between the compared sizes, the more the two sequences were alike. A significant portion of the work being done in NCD, like substituting parts of the sequence and writing the compressed file, can be eliminated by simply comparing the size of the substitution dictionaries. LZJD does exactly that, significantly improving the calculation speed. What’s more, LZJD only compares a small portion of the dictionaries in order to speed up the process even more. The steps taken by the algorithm in order to compare two binary sequences are roughly the following: Generate a Lempel-Ziv dictionary for each of the sequences; Generate a hash of each of the entries in both dictionaries; Sort the hashes and keep only the k=1024 smallest; Calculate the Jaccard distance between both of the lists of smallest hashes. This rather simple approach could enable scaling classification processes significantly [2]. A Rust implementation Raff and Nicholas provided a reference implementation, jLZJD, written in Java. jLZJD’s interface is a lot like that of the well-known classification tool sdhash. Given that our client has written their entire application in Rust, they asked us to create an open-source Rust implementation of LZJD: LZJD-rs. My initial implementation of LZJD-rs and jLZJD did not only differ in programming language. To optimize the architecture a bit, I introduced implementation differences as well. For instance, jLZJD uses a simplified hash set to store dictionary entry hashes in, called IntSetNoRemove. After processing the sequence, the entries in the hash set are sorted and truncated to the desired length for comparison. In contrast, LZJD-rs’ first version used a fixed-length vector in which the entries are inserted in such a way that the result is always an ordered list that can be compared immediately. To keep the entries sorted, a binary search is performed to find the spot in which a hash should be inserted. This way, no unnecessary allocations are done to retain data that will be thrown away later. pub fn frombytesstream(seqiter: I, buildhasher: &H, k: usize) -> Self where I: Iterator, H: BuildHasher, { let mut entries = Vec::with_capacity(k); let mut hasher = buildhasher.buildhasher(); seqiter.foreach(|byte| { // Update hash hasher.write_u8(byte); let hash = hasher.finish(); if let Err(insertat) = entries.binarysearch(&hash) { // If entries does not yet contain current hash if entries.len() (seqiter: I, buildhasher: &H) -> Self where I: Iterator, H: BuildHasher, { let mut dict = HashSet::new(); let mut hasher = buildhasher.buildhasher(); for byte in seq_iter { hasher.write_u8(byte); let hash = hasher.finish() as i32; if dict.insert(hash) { hasher = buildhasher.buildhasher(); } } let mut dict: Vec = dict.iter().cloned().collect(); dict.sort(); LZDict { entries: dict.iter().cloned().take(1000).collect() } } Putting it to the test One of the goals of Rust is optimizing efficiency. LZJD-rs being written in Rust, would it be faster than jLZJD? Well, let’s have a go at benchmarking both implementations. Initially, I created a basic test environment in which both implementations compare two 200MB random files, using a single thread to compute the LZ-dictionaries. LZJD-rs did it in a whopping 72 seconds on average on my machine. However, jLZJD, to my surprise, did it in about 38 seconds on average. That’s almost twice as fast! Not as easy as I thought it would be. To find out what was the cause of my implementation being slow, I used a profiling tool called Perf. Perf uses hardware and operating systems counters to mark so-called hot spots in compiled programs, so that optimization can be focused on the parts where the biggest potential gains lie. Below is a screenshot of the hottest spots found in LZJD-rs: The lines in white, starting with \_ZN, are the mangled names of the functions to which the instructions below them belong. Both red colored instructions are inside Rust’s std::collections::HashSet implementation. It would seem that mimicking jLZJD’s IntSetNoRemove in LZJS-rs or providing another more tailor-made HashSet implementation could benefit the performance LZJS-rs. Future work LZJD-rs is published on Crates.io and GitHub. I’ve been thinking about implementing a simple linear-probing hash set similar to IntSetNoRemove in order to improve performance. It would be a valuable learning experience if other developers propose optimizations. Create a pull request on GitHub and I’d be happy to review it! Citations \[1\] Raff, E., & Nicholas, C. K. (2017). An Alternative to NCD for Large Sequences, Lempel-Ziv Jaccard Distance. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. http://doi.org/10.1145/3097983.3098111 \[2\] LZJD was considered by the Dutch government for classification purposes as part of their Di-Stroy program, see https://www.linkedin.com/posts/activity-6547029450402476033-i3AJ/ (Dutch)

Meetup report: Rust for the (Inter)Net

Wouter
Wouter

At Tweede golf, we’ve been visiting meetups and conferences throughout the years. As we have high hopes for Rust, and a strong personal adoration for the language, we decided on hosting our own Rust meetup and starting a Rust Nijmegen group. The meetup was specifically oriented towards Rust in an (inter)net setting. We invited two speakers, and I took care of the third talk. We ordered some pizzas, brought a few homemade salads, and invited 35 Rustaceans. Real API's in Rust After the introductions we started the meetup with my talk titled "Real API's in Rust". I talked about our experiments at Tweede golf with building a framework for typical REST microservices in Rust. The use of Rocket and Diesel was highlighted, and how macros can be used to reduce the amount of code that actually needs to be written. At the end, the performance of various web frameworks was compared. Our conclusion is that our Rust web framework is useful, but if performance is important Hyper or Actix-web are a better fit. We will publish a blog article that discusses our Rust framework in more detail soon. Slides are here. The future web: HTTP 3 in Rust Dirkjan on the future web - HTTP3 in Rust Dirkjan talked about Quinn, his Rust implementation of the QUIC protocol, which is the underlying protocol of HTTP 3. The various problems addressed by QUIC in HTTP 3 were discussed, such as the head-of-line-blocking, and how TLS-like connections are established. Quinn provides separate crates for the QUIC state machine, the HTTP 3 implementation and an implementation that runs on Linux using Tokio. Organizing the project in separate crates like this sets an example for other crate maintainers. Dirkjan also discussed other (Rust) implementations of HTTP 3, and the politics involved coordinating between them. Slides are here. Speed up your builds with Bazel! Maarten gave us an introduction to Bazel, a build system allowing the user to define a dependency tree of build steps. The system finds out which earlier performed steps can be reused, and which need to be redone. This contrary to Make, which does not provide any guarantee about the consistency of the build products. He illustrated the expressiveness of the language used by Bazel, which is a subset of Python. He also hinted at the possibility of Bazel being very useful in a Rust context, allowing for shorter build times in a CI setting. We’re very interested in trying out Bazel in our own CI setting. Slides are here. Videos of the talks Videos of all three talks are available on Vimeo. Links to the slides are included in the video description. Please be aware we had a quite minimal video setup, resulting in a not very readable capture of the slides. You'll need to keep the slides at hand to get the full experience. Rust for the (Inter)Net Combining meetup groups During one of the breaks, the organizers of the various Rust meetup groups discussed merging into a single Dutch Rust meetup group. We will keep you posted. Looking back As this was our first meetup, we had to figure out quite a bit. Turning our office into a 30+ person conference room in a short time (and back...) and getting the number of pizzas exactly right (oops...) were among the practical challenges. On the more technical side of things, we tried to record the session so that people could (re)watch the presentations. Unfortunately, the minimalist approach to our video recording setup didn't turn out too well. The combination of a simple camera and a marginal beamer resulted in a watchable but not very readable capture of the presentation. Lesson learned: next time we’ll try to arrange a screen capturing setup, and probably a better camera and beamer. We also learned that about 60% of our guests actually showed up. Next time we’ll be more comfortable to overbook the event by about 30%, with room to spare. All in all, it was a great evening and a good experience to have under our belt. Next time we’ll be able to host another meetup with more ease and less effort. Thanks to everyone helping out, to Dirkjan and Maarten and to the attendees for the insightful questions and the interesting chats during the breaks. Feel free to contact us with feedback or ideas for a new meetup. We value your input and like to work with others to push Rust forward. See you next time!

How productive is Rust?

Wouter
Wouter

We often get the question how productive working with Rust is. "We know that it is awesome, but isn't it hard to learn? Don’t you struggle with the borrow checker?". Well, we put it to the test in Google's Hash Code 2019 programming competition. What better way to rate Rust's productiveness than to expose it to a time-pressure contest against seasoned teams using full-grown languages and ecosystems? About the contest Hash Code is a team programming competition where all teams must solve a single engineering problem within a 4-hour time window. This year, over 6000 teams from all over the world competed. We entered a team comprised of two of our developers. And of course, we chose to write our solution in Rust. The problems presented at Hash Code have always been NP-hard optimization problems, for which approximation solutions need to be implemented. Because each team only gets 4 hours to implement, but also run their solutions, they cannot be CPU intensive. The teams are ranked by scoring the results generated by their approximation solutions. The problem This year, the problem involved creating a slideshow from a set of photos. These slideshows are scored on how interesting they are. Each photo has a set of tags associated with them. Two subsequent slides are only as interesting as the number of tags that are represented on both slides, as well as the number of tags that are only on the one and not on the other slide. To make it more interesting, a slide is either comprised of a horizontal (or landscape) photo, or two vertical (or portrait) photos. The competitors The competitors we faced (language-wise) were the usual suspects for an engineering problem, notably Java, C/C++, Python, and C#. All worthy opponents, with a big ecosystem and developers who can be expected to have years of experience in the language. For this type of competition, we would expect the most from languages like C++. Programming languages used in Hash Code 2019 Programming languages used in Hash Code 2019 Our solution The slideshow problem is really hard to solve in a smart way, precisely because it is an NP-hard optimization problem. Hence we started off by just making a solution that yields a random valid slideshow. This phase was done after about 10 minutes of reading the exercise and 30 minutes of pair programming. At this stage, the code was pretty nice as well. Pair programming gave us more confidence that we weren't making any stupid mistakes, as well as ensured that both of us had a grasp of the problem and our data structures. This solution scored 287k (winning score 1250k), which is pretty good considering we just emitted random (valid) garbage. Next up we split up the team to work standalone on separate angles. My colleague continued with the randomness path, adding optimization here and there. The upside of this approach is that you get better results bit by bit. The downside is that it will probably never yield the best scores. I started to implement a classical approximation solution, namely a variant on A* search that yields better results as it goes on. As it turned out this solution would not compute fast enough through the search space to yield better results than the naive approach. The result The naive approach, in the end, yielded our best result of 733k, which landed us in 783rd place out of 6000+ teams, or to frame it a little more glamorously: in the top 12% :) Considering we were a team of two (maximum four) and we didn't have any kind of serious computing power, just our regular desktops, we were happy with our result. We could have performed way better if we had four bright minds. Our experience during the contest Rust prevented us from shooting ourselves in the foot on several occasions. And yes, there were moments where the borrow checker would not be satisfied with our code. However, what most people do not realize is that it is often fine to clone an object: as long as the cloning does not result in extra allocations, or does not occur in the innermost loop. Especially with these kinds of problems the real challenges lie in what you choose to approximate and the complexity of your solution. Rust ended up being very helpful and we managed to be as productive as we could be in any other language. Productive? Yes! Sure, this type of engineering problem is a specific use case, and it is a bit of a stretch to infer productivity from it for general purpose software development. But still, solving a hard programming problem in a short 4-hour window is definitely a scenario where you would expect a non-productive language to fail in dramatic fashion. All in all, I think it is safe to say that Rust is productive.

Our first Rust crate: decrypting ansible vaults


At Tweede golf I've been working with Rust a lot lately. My interest in Rust has been there for years, so I was very happy to start applying it in my working life, about a year ago. Since then I have worked with Rust both for our clients as well as employing it for our operations setup. I have also experimented with Rust for web \[1\]. Until now however we did not contribute to the Rust ecosystem. About time that we get our feet wet and publish our first crate! The use case: CI with Kubernetes Of course we like to continuously deploy our projects. For older projects we use Ansible \[2\] to directly deploy to virtual machines. Our newer projects however are deployed to a Kubernetes cluster hosted by Google \[3\]. In order to generate all the necessary Kubernetes Objects for our projects, we have created our own command line utility called Kuberwave. I will discuss this utility in a future blog post, but for now it suffices to say that it is written in Rust. In order to enable Continuous Deployment our CI-servers require: Credentials which grant access to the Kubernetes cluster. Credentials which grant access to a docker image registry (to enable kubernetes to pull new images). Credentials for resources like databases and third party API’s. We encrypt these secrets in so-called ansible vaults \[4\]. Ansible comes with a handy command line utility called ansible-vault to create, edit and rekey these vaults. Vaults are stored in the git repository of the project itself. As long as we only hand the ansible vault key to the people needing direct access to the kubernetes project namespace (for example, for maintenance), this setup is safe enough. Kuberwave also has the need to access these secrets. Because our staff is already comfortable with using these vaults, we’ve decided to also employ them for our Kubernetes setup. Decrypting ansible vaults For this I created ansible-vault-rs \[5\], a library that can decrypt ansible vaults. Note that it can not create or edit vaults, because I have no need (yet) for this functionality. $ANSIBLE_VAULT;1.1;AES256 30613330376362653133323866376665376365303734633836666136616531396361623761363061 3962356534626335323730643565346134663930363430320a313666376239373162636637613961 64366333313066646635343735626661366537666665653930646562643533363364363037653761 3730336235613732390a363166396664666262316632633536646332316334363361303030383036 3538 > A small ansible vault. The ansible-vault format is not pretty, or clearly documented \[6\]. I had to resort to reading the original source code \[7\] to understand how the format works. I have various problems with this format: It is purely an ascii-based file. Most of the file is comprised of base16 encoded text spaced nicely on 80 character long lines. Clearly this is not efficient, and would only be served by the fact that git diffs look nice. Except that the entire point of the file format is that the contents are encrypted, and thus look like gibberish. The file is wrapped in two layers of base16 encoded text. It becomes quite complicated to implement a streaming reader for this format. Unfortunately due to how an hmac is used, we are only able to verify the passphrase after we have read the entire file. Thus we first read the entire file into memory, verify the HMAC and only then yield the file as a complete byte buffer. The format would benefit from per-block HMAC verification. Our implementation is thus not able to stream the file. For our use-case the vaults are rarely more than a few kilobytes in size. Also we will only parse a handful of files in a run. Most of the resources go into computing the derived key of a vault using PBKDF2. For us this is not a problem, but it might be to others wanting to decrypt large files or a large number of files. If I ever have to store and encrypt large files in a git repository (probably not, due to the nature of git repositories) we will consider moving to a different file format. For now Ansible vault serves our purpose just fine. Publishing the crate Initially the implementation was written as part of Kuberwave, but as an exercise I decided to publish it to crates.io. I added more explicit error handling (as per \[8\]) and added a few tests. When creating a separate cargo.toml file and re-adding the specific dependencies, I noticed I was using a deprecated package: rust-crypto. \[9\] This package has not been updated since May 2016. For a crypto library this is quite dangerous. Cargo would benefit greatly from explicitly deprecating this package, as already suggested in an issue \[10\]. I re-implemented the crypto to use the RustCrypto \[11\] family of crates. The only thing still missing is more extensive documentation. I will probably also implement creating vaults. I am already working on a tool to automagically generate the Kuberwave configuration for simple projects, which will (probably) need this. Crates are easy Publishing the crate to crates.io was easy a pie \[12\]. I only had to register to crates.io as a publisher, add some fields to the cargo.toml file, and run cargo publish. Even though this crate will probably not be used widely and isn't all that exciting, it does what it is meant to do, and I hope some of you can benefit from it. For me it has shown Rust's package manager is easy as well as powerful. The next time I write something in Rust that is general enough to open source it, we will do so. \[1\] https://tweedegolf.nl/blog/23/rust-als-webplatform (Dutch) \[2\] https://github.com/ansible/ansible \[3\] https://cloud.google.com/kubernetes-engine/ \[4\] https://docs.ansible.com/ansible/2.4/vault.html \[5\] https://crates.io/crates/ansible-vault \[6\] https://docs.ansible.com/ansible/2.4/vault.html#vault-payload-format-1-1 \[7\] https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/vault/init.py \[8\] https://doc.rust-lang.org/rust-by-example/error/multipleerrortypes/defineerrortype.html \[9\] https://crates.io/crates/rust-crypto \[10\] https://github.com/DaGenix/rust-crypto/issues/440 \[11\] https://github.com/RustCrypto \[12\] https://doc.rust-lang.org/cargo/reference/publishing.html

Rust als webplatform?!


Wat is over 5 jaar het winnende open source webplatform? Inmiddels begint PHP - ondanks PHP7 - zijn ouderdom te tonen. Alternatieven als NodeJS zijn wel werkbaar, maar in lang niet alle scenario’s geschikt. We vertellen je graag waarom wij denken dat Rust de nieuwe speler kan worden voor high-performance applicaties op het web. Ja, PHP is oud: het barst van de legacy code, de snelheid laat te wensen over, en er is eigenlijk nooit sprake geweest van een net ontwerp van de taal \[1\]. Wij gebruiken PHP - d.w.z. het backend framework Symfony - omdat het ecosysteem, bestaande uit de tooling en beschikbare open source bundles, inmiddels heel volwassen is. Wil je snel een goed schaalbare web app bouwen met open source technieken? Dan is PHP met Symfony momenteel nog steeds The Way To Go. Kijken we verder in de toekomst en nemen we de steeds hogere eisen aan security en performance mee, dan is het duidelijk tijd voor ons om op zoek te gaan naar een moderner, veiliger en sneller alternatief voor PHP/Symfony. Waarom geen Node.js? Javascript is booming. Nieuwe open source projecten rondom het Javascript-platform schieten als paddenstoelen uit de grond. Helaas wordt Node.js veelal geplaagd door vergelijkbare problemen als PHP. Het is óók heel traag. De tooling rondom Javascript is in elkaar gebeund \[2\]. En ook het taalontwerp is langzaamaan gegroeid, wat de taal niet altijd even logisch maakt \[3\]. Browsers hebben dan ook moeite om deze ontwikkelingen bij te houden \[4\]. Nog meer dan bij PHP is het lastig om correcte code in Javascript te schrijven. De opkomst van (gecompileerde) talen voor type annotaties in Javascript \[5\] \[6\] helpen hier een beetje bij. Deze annotaties geven echter geen feitelijke garanties wanneer je de code draait, en zijn daarmee eerder een pleister over het probleem dan een echte oplossing. Rust Een snelle opkomer is de programmeertaal Rust \[7\]. Deze taal bestaat sinds 2010, en wordt actief gesponsord door Mozilla. In de basis is het een systeemtaal: het lijkt erg op C++, en het genereert ook een uitvoerbaar bestand met AMD64 assembly. Het is dus geen scriptingtaal zoals PHP en Javascript. Voordeel hiervan is dat je code volledig gecontroleerd is, voordat er ook maar iets uitgevoerd wordt. Zo heb je meer zekerheid dat wat op je servers draait ook ècht goed is. Veiliger dan C++, sneller dan Go Ditzelfde geldt ook voor de C++, maar bij C++ kan gemakkelijk ongeldig geheugen geadresseerd worden. Het komt vrij vaak voor dat er achteraf buffer overflows in C++-code blijken te zitten. Dit probleem is zo goed als onmogelijk als de code wordt geschreven in Rust, dankzij een goed ontwerp van de standaard library en de aanwezigheid van het Borrow Checker-mechanisme \[8\]. Dit component controleert bij het compileren dat er alleen kan worden gewerkt met bestaande objecten in goed georganiseerde stukken geheugen. Garbage Collection is het gangbare alternatief voor het Borrow Checker-mechanisme. Een Garbage Collected taal zoals Go biedt op die manier vergelijkbare voordelen als Rust. Helaas heeft Garbage Collection performance-nadelen, en geeft het minder grip op geheugengebruik. Samengevat is Rust dus minstens zo snel als C++, en heeft het bovendien de correctheidsgaranties van talen als Java, Go en functionele programmeertalen. Dat maakt Rust zeer geschikt voor toepassingen waarbij performance en security cruciaal zijn. Deze garanties komen ook goed uit bij het ontwikkelen van embedded systemen zoals in een Internet of Things-context. Rust compileert naar LLVM, en kan dus ook code genereren voor ARM chipsets \[9\]. Nog een klein zijspoortje naar Go, omdat het gezien wordt als een uitdager voor Rust: Bij Tweede golf kiezen we niet voor Go o.a. omdat we het typeringssysteem te beperkt vinden. Zo ontbreekt het bij Go aan generics, waardoor de correctheidsgaranties toch minder sterk zijn in vergelijking met Rust. Er valt veel meer te zeggen over deze afweging, maar dit is geen "Rust vs. Go"-artikel \[10\]. Rust als webplatform Als we de Fibonacci-reeks kunnen uitrekenen hebben we nog geen webapplicatie. Eerst hebben we een webplatform voor Rust nodig dat de functionaliteiten biedt die normaal Symfony voor ons regelt. Dit webplatform bestaat simpelweg nog niet \[11\] \[12\]. Omdat we niet bang zijn om onze handen vuil te maken, zijn we een paar maanden geleden begonnen zelf zo'n raamwerk voor backends te ontwikkelen. Hiermee hopen we de drempel om Rust te gebruiken weg te nemen, in eerste instantie voor onze eigen ontwikkelaars. Ons raamwerk bestaat uit en gebruikt: REST, HTTP server en routering: Rocket \[13\] Input validatie: Serde \[14\] ORM / Database: Diesel \[15\] + Postgres Authenticatie: JSON Web Tokens met Medallion \[16\] Onze ervaringen tot nu toe? Wat we ten eerste merken is dat Rust en Rocket moeiteloos onze performance-eisen halen \[17\]. Verder wordt de documentatie van elk stuk Rust-software via Cargo \[18\] met een eenduidige stijl gegenereerd. Er is dus een basisniveau van referentiemateriaal voor alles in het Rust-ecosysteem. We zien ook dat elk van deze libraries en Rust als taal een heel levendige community hebben. Issues en merge requests worden ongelofelijk snel opgepikt en opgelost. Zijn er ook nadelen? Rust voor web klinkt dus veelbelovend. Soms is de community wel net iets tè levendig: onder andere Diesel heeft tijdens de ontwikkeling van ons platform ten minste eenmaal de API volledig omgegooid. Hopelijk kalmeert dit wanneer libraries meer in productie worden ingezet. Ook heeft Rust een steile leercurve: het is moeilijk om de Borrow Checker gerust te stellen dat het geheugen overal goed wordt gebruikt. Even snel een web app programmeren in Rust zit er hierdoor waarschijnlijk niet in. En ook niet iedere programmeur zal affiniteit kunnen kweken en in de taal kunnen werken. Toekomst voor Rust Hoewel we bij Tweede golf erg gecharmeerd zijn van Rust, zullen we voor minder veeleisende applicaties Node.js als backend platform blijven gebruiken. Ook Symfony zal voorlopig belangrijk blijven. Voor high performance backendsystemen die degelijk en duurzaam moeten zijn denken we echter dat Rust de toekomst is. In een volgende blogpost zullen we dieper ingaan op de ontwikkeling van ons Rust webplatform. Overweeg je Rust al voor je product (applicatie of embedded systeem) of wil je de mogelijkheden verkennen? Onze teams helpen je met prototyping of implementatie. Neem contact op met Erik of Hugo. Referenties: \[1\] https://whydoesitsuck.com/why-does-php-suck/ \[2\] https://ponyfoo.com/articles/npm-meltdown-security-concerns \[3\] https://www.destroyallsoftware.com/talks/wat \[4\] https://caniuse.com/ \[5\] https://coffeescript.org/ \[6\] http://www.typescriptlang.org/ \[7\] https://www.rust-lang.org/ \[8\] https://doc.rust-lang.org/1.8.0/book/references-and-borrowing.html \[9\] http://blog.japaric.io/quickstart/ \[10\] http://julio.meroh.net/2018/07/rust-vs-go.html \[11\] https://www.arewewebyet.org/ \[12\] https://github.com/flosse/rust-web-framework-comparison \[13\] https://rocket.rs/ \[14\] https://serde.rs/ \[15\] https://diesel.rs/ \[16\] https://docs.rs/medallion/2.2.3/medallion/ \[17\] https://medium.com/sean3z/rest-api-node-vs-rust-c75aa8c96343 \[18\] https://doc.rust-lang.org/beta/rustdoc/what-is-rustdoc.html