From: Ralf Jung Date: Mon, 28 Sep 2020 17:00:00 +0000 (+0200) Subject: add Miri blog post X-Git-Url: https://git.ralfj.de/web.git/commitdiff_plain/f18b182a38fa49b778a0e7b5e326b34660763fcc?ds=sidebyside add Miri blog post --- diff --git a/personal/_posts/2020-09-28-miri.md b/personal/_posts/2020-09-28-miri.md new file mode 100644 index 0000000..e0529ad --- /dev/null +++ b/personal/_posts/2020-09-28-miri.md @@ -0,0 +1,137 @@ +--- +title: "What (not so) recently happened in Miri" +categories: rust +--- + +A lot has happened in Miri over the last year and a half, and I figured it would be a good idea to advertise all this progress a bit more widely, so here we go. +We also recently performed a breaking change that affects some CI configurations, so this post serves as an announcement for you to update your CI configuration if needed. + +For the uninitiated, [Miri](https://github.com/rust-lang/miri/) is an interpreter that runs your Rust code and checks if it triggers any [Undefined Behavior](https://doc.rust-lang.org/reference/behavior-considered-undefined.html). +You can think of it a as very thorough (and very slow) version of valgrind: Miri will detect when your program uses uninitialized memory incorrectly, performs out-of-bounds memory accesses or pointer arithmetic, violates key language invariants, does not ensure proper pointer alignment, or causes incorrect aliasing. +As such, it is most helpful when writing unsafe code, as it aids in ensuring that you follow all the rules required for unsafe code to be correct and safe. +Miri also detects memory leaks, i.e., it informs you at the end of program execution if there is any memory that was not deallocated properly. + +However, being an interpreter, Miri is limited in the kinds of code it can execute -- everything that would usually involve interacting with C libraries or the operating system needs to be specifically supported, as C code cannot be interpreted by Miri. +Miri also lacks support for some Rust features that are hard to interpret, but we are slowly closing these gaps. + + + +## Recent and past progress in Miri + +During the last 1.5 years, thanks to a series of excellent contributors, we made a lot of progress towards supporting more and more Rust code to run in Miri. +I am going to list some highlights below. + +If you want to learn how to use Miri yourself, scroll down to the end of this post. +If you are using Miri already, maybe you are still passing flags like `--exclude-should-panic` or disabling tests that require concurrency; you should be able to update those flags now. +Also note the breaking change in how `cargo miri` interprets CLI arguments below! + +### Randomness and `HashMap` + +The Rust `HashMap` picks a new random seed for each execution. +This seed in obtained from the operating system, an operation which Miri did not support until @Aaron1011 implemented `getrandom` ([#683](https://github.com/rust-lang/miri/pull/683)). +To ensure the same programs behaves the same way each time it is run by Miri, Miri internally uses a deterministic RNG (seeded with `0`, but that can be changed via `-Zmiri-seed`) to implement getrandom. +This PR also enabled Miri to be used with projects that use the `rand` crate for randomness. + +However, this also means randomness in Miri is actually not random, so *do not use Miri to perform any important cryptographic operations*. + +### Unwinding + +Miri used to just abort program execution in case of a panic. +To better match the behavior of real Rust programs, @Aaron1011 implement proper unwinding support in Miri ([#693](https://github.com/rust-lang/miri/pull/693)). +He even implemented catching panics again, which required aligning quite a few pieces across rustc, the standard library, and Miri itself. +This means Miri can finally also execute `#[should_panic]` tests. +Since recently, this is supported even for Windows targets. + +### Pointer-integer casts + +Thanks to @christianpoveda, Miri now properly supports casting arbitrary pointers to integers and back ([#779](https://github.com/rust-lang/miri/pull/779)). + +Recently, I also adjusted the alignment check to fully take this information into account, so that Miri can now run code that performs its own alignment logic ([#1513](https://github.com/rust-lang/miri/pull/1513)). +Notice however that this can lead to code that just happens to work by pure chance; to properly test such code, the test should be run at least 10 times. + +### File system access + +@christianpoveda went on to implement file system access (this series of PRs started with [#962](https://github.com/rust-lang/miri/pull/962)). +Later, @divergentdave improved that support with directory listing and some related operations (starting with [#1152](https://github.com/rust-lang/miri/pull/1152)). +This means programs running in Miri can now read from and write to files on the host computer. +This is the first form of communication that we support between the interpreted program and the outside world. +Communication needs to be explicitly requested via `-Zmiri-disable-isolation`; by default, Miri isolates the program to ensure that each execution is perfectly reproducible. + +File system access is only supported on Linux and macOS targets, but due to cross-interpretation this is not a problem even for Windows users -- see the next point. + +### Cross-interpretation + +Based on earlier work by @Aaron1011 who made Miri use check-only builds both for the standard library and the interpreted crate itself ([#1136](https://github.com/rust-lang/miri/pull/1136)), +I made Miri support "cross-interpretation" ([#1249](https://github.com/rust-lang/miri/pull/1249)). +This means even when you are on a Windows host, you can pass `--target x86_64-unknown-linux-gnu` so Miri will interpret the program *as if* it was running on Linux, in particular using all the Linux parts of the standard library for the interaction with the operating system. +Sine Miri supports the Linux APIs for file system access, it can interpret these programs even when running on a Windows host. + +This is particularly useful when testing target features that differ from the host platform: for example, even on a 64bit macOS host, you can run programs for the 32bit Linux target (`--target i686-unknown-linux-gnu`), making sure your logic works for different pointer sizes. +Miri also supports big-endian targets like `--target mips64-unknown-linux-gnuabi64`, so if your code is endianess-sensitive, you can test if it behaves correctly on big-endian systems. +And finally cross-interpretation was enormously helpful for developing Miri itself; for example, I relied on this when fixing up our panic and unwinding support for Windows targets. + +### Concurrency + +Earlier this year, @vakaras surprised me by suddenly showing up with a series of patches that equip Miri with support for concurrency ([#1284](https://github.com/rust-lang/miri/pull/1284)). +This is work he did during an internship with Amazon, so also thank you to Amazon for sponsoring this work! +Now Miri programs can spawn threads and interact via locks or atomics. +There are some caveats though: Miri does not detect data races, so programs with incorrect synchronization can cause Undefined Behavior through data races without Miri noticing. +Also Miri's scheduler is rather crude, so programs can be stuck in infinite loops under some circumstances. + +### Better `cargo` compatibility (breaking change!) + +Recently, I mostly re-wrote the main entry point for users to execute programs in Miri, `cargo miri` ([#1540](https://github.com/rust-lang/miri/pull/1540)). +It is now more compatible with cargo itself: `cargo test` and `cargo miri test` support the exact same flags, and likewise for `cargo run` and `cargo miri run`. + +However, this required a breaking change: previously, the way to pass flags to Miri itself and the program when executing the test suite was `cargo miri test -- -- `. +Now flags are passed via `cargo miri test -- ` like they are with `cargo test`; if you need to pass flags to Miri, you can set the `MIRIFLAGS` variable which works like `RUSTFLAGS`. +I also removed support for `cargo miri` without further arguments, which used to be an alias for `cargo miri run`. +The reason is that (a) `cargo miri test` is actually used much more frequently and (b) disambiguating these options while also supporting arbitrary flags is tricky. + +If you have set up your CI to run tests in Miri, please make sure to adjust your configuration to the new format. +For now, Miri still supports the old style (and emits an appropriate warning), but the intention is to remove that support code eventually. +If your project is hosted on GitHub and is affected by the change, you should have already received a notification from me, but I might have missed some projects and of course not everything is on GitHub. +While at it, you can also remove `cargo miri setup` from your CI script; that is no longer needed as thanks to @dtolnay Miri automatically detects when it runs on CI and goes into non-interactive mode. + +### ... and more + +This list is by far not exhaustive. +Many small functions, from trigonometry to environment variable access to timekeeping, have been implemented over the last months, ever growing the range of programs that Miri can execute. +Thank you to @Aaron1011, @christianpoveda, @divergentdave, @JOE1994, and @samrat! +I hope I did not miss anyone... + +## Using Miri + +If this post made you curious and you want to give Miri a try, here's how to do that. +Assuming you have a crate with some unsafe code, and you already have a test suite (you are testing your unsafe code, right?), you can just install Miri (`rustup +nightly component add miri`) and then run `cargo +nightly miri test` to execute all tests in Miri (except for doctests, which are not supported yet). +Note that this requires the nightly toolchain as Miri is still an experimental tool. + +Miri is very slow, so it is likely that some tests will take way too long to be feasible. +You can adjust iteration counts in Miri without affecting non-Miri testing as follows: +{% highlight rust %} +let limit = if cfg!(miri) { 10 } else { 10_000 }; +{% endhighlight %} +If your test suite needs to access OS facilities such as timers or the file system, set `MIRIFLAGS=-Zmiri-disable-isolation` to enable those. +(Miri will tell you when that is necessary.) +If your test suite runs into an unsupported operation, please [report an issue](https://github.com/rust-lang/miri/issues). + +If you want to add Miri to your CI to ensure your test suite keeps working in Miri, please consult our [README](https://github.com/rust-lang/miri/#running-miri-on-ci). +That document is also a great starting point for any other questions you might have. + +Miri is also integrated into the [Rust Playground](https://play.rust-lang.org/): you can select Miri in the "Tools" menu to check the code for Undefined Behavior. + +If Miri complains about your code and you do not understand why, I am happy to help! +The best places to ask probably are Zulip (the #general stream seems fine), and the Miri issue tracker. +Asking publicly is strongly encouraged so other people can help answer the question, and everyone can learn from the responses. +Questions are much easier to answer if you manage to reproduce the problem in a small self-contained bit of example code (ideally on the playground), but feel free to ask even if you do not know how to reduce the problem. + +## Helping Miri + +If you want to help improve Miri, that's awesome! +The [issue tracker](https://github.com/rust-lang/miri/issues) is a good place to start; the list of issues is short enough that you can just browse through it rather quickly to see if anything pikes your interest. +Another good starting point is to try to implement the missing bit of functionality that keeps your test suite from working. +If you need any mentoring, just get in touch. :) + +That's it for now. +I am totally blown away by how many people are already using Miri; this endeavor of re-shaping the way we approach correctness of unsafe code has been way more successful than I expected. +I hope Miri can also help you to ensure correctness of your unsafe code, and I am excited for what the next year of Miri development will bring. :D