Rails test suite benchmark: Apple Silicon M1 vs. Intel Macbook Pro

The Orbit engineering team has been following the debut of the exciting new Apple Silicon M1 lineup for a few months now.

As the CTO, my philosophy is to invest in the best hardware possible for our team including laptops, monitors, and other peripherals like high-quality cameras and microphones. I find this makes a big difference in productivity, health, and fostering connections amongst the team. I take the time to evaluate and benchmark new hardware when it becomes available to see if we have something collectively to gain. I also keep a Notion table of all the hardware I've purchased and reviewed as a resource to help new team members make their selections.

Now, on to the much-anticipated Intel vs. Apple M1 benchmark. In this post, I'll share the numbers I have from running Orbit's Ruby on Rails test suite on 3 different Mac machines:

• 2018 Mac Mini - Intel i3, 4-core, 8 GB RAM (Macmini8,1)
• 2019 16" Macbook Pro 2.4Ghz i9, 8-core, 32 GB RAM (MacBookPro16,1)
• 2020 Mac Mini Apple M1, 8GB RAM (Macmini9,1)

Caveat: this is not a super-scientific, controlled benchmark. Other programs were running at the same time, though I did my best to make sure nothing intensive was going on. Numbers reflect the lowest score of 5 back-to-back runs, which may minimize actual differences.

Orbit's Rails test suite

Orbit is currently running on Rails 6.0.3.4 and ruby 2.7.2p137. We also have a Node.js runtime used by some parts of the Rails app running on node v12.13.1. The underlying database is Postgresql and we also use of Redis for caching and Sidekiq.

The test suite is built with RSpec and consists of 1403 specs. 757 are model specs. 308 are request specs. There are a few Capybara specs that run a headless browser. The rest of the specs fall into different categories including serializers, coponents, services, controllers, helpers, mailers, and workers.

Benchmarks

Running the RSpec suite

The 2019 16" Macbook Pro Intel i9 ran the specs fastest at a total time of 1 minute 8 seconds. Next was the 2018 Mac Mini Intel i3 at 1 minute 19 seconds with the 2020 Mac Mini M1 right behind at 1 minute 22 seconds.

The Mac Mini M1 numbers come from running everything in x86_64 compatibility mode using Rosetta 2. That means that all instructions are translated before they are executed, adding a lot of overhead. We expect the native ARM mode to be a lot faster but haven't gotten the environment fully working natively yet.

Caveats

Benchmarks are not perfect, and this one doesn't really capture the overall feel of the 3 machines. The M1 Mac Mini scores lower than the older Intel Mac Mini in this simple, single-theaded benchmark, but under real-world conditions it's significantly faster and more responsive, in some cases even more than the 16" Macbook Pro, a machine that costs nearly 4 times as much.

On the M1 Mini I can run OBS @ 60 fps, Zoom, and Rails specs at the same time without any degradation in performance, fans, or heat. I can do the same thing on the 16" Macbook but the fans will max out, which is problematic because it creates background noise in the audio. Personally, fans give me anxiety even though I know that is irrational. The M1 Mini stays silent and cool.

The 2018 Mac Mini Intel i3 does well to run the Rails specs when nothing else is going on, but otherwise it feels sluggish and test times can go up as high as 120 seconds if other intensive applications are running. The spec-running variance is much higher than the other 2 machines, even though it's fastest possible run is still comparable. Using OBS on the Intel Mac Mini has not been a good experience, the load average and CPU are too high for the stream to be very stable.

Reloading web pages in Chrome in the development environment feels fastest on the M1 Mac Mini, followed closely by the 2019 16" Macbook Pro. The terminal scrollback in tmux + iTerm2 on the M1 feels particularly fast and light, I would guess due to the improved GPU performance of the M1.

Booting a Rails environment

In terms of Rails boot-up, the M1 was the fastest at 3.39 seconds, followed by the 16" Macbook pro at 4.92s and the Mac Mini at 5.72s. This is *very* good and also promising for what native performance might look like, especially given the frequency of needing to boot up a Rails environment or restart spring.

Temperature observations

After successive runs on the 16" Macbook Pro, the CPU cores (according to iStat Menus) hover around 60-80 degrees celsius. Whether or not this triggers the fans depends on what else is happening on the machine. Running specs while on a video call or using OBS will definitely get the fans going and the laptop surface to a very hot temperature.

After successive runs on the Mac Mini M1, the PMG SOC Die iStat Menu monitor hovered around 20-25 degrees celsius. Basically room temperature. No sensor was higher than 35 degrees at any point during the run. The M1 runs very cool, and with these numbers it's easier to see how the M1 Macbook Air doesn't need fans and still achieves this kind of performance.

M1 = Impressive!

The M1 Mac Mini is a powerful machine and extraordinary considering the price. The model tested here was 779€ purchased in January 2021. For comparison, the Intel Mac Mini was 900€ purchased in September 2020. The 16" Macbook was $3,099 in January 2020.

While the M1 doesn't win the benchmark today, I suspect it will beat even the 16" Macbook Pro when running in native ARM mode. I also suspect that it may be very good at running tests in parallel due to the proliferation of cores and the unified memory architecture, and do it in a way that doesn't bring the rest of the machine to its knees. I'm eager to give that a try and tell you how it goes.

Happy computing in 2021!

We're hiring!

Orbit is hiring engineers to join our distributed engineering team. We're building an exciting new product & API for developer advocates and community leaders using Ruby, Rails, React, and lots of other old and NEW MAGIC. If you enjoy building beautiful software, geeking out about hardware, or space puns, you might enjoy working here. Check out our careers page or our Key Values page to learn more.