Contributing to Rust in 2017 and 2023
June 02, 2024 -This is a comparison of the contribution experience to the Rust standard library in 2017 and 2023. I have the possibly unique perspective of having contributed to the Rust standard library twice separated by many years.
2017
In 2017, I was a PhD student was working on the Noria project, which was a
research project to build a new kind of database. While working on Noria, I ran
into a limitation of the standard library's BufReader
type. The BufReader
type is a type that wraps a reader and provides a buffer to read from. The
problem was that there was no way to check if the buffer was empty without
reading from it.
After discussing it with a labmate, I started by creating an issue:
@fintelia commented on Oct 16, 2017...
There is currently no way to tell whether there is any data buffered inside a BufReader. This is unfortunate because it means that an application has no way to know whether calls to read() and fill_buffer() will return instantly or trigger a potentially expensive call to the underlying Reader. I propose adding an is_empty() method to BufReader to fill this gap.
And then two days later followed up with a PR:
#[unstable(feature = "bufreader_is_empty", issue = "45323", reason = "recently added")]
pub fn is_empty(&self) -> bool {
self.pos == self.cap
}
After a brief bit of discussion, the PR was merged on Nov 6, 2017 and was included in the next nightly build. Then about a week later we started using the new method in Noria.
From here the process started to draw out. The functionality I needed was
already implemented, so I wasn't in a hurry. Over the next few months discussion
continued on the tracking issue. There was a FCP (final comment period) to
stabilize the method, but it was called off in favor of a better approach: a
BufReader::buffer
method.
In July 2018 I checked back on the tracked issue to ask it the new approach was
ready to stabilize. A project member told me that it was still blocked on
someone implementing BufWriter::buffer
which had been suggested as a
follow-up. Three months later someone else asked in the tracking issue if
there'd been any progress. Noticing that the other method still hadn't been
added, I went ahead and made a PR:
#[unstable(feature = "bufreader_buffer", issue = "45323")]
pub fn buffer(&self) -> &[u8] {
&self.buf
}
This one proved slightly harder to get reviewed. I had to ping my rustbot assigned reviewer twice and a triage team member also had to ping the issue. But the whole process concluded in about a month.
Then in January 2019 I again bumped the main tracking issue to ask if the new methods were ready to stabilize. After two more months with no reply someone else commented on the issue asking if there was any reason not to move forward, so I tried tagging the project members who'd been involved previously. After another month without reply I tried pinging the other project members tagged by the earlier FCP. This time it landed back on the libs team's radar and they decided to stabilize the methods. The stabilization PR merged on May 29th and Rust 1.37.0 including the new methods was released on Aug 15, 2019.
All told, the process took just under a year and ten months from the initial PR to stabilization.
2023
By 2023 I'd been maintaining the image
crate for a few years. The crate
depends on a number of format-specific crates to encode and decode images in
various formats. One problem I'd encountered was that Rust standard library's
io
traits had some annoying limitations that made it hard to write efficient
image decoders. In particular, I learned that the BufReader
type didn't have
an efficient implementation of the Seek
trait due to an API limitation.
Other than a documentation PR, I hadn't contributed to Rust since 2019. Thus I went looking for updated guidelines on contributing. I didn't find clear guidance but it seemed that the right thing to do was to create a Pre-RFC on internals.rust-lang.org:
@fintelia - Sep 10, 2023
BufReader::seek_relative
was added to address #31100. As explained in that issue,BufReader
's stable API required that the internal buffer was dumped on any call to seek, so the new method was added to expose a way to seek without doing dumping the buffer.Unfortunately, there isn't really a good way to take advantage of the method in generic code. If a method takes a
R: Read + Seek
, then only the regular (inefficient) seek implementation is exposed. The method could itself create aBufReader<R>
and then call seek_relative, but then depending on whatR
is, it might end up making aBufReader<BufReader<File>>
orBufReader<Cursor<Vec<u8>>>
or some other silly type.If the
Seek
trait had a seek_relative method (with the straightforward implementation as a default impl) then this could be solved.
I didn't get much of a response on the Pre-RFC, so after a month I went ahead and made a PR:
/// trait Seek
#[unstable(feature = "seek_seek_relative", issue = "none")]
fn seek_relative(&mut self, offset: i64) -> Result<()> {
self.seek(SeekFrom::Current(offset))?;
Ok(())
}
// struct BufReader
fn seek_relative(&mut self, offset: i64) -> io::Result<()> {
self.seek_relative(offset)
}
A project member quickly responded to the PR and explained that API change proposal was required to add new methods to the standard library. I created an issue on the rust-lang/libs-team repository as requested, and it was accepted two days later. Around this time non-Rust factors interfered with my ability to work on the feature, though over the next few weeks I did create a tracking issue for it and respond to some minor feedback. By mid-November the implementation PR was merged.
Next started the first waiting period. Given that the library I was maintaining targeted stable Rust, the availability of the feature on nightly was of no use to me. However, the project expects all features to stay in nightly for an indeterminate period of time before stabilization.
I wasn't sure how long to wait, but four months later tried I bumped the tracking issue asking whether the feature was ready for a FCP. It got no response so two weeks later I posted in the rust-lang zulip chat asking about stabilization. After another week, a project member initiated the final comment period, which concluded without any objections.
Someone else then went ahead and made a stabilization PR. Sadly, it didn't get a response so after two weeks I tried pinging the assigned reviewer. After two more weeks without a response, I asked in the zulip chat what the right thing to do was. Another commenter suggested the proper rustbot command to reasign the reviewer. Shortly after, the PR was merged. Unfortunately, the delay reviewing the stabilization PR caused it to miss the merge window so it'll have to wait over ten weeks for Rust 1.80 to reach stable on July 25, 2024.
Of course, the process won't end there. The image
crate -- like much of the
Rust ecosystem -- doesn't force upgrading to new rustc versions immediately
after they're released. It instead supports the last several Rust versions,
which will mean that the new method won't be usable until several months after
it's stabilized.
All told it'll be about 15 months from the initial Pre-RFC to when I'll actually
be able to use the new method in the image
crate. Which is a pretty
significant ordeal for a feature I've never actually used yet.
Takeaways
When I got started writing this article I was expecting to find that the process had gotten slower and more circuitous. It certainly felt that way. However, the actual difference was more in the non-process factors. The actual time time to stabilization was "only" ten and a half months, almost a year faster than the previous time!
In 2017, I was using nightly Rust and was able to use the new method almost immediately after it was merged. This was typical for the time. Rust 1.0 had only been released a few years earlier and the ecosystem was still in a state of flux. By 2023 stable Rust was the norm, particularly for libraries. And not just latest stable, but the expectation of a minimum supported Rust version that is at least several releases back. The net result is that instead of waiting one month to use a new feature, I'll be waiting about 15 months.