From e99d9f53e5557a1cc593728a0ad5c91082518c09 Mon Sep 17 00:00:00 2001 From: apiraino Date: Mon, 13 May 2024 16:21:46 +0200 Subject: [PATCH 01/38] Update Zulip issue team nomination template --- src/triagebot/zulip-notifications.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/triagebot/zulip-notifications.md b/src/triagebot/zulip-notifications.md index 46d9b70ce..5d8f190ae 100644 --- a/src/triagebot/zulip-notifications.md +++ b/src/triagebot/zulip-notifications.md @@ -46,7 +46,9 @@ message_on_close = "Issue #{number} has been closed. Thanks for participating!" message_on_reopen = "Issue #{number} has been reopened. Pinging @*T-types*." # The Zulip notification will not be posted unless the issue/PR has all of these labels. -required_labels = ["I-nominated"] +# Please replace the `{team}` placeholder with the appropriate team to be notified for the nomination +# (ex. `I-compiler-nominated`, `I-lang-nominated`, ...) +required_labels = ["I-{team}-nominated"] ``` ## Implementation From 359e7184bac718abfe5234c07b6ffe41b0d60e60 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Le=C3=B3n=20Orell=20Valerian=20Liehr?= Date: Mon, 13 May 2024 18:36:29 +0200 Subject: [PATCH 02/38] Tweak and update subsection Labels of Issue Triaging --- src/release/issue-triaging.md | 53 ++++++++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 14 deletions(-) diff --git a/src/release/issue-triaging.md b/src/release/issue-triaging.md index aeeb77398..070604a29 100644 --- a/src/release/issue-triaging.md +++ b/src/release/issue-triaging.md @@ -75,25 +75,50 @@ Triaging them the same way as `needs-triage` is also useful. There are many different labels that can be applied to issues. - `needs-triage`: Signals that an issue is new and needs initial triage -- `T-*`: Specifies the team or teams that this issue is relevant to, for example compiler, types or libs -- `WG-*`: Specifies the working groups that this issue is relevant to, for example WG-debugging. -- `C-*`: Specifies the category of the label, for example a bug, tracking issue or discussion - - `A-diagnostics` issues usually don't have any C label. +- [`T-*`]: Specifies the team or teams that this issue is relevant to. For example `T-compiler`, `T-types` or `T-libs`. +- [`WG-*`]: Specifies the working groups that this issue is relevant to, for example `WG-debugging`. +- [`PG-*`]: Specifies the project groups that this issue is relevant to, for example the `PG-exploit-mitigations`. +- [`C-*`]: Specifies the category of the label, for example a bug, tracking issue or discussion + - `A-diagnostics` issues usually don't have any `C-*` label. - Also note `C-optimization` for missed compiler optimizations. -- `O-*`: For platform-specific issues, specifies the platform (architecture or operating system). For example macos, aarch64, windows -- `A-*`: The areas that the issue is relevant to, for example linkage, patterns, diagnostics -- `F-*`: When the issue concerns a specific (usually unstable) feature +- [`O-*`]: For target-specific issues, specifies the compile target[^1] or compile target family (most notably the platform, i.e., the architecture or operating system). For example `O-macos`, `O-aarch64`, `O-windows`, `O-windows-msvc`. +- [`A-*`]: The areas that the issue is relevant to, for example `A-linkage`, `A-patterns`, `A-diagnostics`. +- [`L-*`]: When the issue concerns a specific lint. +- [`F-*`]: When the issue concerns a specific (usually unstable, usually language) feature. +- [`-Z*`]: When the issue concerns a specific unstable `-Z` compiler flag. - `requires-nightly`: This issue is not relevant to the stable compiler - `requires-{incomplete,internal}-features`: This issue requires an incomplete or internal feature. The latter often means that the issue should be closed in accordance with compiler [MCP 620](https://github.com/rust-lang/compiler-team/issues/620). -- `regression-*`: Labels for tracking issues that are regressions. -- `D-*`: Labels for diagnostics issue. -- `I-*`: Different labels about the nature (originally, importance) of a bug. For example ICE, slow code, heavy code (binary size), crashes, unsoundness. - There are also some other `I-*` labels that don't really fit into this. For triaging, focus on `I-slow`, `I-heavy`, `I-ICE`, `I-crash`, `I-unsound`. -- `P-*`: Priority labels. Applied using the [Compiler Prioritization procedure](../compiler/prioritization.md) -- `S-*`: The status of an issue, for example S-needs-repro. -- `E-*`: Calls for participation, for example to minimize an issue +- [`regression-*`]: Labels for tracking issues that are regressions. +- [`D-*`]: Labels for diagnostic issues. +- [`I-*`]: Different labels about the nature[^2] of a bug. For example ICE, slow code, heavy code (binary size), crashes, unsoundness. + There are also some other `I-*` labels that don't really fit into this. For triaging, focus on `I-ICE`, `I-crash`, `I-hang`, `I-slow`, `I-heavy`, `I-compiletime` and `I-unsound`. +- [`P-*`]: Priority labels. Applied using the [compiler prioritization procedure](../compiler/prioritization.md). +- [`S-*`]: The status of an issue, for example `S-needs-repro`. +- [`E-*`]: Calls for participation[^3], for example to minimize an issue. - `E-mentor`: A mentor is available to help with the issue, which makes for good first issues. - `E-needs-mcve`: This issue has a reproduction, but it is not minimal, it should be minimized. + - `E-needs-bisection`: This issue needs a bisection, for example using [cargo-bisect-rustc](https://github.com/rust-lang/cargo-bisect-rustc). - `E-needs-test`: The issue has been fixed, but no test has been added for it. After someone adds a test, it can be closed. - `E-{easy,medium,hard}`: Someone has estimated how hard the issue is to fix. This can help with finding good first issues, but is [bound to be inaccurate](https://en.wikipedia.org/wiki/Curse_of_knowledge). + +See also section [Issue Triage](https://rustc-dev-guide.rust-lang.org/contributing.html#issue-triage) in the Rust Compiler Development Guide. + +[`T-*`]: https://github.com/rust-lang/rust/labels?q=T- +[`WG-*`]: https://github.com/rust-lang/rust/labels?q=WG- +[`PG-*`]: https://github.com/rust-lang/rust/labels?q=PG- +[`C-*`]: https://github.com/rust-lang/rust/labels?q=C- +[`O-*`]: https://github.com/rust-lang/rust/labels?q=O- +[`A-*`]: https://github.com/rust-lang/rust/labels?q=A- +[`L-*`]: https://github.com/rust-lang/rust/labels?q=L- +[`F-*`]: https://github.com/rust-lang/rust/labels?q=F- +[`-Z*`]: https://github.com/rust-lang/rust/labels?q=-Z +[`regression-*`]: https://github.com/rust-lang/rust/labels?q=regression- +[`D-*`]: https://github.com/rust-lang/rust/labels?q=D- +[`I-*`]: https://github.com/rust-lang/rust/labels?q=I- +[`P-*`]: https://github.com/rust-lang/rust/labels?q=P- +[`S-*`]: https://github.com/rust-lang/rust/labels?q=S- +[`E-*`]: https://github.com/rust-lang/rust/labels?q=E- +[^1]: The `O` in `O-*` labels originally stood for *operating system (OS)*. +[^2]: The `I` in `I-*` labels originally stood for *importance*. This makes the most sense for the `I-*-nominated` labels. For most `I-*` labels however it makes sense to interpret the `I` as *issue (kind)*. +[^3]: The `E` in `E-*` labels stands for *experience*. From 27b3d33cd51745ad0c5331a4b1dddf9ba975918b Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Thu, 16 May 2024 10:14:01 -0700 Subject: [PATCH 03/38] Remove triagebot manual note. --- src/triagebot/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/triagebot/README.md b/src/triagebot/README.md index 1e0d285f4..96d801b37 100644 --- a/src/triagebot/README.md +++ b/src/triagebot/README.md @@ -34,8 +34,6 @@ For example, the `rust-lang/rust` configuration file is at Date: Sat, 4 May 2024 17:22:56 -0700 Subject: [PATCH 04/38] Add Archive of Rust Stable Standalone Installers --- blacksmith/src/lib.rs | 109 +++++++++++++++++- src/SUMMARY.md | 1 + .../archive-stable-version-installers.md | 9 ++ src/infra/other-installation-methods.md | 27 +---- src/infra/shared-standalone-installers.md | 20 ++++ 5 files changed, 141 insertions(+), 25 deletions(-) create mode 100644 src/infra/archive-stable-version-installers.md create mode 100644 src/infra/shared-standalone-installers.md diff --git a/blacksmith/src/lib.rs b/blacksmith/src/lib.rs index 887926e36..5c8bd6968 100644 --- a/blacksmith/src/lib.rs +++ b/blacksmith/src/lib.rs @@ -15,6 +15,7 @@ use mdbook::{ const CHANNELS: &[&str] = &["stable", "beta", "nightly"]; const CHANNEL_URL_PREFIX: &str = "https://static.rust-lang.org/dist/channel-rust-"; +const MANIFESTS_URL: &str = "https://static.rust-lang.org/manifests.txt"; const RUSTUP_URLS: &str = "https://raw.githubusercontent.com/rust-lang/rustup.rs/stable/ci/cloudfront-invalidation.txt"; @@ -45,6 +46,8 @@ pub struct Blacksmith { rustup: Vec, stable_version: Option, platforms: BTreeMap, + #[serde(default)] + previous_stable_versions: Vec<(String, Vec)>, } impl Blacksmith { @@ -123,6 +126,64 @@ impl Blacksmith { } } + let latest_stable_version = &blacksmith.stable_version.clone().unwrap(); + + // Go over stable versions in https://static.rust-lang.org/manifests.txt in reverse order. + let manifests_content = reqwest::blocking::get(MANIFESTS_URL)?.text()?; + let stable_manifest_url_regex = + regex::Regex::new(r"^static\.rust-lang\.org/dist/\d{4}-\d{2}-\d{2}/channel-rust-1\.(\d+)\.(\d+)\.toml$").unwrap(); + for manifest_url in manifests_content.lines().rev() { + let minor; + let patch; + + // Check if it's a stable version. + if let Some(captures) = stable_manifest_url_regex.captures(&(manifest_url)) { + minor = captures.get(1).unwrap().as_str(); + patch = captures.get(2).unwrap().as_str(); + } else { + continue + } + + let full_version = format!("1.{}.{}", minor, patch); + + // Skip latest stable version. + if &full_version == latest_stable_version { + continue + } + + // Download https://static.rust-lang.org/dist/channel-rust-{major.minor.patch}.toml and process it. + let channel_url = format!("{}{}.toml", CHANNEL_URL_PREFIX, full_version); + + let content = reqwest::blocking::get(&channel_url)?.text()?; + let rust = toml::from_str::(&content)? + .pkg + .rust; + + log::info!( + "Found {} targets for stable v{}", + rust.target.len(), + rust.version + ); + + let version = rust.version.split(' ').next().unwrap().to_string(); + + let platforms = rust + .target + .into_iter() + .filter_map(|(target, content)| { + if content.available { + Some(target) + } else { + None + } + }) + .collect::>(); + + blacksmith + .previous_stable_versions + .push((version.clone(), platforms)); + } + blacksmith.last_update = unix_time(); Ok(blacksmith) } @@ -209,6 +270,45 @@ impl Blacksmith { buffer } + /// Generates tables of links to the previous stable standalone installer packages for + /// each platform. + fn generate_previous_stable_standalone_installers_tables(&self) -> String { + let mut buffer = String::new(); + + for (stable_version, platforms) in &self.previous_stable_versions { + writeln!(buffer, "## Stable ({})", stable_version).unwrap(); + writeln!(buffer, "").unwrap(); + + writeln!(buffer, "platform | stable ({})", stable_version).unwrap(); + writeln!(buffer, "---------|--------").unwrap(); + + for name in platforms { + let extension = if name.contains("windows") { + "msi" + } else if name.contains("darwin") { + "pkg" + } else { + "tar.gz" + }; + + let stable_links = + generate_standalone_links("rust", stable_version, name, extension); + + writeln!( + buffer, + "`{name}` | {stable}", + name = name, + stable = stable_links, + ) + .unwrap(); + } + + writeln!(buffer, "").unwrap(); + } + + buffer + } + /// Generates a similar table to `generate_standalone_installers_table` /// except for the rust source code packages. fn generate_source_code_table(&self) -> String { @@ -288,16 +388,23 @@ impl Preprocessor for Blacksmith { let rustup_init_list = self.generate_rustup_init_list(); let standalone_installers_table = self.generate_standalone_installers_table(); + let previous_stable_standalone_installers_tables = + self.generate_previous_stable_standalone_installers_tables(); let source_code_table = self.generate_source_code_table(); // TODO: Currently we're performing a global search for any of the // variables as that's the most flexible for adding more dynamic - // content, and the time to traverse is fast enough to not be noticable. + // content, and the time to traverse is fast enough to not be noticeable. // However if the processing time begins to become a bottleneck this // should change. for item in &mut book.sections { recursive_replace(item, "{{#rustup_init_list}}", &rustup_init_list); recursive_replace(item, "{{#installer_table}}", &standalone_installers_table); + recursive_replace( + item, + "{{#previous_stable_standalone_installers_tables}}", + &previous_stable_standalone_installers_tables, + ); recursive_replace(item, "{{#source_code_table}}", &source_code_table); } diff --git a/src/SUMMARY.md b/src/SUMMARY.md index cab30a065..87d284deb 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -68,6 +68,7 @@ - [Moderation](./governance/moderation.md) - [Infrastructure](./infra/README.md) - [Other Installation Methods](./infra/other-installation-methods.md) + - [Archive of Rust Stable Standalone Installers](./infra/archive-stable-version-installers.md) - [Release Channel Layout](./infra/channel-layout.md) - [Service Infrastructure](./infra/service-infrastructure.md) - [Team Maintenance](./infra/team-maintenance.md) diff --git a/src/infra/archive-stable-version-installers.md b/src/infra/archive-stable-version-installers.md new file mode 100644 index 000000000..2dcf349cf --- /dev/null +++ b/src/infra/archive-stable-version-installers.md @@ -0,0 +1,9 @@ +# Archive of Rust Stable Standalone Installers + +**Note: The Rust project only supports the latest stable release with security patches. +Generally speaking these archives should not be used without some extra mechanisms +to provide for patching.** + +{{#include shared-standalone-installers.md}} + +{{#previous_stable_standalone_installers_tables}} diff --git a/src/infra/other-installation-methods.md b/src/infra/other-installation-methods.md index c43fd8cb1..1d661b554 100644 --- a/src/infra/other-installation-methods.md +++ b/src/infra/other-installation-methods.md @@ -79,26 +79,9 @@ what they each specifically generate. -The official Rust standalone installers contain a single release of Rust, and -are suitable for offline installation. They come in three forms: tarballs -(extension `.tar.xz`), that work in any Unix-like environment, Windows -installers (`.msi`), and Mac installers (`.pkg`). These installers come with -`rustc`, `cargo`, `rustdoc`, the standard library, and the standard -documentation, but do not provide access to additional cross-targets like -`rustup` does. +{{#include shared-standalone-installers.md}} -The most common reasons to use these are: - -- Offline installation -- Preferring a more platform-integrated, graphical installer on Windows - -Each of these binaries is signed with the [Rust signing key], which is -[available on keybase.io], by the Rust build infrastructure, with [GPG]. In the -tables below, the `.asc` files are the signatures. - - +Past releases can be found in [the archive]. {{#installer_table}} @@ -170,8 +153,4 @@ diff rustc-*-src.tar build/dist/rustc-*-src.tar [chocolatey]: http://chocolatey.org/ [scoop]: https://scoop.sh/ [three tiers]: https://doc.rust-lang.org/nightly/rustc/platform-support.html -[rust signing key]: https://static.rust-lang.org/rust-key.gpg.ascii -[gpg]: https://gnupg.org/ -[available on keybase.io]: https://keybase.io/rust -[the archives]: https://static.rust-lang.org/dist/index.html - +[the archive]: ../infra/archive-stable-version-installers.md diff --git a/src/infra/shared-standalone-installers.md b/src/infra/shared-standalone-installers.md new file mode 100644 index 000000000..0824b88ce --- /dev/null +++ b/src/infra/shared-standalone-installers.md @@ -0,0 +1,20 @@ +The official Rust standalone installers contain a single release of Rust, and +are suitable for offline installation. They come in three forms: tarballs +(extension `.tar.xz`), that work in any Unix-like environment, Windows +installers (`.msi`), and Mac installers (`.pkg`). These installers come with +`rustc`, `cargo`, `rustdoc`, the standard library, and the standard +documentation, but do not provide access to additional cross-targets like +`rustup` does. + +The most common reasons to use these are: + +- Offline installation +- Preferring a more platform-integrated, graphical installer on Windows + +Each of these binaries is signed with the [Rust signing key], which is +[available on keybase.io], by the Rust build infrastructure, with [GPG]. In the +tables below, the `.asc` files are the signatures. + +[rust signing key]: https://static.rust-lang.org/rust-key.gpg.ascii +[available on keybase.io]: https://keybase.io/rust +[gpg]: https://gnupg.org/ From 212ffd54519719d09ae4d2c37472ace12b41ca35 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20Ber=C3=A1nek?= Date: Mon, 6 May 2024 21:58:29 +0200 Subject: [PATCH 05/38] Use a safer command for rebasing --- src/triagebot/no-merge.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/triagebot/no-merge.md b/src/triagebot/no-merge.md index 1d4883a50..a544af269 100644 --- a/src/triagebot/no-merge.md +++ b/src/triagebot/no-merge.md @@ -39,8 +39,7 @@ There are three optional values that can be specified in the table: > > ```shell-session > $ # rebase -> $ git rebase -i master -> $ # delete any merge commits in the editor that appears +> $ git pull --rebase https://github.com/rust-lang/rust.git master > $ git push --force-with-lease > ``` From dadc4972891ed64828b54d920647f9723d9bec50 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Mon, 10 Jun 2024 15:35:15 -0700 Subject: [PATCH 06/38] Update GitHub access policy docs. --- src/platforms/github.md | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/src/platforms/github.md b/src/platforms/github.md index 2036c3882..3e1444294 100644 --- a/src/platforms/github.md +++ b/src/platforms/github.md @@ -19,15 +19,22 @@ of its discussions. [`rust-secure-code`]: https://github.com/rust-secure-code [`rust-gamedev`]: https://github.com/rust-gamedev -## Administration FAQ +## rust-lang organization policy -> Who administrates the `rust-lang` organisation? +The following is the policy for management of the [`rust-lang`] organization. -The infrastructure team is responsible for administrating the GitHub organization. +### Access -> How do I create a new repository under the `rust-lang` organisation, or make - changes that require `admin` level permissions? +All access to the `rust-lang` GitHub organization is managed via the [`team` repository][team][^team-policy]. Teams that want to assign access levels, or create new repositories should open a Pull Request to that repository to request the change. -New repositories may be created by submitting a PR to add the repository in the [Rust team database][teams]. +The [Infrastructure Team] is responsible for overall administration of the `rust-lang` GitHub organization. Selected members of the Infrastructure Team may be organization owners if their work requires it. -[teams]: https://github.com/rust-lang/team +All GitHub accounts used to interact with the Rust-Lang GitHub organization (owner or non-owner) must have [2FA] enabled. This is enforced by GitHub. + +Bot accounts controlled by the Infrastructure Team (such as the triagebot) can be granted any level of access required for them to work at the discretion of the Infrastructure Team. + +[^team-policy]: See [Team Maintenance](../infra/team-maintenance.md) for policy on how the team repo is managed. + +[Infrastructure Team]: https://github.com/rust-lang/infra-team/ +[team]: https://github.com/rust-lang/team +[2FA]: https://docs.github.com/en/authentication/securing-your-account-with-two-factor-authentication-2fa From d12ceb96e9d66faa6c81ed533aa55cef73a874f4 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Wed, 12 Jun 2024 15:56:16 -0700 Subject: [PATCH 07/38] Include mdbook in list of tools that need edition support. --- src/editions/edition-releases.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/editions/edition-releases.md b/src/editions/edition-releases.md index e74772791..cb5ead766 100644 --- a/src/editions/edition-releases.md +++ b/src/editions/edition-releases.md @@ -44,6 +44,7 @@ Running an Edition involves many steps, coordinated across the project. - [`rustfmt` support for `--edition` flag](https://github.com/rust-lang/rust/pull/94461) - [`clippy` support for `--edition` flag](https://github.com/rust-lang/rust/pull/94461) - [`rust-analyzer` support for editions](https://github.com/rust-lang/rust-analyzer/pull/7123) + - [`mdbook` support for editions](https://github.com/rust-lang/mdBook/pull/2398) 1. Teams start their proposals and implementation work. 1. The Leadership Council sets up a Project Group to run the edition (approximately 1 year before the final release). 1. Final deadline phase. From aa5a929e92a8d11aed4dba291b8e3ea0a98192ca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20Ber=C3=A1nek?= Date: Thu, 13 Jun 2024 15:37:18 +0200 Subject: [PATCH 08/38] Add a link to CI documentation in rustc-dev-guide --- src/infra/docs/rustc-ci.md | 276 +------------------------------------ 1 file changed, 4 insertions(+), 272 deletions(-) diff --git a/src/infra/docs/rustc-ci.md b/src/infra/docs/rustc-ci.md index 08c0da8df..021a4b8e5 100644 --- a/src/infra/docs/rustc-ci.md +++ b/src/infra/docs/rustc-ci.md @@ -1,276 +1,8 @@ # How the Rust CI works -Rust CI ensures that the master branch of rust-lang/rust is always in a valid state. +Continuous integration (CI) workflows on the `rust-lang/rust` repository ensure that the `master` branch +is always in a valid state. -A developer submitting a pull request to rust-lang/rust, experiences the following: +The CI infrastructure is described in detail in the [rustc-dev-guide][rustc-dev-guide]. -- A small subset of tests and checks are run on each commit to catch common errors. -- When the PR is ready and approved, the "bors" tool enqueues a full CI run. -- The full run either queues the specific PR or the PR is "rolled up" with other changes. -- Eventually a CI run containing the changes from the PR is performed and either passes or fails with an error the developer must address. - -## Which jobs we run - -The `rust-lang/rust` repository uses GitHub Actions to test [all the -platforms][platforms] we support. We currently have two kinds of jobs running -for each commit we want to merge to master: - -- Dist jobs build a full release of the compiler for that platform, including - all the tools we ship through rustup; Those builds are then uploaded to the - `rust-lang-ci2` S3 bucket and are available to be locally installed with the - [rustup-toolchain-install-master] tool; The same builds are also used for - actual releases: our release process basically consists of copying those - artifacts from `rust-lang-ci2` to the production endpoint and signing them. -- Non-dist jobs run our full test suite on the platform, and the test suite of - all the tools we ship through rustup; The amount of stuff we test depends on - the platform (for example some tests are run only on Tier 1 platforms), and - some quicker platforms are grouped together on the same builder to avoid - wasting CI resources. - -All the builds except those on macOS and Windows are executed inside that -platform’s custom [Docker container]. This has a lot of advantages for us: - -- The build environment is consistent regardless of the changes of the - underlying image (switching from the trusty image to xenial was painless for - us). -- We can use ancient build environments to ensure maximum binary compatibility, - for example [using older CentOS releases][dist-x86_64-linux] on our Linux builders. -- We can avoid reinstalling tools (like QEMU or the Android emulator) every - time thanks to Docker image caching. -- Users can run the same tests in the same environment locally by just running - `src/ci/docker/run.sh image-name`, which is awesome to debug failures. - -The docker images prefixed with `dist-` are used for building artifacts while those without that prefix run tests and checks. - -We also run tests for less common architectures (mainly Tier 2 and Tier 3 -platforms) in CI. Since those platforms are not x86 we either run -everything inside QEMU or just cross-compile if we don’t want to run the tests -for that platform. - -These builders are running on a special pool of builders set up and maintained for us by GitHub. - -Almost all build steps shell out to separate scripts. This keeps the CI fairly platform independent (i.e., we are not -overly reliant on GitHub Actions). GitHub Actions is only relied on for bootstrapping the CI process and for orchestrating -the scripts that drive the process. - -[platforms]: https://doc.rust-lang.org/nightly/rustc/platform-support.html -[rustup-toolchain-install-master]: https://github.com/kennytm/rustup-toolchain-install-master -[Docker container]: https://github.com/rust-lang/rust/tree/master/src/ci/docker -[dist-x86_64-linux]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile - -## Merging PRs serially with bors - -CI services usually test the last commit of a branch merged with the last -commit in master, and while that’s great to check if the feature works in -isolation it doesn’t provide any guarantee the code is going to work once it’s -merged. Breakages like these usually happen when another, incompatible PR is -merged after the build happened. - -To ensure a master that works all the time we forbid manual merges: instead all -PRs have to be approved through our bot, [bors] (the software behind it is -called [homu]). All the approved PRs are put [in a queue][homu-queue] (sorted -by priority and creation date) and are automatically tested one at the time. If -all the builders are green the PR is merged, otherwise the failure is recorded -and the PR will have to be re-approved again. - -Bors doesn’t interact with CI services directly, but it works by pushing the -merge commit it wants to test to a branch called `auto`, and detecting the -outcome of the build by listening for either Commit Statuses or Check Runs. -Since the merge commit is based on the latest master and only one can be tested -at the same time, when the results are green master is fast-forwarded to that -merge commit. - -The `auto` branch and other branches used by bors live on a fork of rust-lang/rust: -[rust-lang-ci/rust]. This was originally done due to some security limitations in GitHub -Actions. These limitations have been addressed, but we've not yet done the work of removing -the use of the fork. - -Unfortunately testing a single PR at the time, combined with our long CI (~3 -hours for a full run)[^1], means we can’t merge too many PRs in a single day, and a -single failure greatly impacts our throughput for the day. The maximum number -of PRs we can merge in a day is around 8. - -The large CI run times and requirement for a large builder pool is largely due to the -fact that full release artifacts are built in the `dist-` builders. This is worth it -because these release artifacts: - -- allow perf testing even at a later date -- allow bisection when bugs are discovered later -- ensure release quality since if we're always releasing, we can catch problems early - -Bors [runs on ecs](https://github.com/rust-lang/simpleinfra/blob/master/terraform/bors/app.tf) and uses a sqlite database running in a volume as storage. - -[^1]: As of January 2023, the bottleneck are the `dist-x86_64-linux` and `dist-x86_64-linux-alt` runners because of their usage of [BOLT] and [PGO] optimization tooling. - -[bors]: https://github.com/bors -[homu]: https://github.com/rust-lang/homu -[homu-queue]: https://bors.rust-lang.org/queue/rust -[rust-lang-ci/rust]: https://github.com/rust-lang-ci/rust -[BOLT]: https://github.com/facebookincubator/BOLT -[PGO]: https://en.wikipedia.org/wiki/Profile-guided_optimization - -### Rollups - -Some PRs don’t need the full test suite to be executed: trivial changes like -typo fixes or README improvements *shouldn’t* break the build, and testing -every single one of them for 2 to 3 hours is a big waste of time. To solve this -we do a "rollup", a PR where we merge all the trivial PRs so they can be tested -together. Rollups are created manually by a team member using the "create a rollup" button on the [bors queue]. The team member uses their -judgment to decide if a PR is risky or not, and are the best tool we have at -the moment to keep the queue in a manageable state. - -[bors queue]: https://bors.rust-lang.org/queue/rust - -### Try builds - -Sometimes we need a working compiler build before approving a PR, usually for -[benchmarking][perf] or [checking the impact of the PR across the -ecosystem][crater]. Bors supports creating them by pushing the merge commit on -a separate branch (`try`), and they basically work the same as normal builds, -without the actual merge at the end. Any number of try builds can happen at the -same time, even if there is a normal PR in progress. - -You can see the CI configuration for try builds [here](https://github.com/rust-lang/rust/blob/9d46c7a3e69966782e163877151c1f0cea8b630a/src/ci/github-actions/ci.yml#L728-L741). - -If you want to perform a try build with a different configuration (e.g. try to -perform a compiler build for a different architecture), you can temporarily change -the `try` CI job in your PR: - -1) Open `src/ci/github-actions/ci.yml` -2) Find the CI job that you want to run (e.g. `dist-aarch64-linux`) -3) Copy-paste the entry of the CI job -4) Find the `try:` job in the file -5) Replace the `dist-x86_64-linux` job in the matrix with the copied entry from step 3) -6) Run `python3 x.py run src/tools/expand-yaml-anchors` -7) Push your changes and start a try build with `@bors try` - -[perf]: https://perf.rust-lang.org -[crater]: https://github.com/rust-lang/crater - -## Which branches we test - -Our builders are defined in [`src/ci/github-actions/ci.yml`]. - -[`src/ci/github-actions/ci.yml`]: https://github.com/rust-lang/rust/blob/master/src/ci/github-actions/ci.yml - -### PR builds - -All the commits pushed in a PR run a limited set of tests: a job containing a -bunch of lints plus a cross-compile check build to Windows mingw (without -producing any artifacts) and the `x86_64-gnu-llvm-##` non-dist builder (where -`##` is the *system* LLVM version we are currently testing). Those two -builders are enough to catch most of the common errors introduced in a PR, but -they don’t cover other platforms at all. Unfortunately it would take too many -resources to run the full test suite for each commit on every PR. - -Additionally, if the PR changes certain tools (or certain platform-specific -parts of std to check for miri breakage), the `x86_64-gnu-tools` non-dist -builder is run. - -### The `try` branch - -On the main rust repo, `try` builds produce just a Linux toolchain using the -`dist-x86_64-linux` image. - -### The `auto` branch - -This branch is used by bors to run all the tests on a PR before merging it, so -all the builders are enabled for it. bors will repeatedly force-push on it -(every time a new commit is tested). - -### The `master` branch - -Since all the commits to `master` are fast-forwarded from the `auto` branch (if -they pass all the tests there) we don’t need to build or test anything. A quick -job is executed on each push to update toolstate (see the toolstate description -below). - -### Other branches - -Other branches are just disabled and don’t run any kind of builds, since all -the in-progress branches will eventually be tested in a PR. - -## Caching - -The main rust repository doesn’t use the native GitHub Actions caching tools. -All our caching is uploaded to an S3 bucket we control -(`rust-lang-ci-sccache2`), and it’s used mainly for two things: - -### Docker images caching - -The Docker images we use to run most of the Linux-based builders take a *long* -time to fully build. To speed up the build, we cache the exported images on the -S3 bucket (with `docker save`/`docker load`). - -Since we test multiple, diverged branches (`master`, `beta` and `stable`) we -can’t rely on a single cache for the images, otherwise builds on a branch would -override the cache for the others. Instead we store the images identifying them -with a custom hash, made from the host’s Docker version and the contents of all -the Dockerfiles and related scripts. - -### LLVM caching with sccache - -We build some C/C++ stuff during the build and we rely on [sccache] to cache -intermediate LLVM artifacts. Sccache is a distributed ccache developed by -Mozilla, and it can use an object storage bucket as the storage backend, like -we do with our S3 bucket. - -[sccache]: https://github.com/mozilla/sccache - -## Custom tooling around CI - -During the years we developed some custom tooling to improve our CI experience. - -### Rust Log Analyzer to show the error message in PRs - -The build logs for `rust-lang/rust` are huge, and it’s not practical to find -what caused the build to fail by looking at the logs. To improve the -developers’ experience we developed a bot called [Rust Log Analyzer][rla] (RLA) -that receives the build logs on failure and extracts the error message -automatically, posting it on the PR. - -The bot is not hardcoded to look for error strings, but was trained with a -bunch of build failures to recognize which lines are common between builds and -which are not. While the generated snippets can be weird sometimes, the bot is -pretty good at identifying the relevant lines even if it’s an error we've never -seen before. - -[rla]: https://github.com/rust-lang/rust-log-analyzer - -### Toolstate to support allowed failures - -The `rust-lang/rust` repo doesn’t only test the compiler on its CI, but also a -variety of tools and documentation. Some documentation is pulled in via git -submodules. If we blocked merging rustc PRs on the documentation being fixed, -we would be stuck in a chicken-and-egg problem, because the documentation's CI -would not pass since updating it would need the not-yet-merged version of -rustc to test against (and we usually require CI to be passing). - -To avoid the problem, submodules are allowed to fail, and their status is -recorded in [rust-toolstate]. When a submodule breaks, a bot automatically -pings the maintainers so they know about the breakage, and it records the -failure on the toolstate repository. The release process will then ignore -broken tools on nightly, removing them from the shipped nightlies. - -While tool failures are allowed most of the time, they’re automatically -forbidden a week before a release: we don’t care if tools are broken on nightly -but they must work on beta and stable, so they also need to work on nightly a -few days before we promote nightly to beta. - -More information is available in the [toolstate documentation]. - -### GitHub Actions Templating - -GitHub Actions does not natively support templating which can cause configurations to be large and difficult to change. We use YAML anchors for templating and a custom tool, [`expand-yaml-anchors`], to expand [the template] into the CI configuration that [GitHub uses][ci config]. - -This templating language is fairly straightforward: - -- `&` indicates a template section -- `*` expands the indicated template in place -- `<<` merges yaml dictionaries - -[rust-toolstate]: https://rust-lang-nursery.github.io/rust-toolstate -[toolstate documentation]: ../toolstate.md -[`expand-yaml-anchors`]: https://github.com/rust-lang/rust/tree/master/src/tools/expand-yaml-anchors -[the template]: https://github.com/rust-lang/rust/blob/736c675d2ab65bcde6554e1b73340c2dbc27c85a/src/ci/github-actions/ci.yml -[ci config]: https://github.com/rust-lang/rust/blob/master/.github/workflows/ci.yml \ No newline at end of file +[rustc-dev-guide]: https://rustc-dev-guide.rust-lang.org/tests/ci.html From 474bdd5d3afe51fb64318bf71a782e4ba708ecbd Mon Sep 17 00:00:00 2001 From: James Munns Date: Fri, 21 Jun 2024 09:49:40 +0200 Subject: [PATCH 09/38] Update council.md Updates the list of top level teams on the council. Some teams have been combined, leaving a total of seven current top level teams. NB: https://www.rust-lang.org/governance/teams/leadership-council for current status. --- src/governance/council.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/governance/council.md b/src/governance/council.md index fdecf897d..688ee236e 100644 --- a/src/governance/council.md +++ b/src/governance/council.md @@ -124,14 +124,12 @@ When the Council creates a new top-level team, that team then designates a Counc The set of top-level teams is: - Compiler -- Crates.io - Dev tools - Infrastructure - Language - Launching Pad - Library - Moderation -- Release ### The Launching Pad top-level team [launching-pad]: #the-launching-pad-top-level-team From 48c0a30d9ee68ff0f01c2140f90b4fff40c238a7 Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Thu, 27 Jun 2024 12:50:39 -0700 Subject: [PATCH 10/38] Copy RFC 3119 to the Forge. The current text is taken verbatim from the RFC repo. --- src/SUMMARY.md | 2 + src/policies/README.md | 3 + src/policies/crate-ownership.md | 160 ++++++++++++++++++++++++++++++++ 3 files changed, 165 insertions(+) create mode 100644 src/policies/README.md create mode 100644 src/policies/crate-ownership.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 87d284deb..4fbdab43d 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -66,6 +66,8 @@ - [Governance](./governance/README.md) - [Leadership Council](./governance/council.md) - [Moderation](./governance/moderation.md) +- [Policies](./policies/README.md) + - [Crate ownership policy](./policies/crate-ownership.md) - [Infrastructure](./infra/README.md) - [Other Installation Methods](./infra/other-installation-methods.md) - [Archive of Rust Stable Standalone Installers](./infra/archive-stable-version-installers.md) diff --git a/src/policies/README.md b/src/policies/README.md new file mode 100644 index 000000000..0e67d42c3 --- /dev/null +++ b/src/policies/README.md @@ -0,0 +1,3 @@ +# Policies + +These chapters contain policies covering the Rust project and its members. diff --git a/src/policies/crate-ownership.md b/src/policies/crate-ownership.md new file mode 100644 index 000000000..f86bf32c3 --- /dev/null +++ b/src/policies/crate-ownership.md @@ -0,0 +1,160 @@ +# Rust crate ownership policy + +- Feature Name: none +- Start Date: 2021-05-04 +- RFC PR: [rust-lang/rfcs#3119](https://github.com/rust-lang/rfcs/pull/3119) +- Rust Issue: [rust-lang/rust#88867](https://github.com/rust-lang/rust/issues/88867) + + +# Summary +[summary]: #summary + +Have a more intentional policy around crates published by the Rust project, to be applied to existing and future crates published by us. + +# Motivation +[motivation]: #motivation + + +Currently there are around a hundred crates that are maintained under a rust-lang GitHub organization and published to crates.io. These exist for a wide range of reasons: some are published for the express purposes of being used by the wider Rust community, others are internal dependencies of rustc (or otherwise), yet others are experiments. + +Given that the stamp of an official Rust team carries a degree of weight, it is confusing for community members to have to differentiate between the two, and can lead to incorrect expectations being set. Over a prolonged period of time, this can end up in crates that were never intended to be used widely becoming key dependencies in the ecosystem. + +Furthermore, these crates are not necessarily clear on who owns them. Some are owned (in the crates.io sense) by the generic `rust-lang-owner` crates.io account, some are owned by a GitHub team (like `rust-lang/libs`), and yet others are only owned by personal accounts. It seems like we should have some consistency here. + +# Reference-Level Explanation + +Once accepted, the policy sections of this RFC should be posted on + in a "Rust-lang Crates Policy" section; this RFC will not be the canonical home of the up-to-date crates policy. + +## Categories + +We propose the following categories of published crates: + + + - **Intentional artifacts**: These are crates which are intentionally released by some team (usually libs), are actively maintained, are intended to be used by external users, and intentionally have an air of officialness. Example: [libc](https://crates.io/crates/libc) + - **Internal use**: These are crates which are used by some “internal client”, like rustc, crates.io, docs.rs, etc. Their primary purpose is not to be used by external users, though the teams that maintain them (typically the teams of their internal client) may wish for the crate to have wider adoption. The line can be blurry between these and “intentional artifacts” and ultimately depends on the goals of the team. Example: [conduit](https://crates.io/crates/conduit), [measureme](https://crates.io/crates/measureme). There are two subcategories based on whether they are intended to ever show up as a transitive dependency: + - **Transitively intentional**: These are dependencies of intentional artifact libraries, and will show up in users' dependency trees, even if they are not intended to be _directly_ used. The Rust Project still needs to handle security issues in these crates _as if_ they are "intentional artifacts". + - **Not transitively intentional**: These are dependencies of shipped binaries, CI tooling, the stdlib, or are otherwise not expected to show up in users' dependency trees. The Rust Project may need to handle security issues in these crates _internally_, but does not necessarily need to message the wider public about security issues in these crates. If a security issue in one of these crates affects a published binary (or crates.io, etc), that will still need to be handled as a bug in the binary or website. + - **Experiment**: This was an experiment by a team, intended to be picked up by users to better inform API design (or whatever), without a long-term commitment to maintainership. Example: [failure](https://crates.io/crates/failure) + - **Deprecated**: This used to be an “intentional artifact” (or experiment/internal use) but isn’t anymore. Example: [rustc-serialize](https://crates.io/crates/rustc-serialize) + - **Placeholder**: Not a functional crate, used for holding on to the name of an official tool, etc. Example: [rustup](https://crates.io/crates/rustup) + - **Expatriated**: This may have been an “intentional artifact”, and still is intended to be used by external users, but is no longer intended to be official. In such cases the crate is no longer owned/managed by the Rust project. Example: [rand](https://crates.io/crates/rand) + +## Policy + +Every crate in the organization must be owned by at least one team on crates.io. Teams should use `rust-lang/foo` teams for this. Non-expatriated crates may not have personal accounts as owners; if a crate needs additional owners that are not part of teams; the team should create a project group. Note that this does not forbid non-team (or project group) users from having maintainer access to the repository; it simply forbids them from _publishing_. + +Currently it is not possible for a crate to be owned by _only_ a team; the `rust-lang-owner` account (or a similar account to be decided by the infra team) can be used as a stopgap in such cases. We should try to phase this account out as much as possible, in order to make sure it is clear who is responsible for each crate. For crates being auto-published, a `rust-lang/publish-bots` team (or individual bot accounts) can be used to allow bot accounts to publish crates. + +Each crate in the organization, and any future crates in the organization, must decide which to which category they belong in according to the above categorization. If you're not sure what the category should be when registering a crate, or do not wish to make a decision just yet, pick "Experimental". + +Each published crate must contain a README. At a minimum, this README must mention the primary owning team. Based on their categories, crates are also required to include the following information in their READMEs and documentation roots: + +### Intentional artifact + +“Intentional artifact” crates can choose their commitments but should be clear about what they are in their messaging. If and when a team has a charter, the crate should also be mentioned in the charter as an intentional artifact. Deprecating an intentional artifact should not be taken lightly and will require an RFC. + +An example of such messaging would be text like: + +> This crate is maintained by The Rust \[team\] Team for use by the wider ecosystem. This crate is post-1.0 and follows [semver compatibility](https://doc.rust-lang.org/cargo/reference/semver.html) for its APIs. + + +Security issues in these crates should be handled with the appropriate weight and careful messaging by the Security Response WG, and should be reported [according to the project's security policy](https://www.rust-lang.org/policies/security). + +### Internal use +“Internal use” crates should contain the following text near the top of the readme/documentation: + +> This crate is maintained by \[team\], primarily for use by \[rust project(s)\] and not intended for external use (except as a transitive dependency). This crate may make major changes to its APIs or be deprecated without warning. + + +The "except as a transitive dependency" text should be included if the crate is a dependency of an intentional-artifact library ("transitively intentional"). + +Security issues in transitively intentional libraries should be handled as if they were intentional artifacts. + + +### Experiment + +“Experiment” crates should mention they are experiments. Experiment crates may be intended to be used in a scoped sort of way; so if they are intended to be used they should be clear about what they are guaranteeing. + +An example of such messaging would be text like: + +> This crate is maintained by \[team\] as a part of an experiment around \[thingy\]. We encourage people to try to use this crate in their projects and provide feedback through \[method\], but do not guarantee long term maintenance. + +or, for experiments that are not intended to be used at all: + +> This crate is maintained by \[team\] and is an internal experiment. We do not guarantee stability or long term maintenance, use at your own risk. + +Ideally, experimental crates that are published for feedback purposes will have a document to link to that lists out the purpose, rough duration, and processes of the experiment. + +### Deprecated +“Deprecated” crates should contain the following text near the top of the readme/documentation: + +> This crate is deprecated and not intended to be used. + +### Placeholder + +“Placeholder” crates should contain the following text in their published readme/documentation: + +> This crate is a functionally empty crate that exists to reserve the crate name of \[tool\]. It should not be used. + +In general it is better to have an empty placeholder crate published instead of reserving the crate via yanking, so that there is a readme that helps people understand why the crate is unavailable. + + +### Expatriated + +It's unclear if any action should be taken on these beyond removing any semblance of officialness (including rust-lang/foo team owners). We currently have only one such crate (`rand`). + +These should by and large not be considered to be "team managed" crates; this category is in this RFC for completeness to be able to talk about expatriation as an end state. + +## Transitions and new crates + +Teams should feel free to create new crates in any of these categories; however "Intentional Artifact" crates must be accompanied with an RFC. As we move towards having team charters, this can transition to being a charter change (which may require an RFC or use its own process). Teams should notify core@rust-lang.org when they've created such crates so that the core team may track these crates and ensure this policy is applied. + +From time to time a team's plan for a crate may change: experiments may conclude, crates may need to be deprecated, or the team may decide to release something for wider usage. + +In general, teams should notify core@rust-lang.org when such a transition is being made. + +Any transition _away_ from "Intentional Artifact" requires an RFC. + +Any transition to "Intentional Artifact" should ideally be accompanied by an RFC, and an update to the team charter if there is one. + +Expatriation should basically _never_ occur anymore, but it also requires an RFC and core team approval in case it is really necessary. If a team wishes to stop working on a crate, they should deprecate it and encourage the community to fork it or build their own thing. The repository may be transferred out, however the `crates.io` name is kept by the Rust project and the new group of maintainers will need to pick a new crate name. + +If "transitively intentional" crates are being deprecated care should be taken to ensure security issues will still be handled. + +Transitions between the other types can be made at will since they explicitly and clearly state their lack of a strong stability/maintenance guarantee. + + +## Applying this to existing crates + +An audit should be performed on all existing potentially "official" crates, collecting them in a list and roughly determining what their team and category should be. + +(We have a list with a preliminary audit already and plan to post it to this RFC as an example soon) + +Once we have this list, we can approach teams with lists of crates and request that they verify that the categorization is accurate. In the case of some crates this might take some time as the team may need to work out what their intentions are with a particular crate. + +Then, working with the teams, we make these changes to their documentation. We also make sure all crates have the appropriate `rust-lang/teamname` github owner, and remove personal accounts from the owners list. + +For crates that are in direct use by a lot of the wider community, if we end up categorizing them as anything other than "intentional artifact", there should be an attempt to announce this "change" to the community. While there was no formal commitment made in case of these crates, the vague sense of officialness may have made people believe there was, and we should at least try to rectify this so that people are not continually misled. Whether or not this needs to be done, and how, can be figured out by the individual teams. + +A large part of this work can be parallelized; and it does not need to occur all at once. + +# Drawbacks +[drawbacks]: #drawbacks + +This is a lot of work, but as we move towards a more deliberately structured project, it is probably necessary work. + + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +An alternative here is mostly to continue as is. This will become increasingly untenable as we add more and more crates; with the constant danger of internal crates becoming accidental artifacts that the ecosystem depends on. + +Another alternative is to ask teams to be clear about the level of support offered in their crates without standardizing the process. This could work, but could lead to less cross-team legibility and would be harder to track. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + + - How should we handle expatriated crates? + - Are there any missing categories? + - What should the text blurbs be for the various categories? Should we be mandating a specific text blurb, or just require a general idea be communicated with some leeway? From 92307f2a55da4136a9c088b08337a2c193fb7baa Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Thu, 27 Jun 2024 13:04:17 -0700 Subject: [PATCH 11/38] Remove the RFC-specific content and update mentions of "core". --- src/policies/crate-ownership.md | 57 ++++----------------------------- 1 file changed, 6 insertions(+), 51 deletions(-) diff --git a/src/policies/crate-ownership.md b/src/policies/crate-ownership.md index f86bf32c3..8a6885db0 100644 --- a/src/policies/crate-ownership.md +++ b/src/policies/crate-ownership.md @@ -1,35 +1,12 @@ # Rust crate ownership policy -- Feature Name: none -- Start Date: 2021-05-04 -- RFC PR: [rust-lang/rfcs#3119](https://github.com/rust-lang/rfcs/pull/3119) -- Rust Issue: [rust-lang/rust#88867](https://github.com/rust-lang/rust/issues/88867) +## Introduction - -# Summary -[summary]: #summary - -Have a more intentional policy around crates published by the Rust project, to be applied to existing and future crates published by us. - -# Motivation -[motivation]: #motivation - - -Currently there are around a hundred crates that are maintained under a rust-lang GitHub organization and published to crates.io. These exist for a wide range of reasons: some are published for the express purposes of being used by the wider Rust community, others are internal dependencies of rustc (or otherwise), yet others are experiments. - -Given that the stamp of an official Rust team carries a degree of weight, it is confusing for community members to have to differentiate between the two, and can lead to incorrect expectations being set. Over a prolonged period of time, this can end up in crates that were never intended to be used widely becoming key dependencies in the ecosystem. - -Furthermore, these crates are not necessarily clear on who owns them. Some are owned (in the crates.io sense) by the generic `rust-lang-owner` crates.io account, some are owned by a GitHub team (like `rust-lang/libs`), and yet others are only owned by personal accounts. It seems like we should have some consistency here. - -# Reference-Level Explanation - -Once accepted, the policy sections of this RFC should be posted on - in a "Rust-lang Crates Policy" section; this RFC will not be the canonical home of the up-to-date crates policy. +This document covers the policy for crates published by the Rust project. This was initially adopted via [RFC 3119](https://github.com/rust-lang/rfcs/pull/3119). ## Categories -We propose the following categories of published crates: - +Rust crates published by the Rust project fall into one of the following categories: - **Intentional artifacts**: These are crates which are intentionally released by some team (usually libs), are actively maintained, are intended to be used by external users, and intentionally have an air of officialness. Example: [libc](https://crates.io/crates/libc) - **Internal use**: These are crates which are used by some “internal client”, like rustc, crates.io, docs.rs, etc. Their primary purpose is not to be used by external users, though the teams that maintain them (typically the teams of their internal client) may wish for the crate to have wider adoption. The line can be blurry between these and “intentional artifacts” and ultimately depends on the goals of the team. Example: [conduit](https://crates.io/crates/conduit), [measureme](https://crates.io/crates/measureme). There are two subcategories based on whether they are intended to ever show up as a transitive dependency: @@ -108,17 +85,17 @@ These should by and large not be considered to be "team managed" crates; this ca ## Transitions and new crates -Teams should feel free to create new crates in any of these categories; however "Intentional Artifact" crates must be accompanied with an RFC. As we move towards having team charters, this can transition to being a charter change (which may require an RFC or use its own process). Teams should notify core@rust-lang.org when they've created such crates so that the core team may track these crates and ensure this policy is applied. +Teams should feel free to create new crates in any of these categories; however "Intentional Artifact" crates must be accompanied with an RFC. As we move towards having team charters, this can transition to being a charter change (which may require an RFC or use its own process). Teams should notify when they've created such crates so that the Leadership Council may track these crates and ensure this policy is applied. From time to time a team's plan for a crate may change: experiments may conclude, crates may need to be deprecated, or the team may decide to release something for wider usage. -In general, teams should notify core@rust-lang.org when such a transition is being made. +In general, teams should notify when such a transition is being made. Any transition _away_ from "Intentional Artifact" requires an RFC. Any transition to "Intentional Artifact" should ideally be accompanied by an RFC, and an update to the team charter if there is one. -Expatriation should basically _never_ occur anymore, but it also requires an RFC and core team approval in case it is really necessary. If a team wishes to stop working on a crate, they should deprecate it and encourage the community to fork it or build their own thing. The repository may be transferred out, however the `crates.io` name is kept by the Rust project and the new group of maintainers will need to pick a new crate name. +Expatriation should basically _never_ occur anymore, but it also requires an RFC and Leadership Council approval in case it is really necessary. If a team wishes to stop working on a crate, they should deprecate it and encourage the community to fork it or build their own thing. The repository may be transferred out, however the `crates.io` name is kept by the Rust project and the new group of maintainers will need to pick a new crate name. If "transitively intentional" crates are being deprecated care should be taken to ensure security issues will still be handled. @@ -129,8 +106,6 @@ Transitions between the other types can be made at will since they explicitly an An audit should be performed on all existing potentially "official" crates, collecting them in a list and roughly determining what their team and category should be. -(We have a list with a preliminary audit already and plan to post it to this RFC as an example soon) - Once we have this list, we can approach teams with lists of crates and request that they verify that the categorization is accurate. In the case of some crates this might take some time as the team may need to work out what their intentions are with a particular crate. Then, working with the teams, we make these changes to their documentation. We also make sure all crates have the appropriate `rust-lang/teamname` github owner, and remove personal accounts from the owners list. @@ -138,23 +113,3 @@ Then, working with the teams, we make these changes to their documentation. We a For crates that are in direct use by a lot of the wider community, if we end up categorizing them as anything other than "intentional artifact", there should be an attempt to announce this "change" to the community. While there was no formal commitment made in case of these crates, the vague sense of officialness may have made people believe there was, and we should at least try to rectify this so that people are not continually misled. Whether or not this needs to be done, and how, can be figured out by the individual teams. A large part of this work can be parallelized; and it does not need to occur all at once. - -# Drawbacks -[drawbacks]: #drawbacks - -This is a lot of work, but as we move towards a more deliberately structured project, it is probably necessary work. - - -# Rationale and alternatives -[rationale-and-alternatives]: #rationale-and-alternatives - -An alternative here is mostly to continue as is. This will become increasingly untenable as we add more and more crates; with the constant danger of internal crates becoming accidental artifacts that the ecosystem depends on. - -Another alternative is to ask teams to be clear about the level of support offered in their crates without standardizing the process. This could work, but could lead to less cross-team legibility and would be harder to track. - -# Unresolved questions -[unresolved-questions]: #unresolved-questions - - - How should we handle expatriated crates? - - Are there any missing categories? - - What should the text blurbs be for the various categories? Should we be mandating a specific text blurb, or just require a general idea be communicated with some leeway? From e6c795c3126cd754688bb77b42d0877b03e57cc2 Mon Sep 17 00:00:00 2001 From: Chris Denton Date: Sat, 29 Jun 2024 05:27:49 +0000 Subject: [PATCH 12/38] Document archives for platforms with installers --- blacksmith/src/lib.rs | 63 ++++++++++++++++++++++++------------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/blacksmith/src/lib.rs b/blacksmith/src/lib.rs index 5c8bd6968..55335df68 100644 --- a/blacksmith/src/lib.rs +++ b/blacksmith/src/lib.rs @@ -230,28 +230,28 @@ impl Blacksmith { writeln!(buffer, "---------|--------|------|--------").unwrap(); for (name, platform) in &self.platforms { - let extension = if name.contains("windows") { - "msi" + let extensions: &[&str] = if name.contains("windows") { + &["msi", "tar.xz"] } else if name.contains("darwin") { - "pkg" + &["pkg", "tar.xz"] } else { - "tar.xz" + &["tar.xz"] }; let stable_links = platform .stable .as_ref() - .map(|version| generate_standalone_links("rust", version, name, extension)) + .map(|version| generate_standalone_links("rust", version, name, extensions)) .unwrap_or_else(String::new); let beta_links = if platform.beta { - generate_standalone_links("rust", "beta", name, extension) + generate_standalone_links("rust", "beta", name, extensions) } else { String::new() }; let nightly_links = if platform.nightly { - generate_standalone_links("rust", "nightly", name, extension) + generate_standalone_links("rust", "nightly", name, extensions) } else { String::new() }; @@ -283,16 +283,16 @@ impl Blacksmith { writeln!(buffer, "---------|--------").unwrap(); for name in platforms { - let extension = if name.contains("windows") { - "msi" + let extensions: &[&str] = if name.contains("windows") { + &["msi", "tar.gz"] } else if name.contains("darwin") { - "pkg" + &["pkg", "tar.gz"] } else { - "tar.gz" + &["tar.gz"] }; let stable_links = - generate_standalone_links("rust", stable_version, name, extension); + generate_standalone_links("rust", stable_version, name, extensions); writeln!( buffer, @@ -324,7 +324,7 @@ impl Blacksmith { buffer, "stable ({}) | {}", stable_version, - generate_standalone_links("rustc", stable_version, "src", "tar.xz") + generate_standalone_links("rustc", stable_version, "src", &["tar.xz"]) ) .unwrap(); } else { @@ -332,7 +332,7 @@ impl Blacksmith { buffer, "{} | {}", channel, - generate_standalone_links("rustc", &channel, "src", "tar.xz") + generate_standalone_links("rustc", &channel, "src", &["tar.xz"]) ) .unwrap(); } @@ -344,20 +344,27 @@ impl Blacksmith { /// Generates links to standalone installers provided a rust version or channel, /// target name, and file extension. -fn generate_standalone_links(base: &str, stem: &str, name: &str, extension: &str) -> String { - let url = format!( - "https://static.rust-lang.org/dist/{base}-{stem}-{name}.{extension}", - extension = extension, - name = name, - stem = stem, - base = base, - ); - - format!( - "[{extension}]({url})
[{extension}.asc]({url}.asc)", - extension = extension, - url = url, - ) +fn generate_standalone_links(base: &str, stem: &str, name: &str, extensions: &[&str]) -> String { + let mut result = String::new(); + for extension in extensions { + if !result.is_empty() { + result.push_str("
"); + } + let url = format!( + "https://static.rust-lang.org/dist/{base}-{stem}-{name}.{extension}", + extension = extension, + name = name, + stem = stem, + base = base, + ); + + result.push_str(&format!( + "[{extension}]({url})
[{extension}.asc]({url}.asc)", + extension = extension, + url = url, + )); + } + result } fn unix_time() -> Option { From 77ea390be7d83f15a2dbac8a6c78d42cc484d22c Mon Sep 17 00:00:00 2001 From: Chris Denton Date: Sat, 29 Jun 2024 05:58:05 +0000 Subject: [PATCH 13/38] Format some files --- blacksmith/src/channel.rs | 3 +-- blacksmith/src/lib.rs | 10 ++++++---- book.toml | 4 ++-- 3 files changed, 9 insertions(+), 8 deletions(-) diff --git a/blacksmith/src/channel.rs b/blacksmith/src/channel.rs index 039ed67de..5c201cac6 100644 --- a/blacksmith/src/channel.rs +++ b/blacksmith/src/channel.rs @@ -1,5 +1,5 @@ -use std::collections::BTreeMap; use serde::Deserialize; +use std::collections::BTreeMap; #[derive(Deserialize)] pub struct Target { @@ -21,4 +21,3 @@ pub struct Packages { pub struct Channel { pub pkg: Packages, } - diff --git a/blacksmith/src/lib.rs b/blacksmith/src/lib.rs index 5c8bd6968..7c960e26a 100644 --- a/blacksmith/src/lib.rs +++ b/blacksmith/src/lib.rs @@ -130,8 +130,10 @@ impl Blacksmith { // Go over stable versions in https://static.rust-lang.org/manifests.txt in reverse order. let manifests_content = reqwest::blocking::get(MANIFESTS_URL)?.text()?; - let stable_manifest_url_regex = - regex::Regex::new(r"^static\.rust-lang\.org/dist/\d{4}-\d{2}-\d{2}/channel-rust-1\.(\d+)\.(\d+)\.toml$").unwrap(); + let stable_manifest_url_regex = regex::Regex::new( + r"^static\.rust-lang\.org/dist/\d{4}-\d{2}-\d{2}/channel-rust-1\.(\d+)\.(\d+)\.toml$", + ) + .unwrap(); for manifest_url in manifests_content.lines().rev() { let minor; let patch; @@ -141,14 +143,14 @@ impl Blacksmith { minor = captures.get(1).unwrap().as_str(); patch = captures.get(2).unwrap().as_str(); } else { - continue + continue; } let full_version = format!("1.{}.{}", minor, patch); // Skip latest stable version. if &full_version == latest_stable_version { - continue + continue; } // Download https://static.rust-lang.org/dist/channel-rust-{major.minor.patch}.toml and process it. diff --git a/book.toml b/book.toml index c4a57680e..6fa9731d2 100644 --- a/book.toml +++ b/book.toml @@ -11,8 +11,8 @@ command = "./build.bash" [output.html] additional-js = ["js/moment.min.js", "js/index.js"] -no-section-label=true -git-repository-url="https://github.com/rust-lang/rust-forge" +no-section-label = true +git-repository-url = "https://github.com/rust-lang/rust-forge" edit-url-template = "https://github.com/rust-lang/rust-forge/edit/master/{path}" curly-quotes = true From 5bf090bccedc372209b9df5fee931b31f0e7bf9d Mon Sep 17 00:00:00 2001 From: Chris Denton Date: Sat, 29 Jun 2024 06:41:42 +0000 Subject: [PATCH 14/38] Check blacksmith formatting in CI --- .github/workflows/deploy.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 02b416ec5..809fbbc30 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -18,6 +18,8 @@ jobs: - uses: actions/checkout@v3 - name: Install mdbook run: curl -sSL https://github.com/rust-lang/mdBook/releases/download/v0.4.35/mdbook-v0.4.35-x86_64-unknown-linux-gnu.tar.gz | tar -xz + - name: Check blacksmith format + run: cargo fmt --check --manifest-path=blacksmith/Cargo.toml - name: Build book run: ./mdbook build - name: Deploy book From f05d950e97bd8f2560cd6753e57135ec70217e33 Mon Sep 17 00:00:00 2001 From: oskgo <92018610+oskgo@users.noreply.github.com> Date: Thu, 11 Jul 2024 21:27:39 +0200 Subject: [PATCH 15/38] Add `S-waiting-on-ACP` to PR triage docs --- src/release/triage-procedure.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/release/triage-procedure.md b/src/release/triage-procedure.md index 4958e3e2b..45b7d58f6 100644 --- a/src/release/triage-procedure.md +++ b/src/release/triage-procedure.md @@ -20,6 +20,7 @@ ecosystem - [S-waiting-on-bikeshed] - Waiting on the consensus over a minor detail - [S-waiting-on-perf] - Waiting on the results of a perf run +- [S-waiting-on-ACP] - Waiting on API change proposal (ACP) - [S-blocked] - Waiting for another PR to be merged or for discussion to be resolved - [S-inactive] - Hasn't had activity in a while @@ -40,10 +41,11 @@ not need a review yet.) [s-waiting-on-crater]: https://github.com/rust-lang/rust/pulls?q=is%3Aopen+is%3Apr+label%3AS-waiting-on-crater+sort%3Aupdated-asc [s-waiting-on-bikeshed]: https://github.com/rust-lang/rust/pulls?q=is%3Aopen+is%3Apr+label%3AS-waiting-on-bikeshed+sort%3Aupdated-asc [s-waiting-on-perf]: https://github.com/rust-lang/rust/pulls?q=is%3Aopen+is%3Apr+label%3AS-waiting-on-perf+sort%3Aupdated-asc +[s-waiting-on-acp]: https://github.com/rust-lang/rust/pulls?q=is%3Aopen+is%3Apr+label%3AS-waiting-on-ACP+sort%3Aupdated-asc [s-blocked]: https://github.com/rust-lang/rust/pulls?q=is%3Aopen+is%3Apr+label%3AS-blocked+sort%3Aupdated-asc [s-inactive]: https://github.com/rust-lang/rust/pulls?q=is%3Aopen+is%3Apr+label%3AS-inactive+sort%3Aupdated-asc [s-experimental]: https://github.com/rust-lang/rust/pulls?q=is%3Apr+label%3AS-experimental+sort%3Aupdated-asc -[no-status-tags]: https://github.com/rust-lang/rust/pulls?q=is%3Apr+is%3Aopen+-label%3AS-waiting-on-author+-label%3AS-waiting-on-review+-label%3AS-waiting-on-team+-label%3AS-waiting-on-bors+-label%3AS-waiting-on-crater+-label%3AS-waiting-on-bikeshed+-label%3AS-waiting-on-perf+-label%3AS-blocked+-label%3AS-inactive+-label%3AS-waiting-on-fcp+-label%3AS-experimental +[no-status-tags]: https://github.com/rust-lang/rust/pulls?q=is%3Apr+is%3Aopen+-label%3AS-waiting-on-author+-label%3AS-waiting-on-review+-label%3AS-waiting-on-team+-label%3AS-waiting-on-bors+-label%3AS-waiting-on-crater+-label%3AS-waiting-on-bikeshed+-label%3AS-waiting-on-perf+-label%3AS-blocked+-label%3AS-inactive+-label%3AS-waiting-on-fcp+-label%3AS-waiting-on-ACP+-label%3AS-experimental [crater]: https://github.com/rust-lang-nursery/crater [bors]: https://github.com/rust-lang/homu From bf2c01997345b053dfb7b67ed59a8b4bf5df5d1f Mon Sep 17 00:00:00 2001 From: binarycat Date: Thu, 11 Jul 2024 18:41:49 -0400 Subject: [PATCH 16/38] note that unsized &mut refs cannot be swapped --- src/libs/maintaining-std.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/libs/maintaining-std.md b/src/libs/maintaining-std.md index b8dc159df..2dce58345 100644 --- a/src/libs/maintaining-std.md +++ b/src/libs/maintaining-std.md @@ -277,7 +277,7 @@ struct OptionCell { #### `mem::replace` and `mem::swap` -Any value behind a `&mut` reference can be replaced with a new one using `mem::replace` or `mem::swap`, so code shouldn't assume any reachable mutable references can't have their internals changed by replacing. +Any `Sized` value behind a `&mut` reference can be replaced with a new one using `mem::replace` or `mem::swap`, so code shouldn't assume any reachable mutable references can't have their internals changed by replacing. #### `mem::forget` From 6356275a1f5ab9993875c059d371fd93ceb636ff Mon Sep 17 00:00:00 2001 From: binarycat Date: Thu, 11 Jul 2024 19:00:17 -0400 Subject: [PATCH 17/38] actually link to the rust blogs --- src/platforms/blogs.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/platforms/blogs.md b/src/platforms/blogs.md index 97606a11f..e826b70c8 100644 --- a/src/platforms/blogs.md +++ b/src/platforms/blogs.md @@ -2,8 +2,11 @@ ## Context -The Rust project maintains two blogs. The “main blog” (blog.rust-lang.org) and a “team blog” -(blog.rust-lang.org/inside-rust). This document provides the guidelines for what it takes to write +The Rust project maintains two blogs. +The “main blog” ([blog.rust-lang.org](https://blog.rust-lang.org/index.html)) +and a “team blog” +([blog.rust-lang.org/inside-rust](https://blog.rust-lang.org/inside-rust/)). +This document provides the guidelines for what it takes to write a post for each of those blogs, as well as how to propose a post and to choose which blog is most appropriate. From 132ca44de9994a7be2c17d41f1deb87c9d3b12f8 Mon Sep 17 00:00:00 2001 From: Chris Denton Date: Wed, 17 Jul 2024 08:09:21 +0000 Subject: [PATCH 18/38] Documents `users_on_vacation` --- src/triagebot/pr-assignment.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/triagebot/pr-assignment.md b/src/triagebot/pr-assignment.md index 6ca1e2a78..e162e3176 100644 --- a/src/triagebot/pr-assignment.md +++ b/src/triagebot/pr-assignment.md @@ -74,6 +74,16 @@ fallback = ["@Mark-Simulacrum"] If the `owners` map is configured, then triagebot will automatically select a reviewer based on which files were modified in the PR. The existence of the owners table also enables the ability for users to post a comment with `r? name` to set the assignment to a specific user. +### Vacation + +If a reviewer wants to temporarily prevent themselves from being assigned (automatically or manually) they can add themselves to the special +`assign.users_on_vacation` group. + +```toml +[assign] +users_on_vacation = ["jyn514", "ChrisDenton"] +``` + ### Additional new PR trigger options Triagebot will also post a welcome message to the user. From 736b5f89f26cb388fd37241482afec6877d459e8 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 22 Jul 2024 18:10:21 +0000 Subject: [PATCH 19/38] Bump openssl from 0.10.60 to 0.10.66 in /blacksmith Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.60 to 0.10.66. - [Release notes](https://github.com/sfackler/rust-openssl/releases) - [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-v0.10.60...openssl-v0.10.66) --- updated-dependencies: - dependency-name: openssl dependency-type: indirect ... Signed-off-by: dependabot[bot] --- blacksmith/Cargo.lock | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/blacksmith/Cargo.lock b/blacksmith/Cargo.lock index 54868cfc9..21e14d949 100644 --- a/blacksmith/Cargo.lock +++ b/blacksmith/Cargo.lock @@ -890,9 +890,9 @@ dependencies = [ [[package]] name = "openssl" -version = "0.10.60" +version = "0.10.66" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "79a4c6c3a2b158f7f8f2a2fc5a969fa3a068df6fc9dbb4a43845436e3af7c800" +checksum = "9529f4786b70a3e8c61e11179af17ab6188ad8d0ded78c5529441ed39d4bd9c1" dependencies = [ "bitflags 2.4.1", "cfg-if", @@ -922,9 +922,9 @@ checksum = "ff011a302c396a5197692431fc1948019154afc178baf7d8e37367442a4601cf" [[package]] name = "openssl-sys" -version = "0.9.96" +version = "0.9.103" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3812c071ba60da8b5677cc12bcb1d42989a65553772897a7e0355545a819838f" +checksum = "7f9e8deee91df40a943c71b917e5874b951d32a802526c85721ce3b776c929d6" dependencies = [ "cc", "libc", From 9843fc8455c3d89673c6c5d8b1cecbf5694c691f Mon Sep 17 00:00:00 2001 From: MarcoIeni <11428655+MarcoIeni@users.noreply.github.com> Date: Thu, 1 Aug 2024 14:34:21 +0200 Subject: [PATCH 20/38] Improve bastion IP docs --- src/infra/docs/bastion.md | 59 +++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 34 deletions(-) diff --git a/src/infra/docs/bastion.md b/src/infra/docs/bastion.md index a3243f053..fb39ddfc8 100644 --- a/src/infra/docs/bastion.md +++ b/src/infra/docs/bastion.md @@ -50,51 +50,42 @@ access to. Once that's done [apply the playbook][ansible-apply] and [add a new whitelisted IP address](#adding-a-whitelisted-ip). -### Adding a whitelisted IP +## Editing a whitelisted IP Due to privacy reasons, all the static IP addresses of team members with access to the bastion are stored on [AWS SSM Parameter Store][ssm] instead of public -git repositories. To add an IP address you can run this command (taking care of -replacing `USERNAME` and `IP_ADDRESS` with the proper values): +git repositories. +When running the following commands, replace `USERNAME` and `IP_ADDRESS` with the proper values. -``` -aws ssm put-parameter --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" -``` +### Adding a whitelisted IP -You'll also need to add the username to the list in -[`terraform/bastion/firewall.tf`][allowed-ips] (local variable -`allowed_users`). Once you made all the needed changes you wanted you need to -[apply the Terraform configuration][terraform-apply]. +1. Run: + ``` + aws ssm put-parameter --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" + ``` +2. Add the username to the list in + [`terraform/bastion/firewall.tf`][allowed-ips] (local variable + `allowed_users`). +3. [apply the Terraform configuration][terraform-apply]. ### Updating a whitelisted IP -Due to privacy reasons, all the static IP addresses of team members with access -to the bastion are stored on [AWS SSM Parameter Store][ssm] instead of public -git repositories. To update an IP address you can run this command (taking care -of replacing `USERNAME` and `IP_ADDRESS` with the proper values): - -``` -aws ssm put-parameter --overwrite --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" -``` - -Once you made all the needed changes you wanted you need to [apply the -Terraform configuration][terraform-apply]. +1. Run: + ``` + aws ssm put-parameter --overwrite --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" + ``` +2. [apply the Terraform configuration][terraform-apply]. ### Removing a whitelisted IP -Due to privacy reasons, all the static IP addresses of team members with access -to the bastion are stored on [AWS SSM Parameter Store][ssm] instead of public -git repositories. To remove an IP address you can run this command (taking care -of replacing `USERNAME` with the proper value): - -``` -aws ssm delete-parameter --name "/prod/bastion/allowed-ips/USERNAME" -``` - -You'll also need to remove the username from the list in -[`terraform/bastion/firewall.tf`][allowed-ips] (local variable -`allowed_users`). Once you made all the needed changes you wanted you need to -[apply the Terraform configuration][terraform-apply]. +1. Run: + ``` + aws ssm delete-parameter --name "/prod/bastion/allowed-ips/USERNAME" + ``` +2. Remove the username from the list in + [`terraform/bastion/firewall.tf`][allowed-ips] (local variable + `allowed_users`). +3. [apply the Terraform configuration][terraform-apply]. [ansible]: https://github.com/rust-lang/simpleinfra/blob/master/ansible/playbooks/bastion.yml [terraform]: https://github.com/rust-lang/simpleinfra/tree/master/terraform/bastion From cd020204f94068cc899b395cf1ebc25b230d8e5c Mon Sep 17 00:00:00 2001 From: MarcoIeni <11428655+MarcoIeni@users.noreply.github.com> Date: Mon, 5 Aug 2024 14:17:38 +0200 Subject: [PATCH 21/38] specify aws region in bastion docs --- src/infra/docs/bastion.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/infra/docs/bastion.md b/src/infra/docs/bastion.md index fb39ddfc8..e9cb02cf6 100644 --- a/src/infra/docs/bastion.md +++ b/src/infra/docs/bastion.md @@ -61,7 +61,7 @@ When running the following commands, replace `USERNAME` and `IP_ADDRESS` with th 1. Run: ``` - aws ssm put-parameter --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" + aws ssm put-parameter --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" --region us-west-1 ``` 2. Add the username to the list in [`terraform/bastion/firewall.tf`][allowed-ips] (local variable @@ -72,7 +72,7 @@ When running the following commands, replace `USERNAME` and `IP_ADDRESS` with th 1. Run: ``` - aws ssm put-parameter --overwrite --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" + aws ssm put-parameter --overwrite --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" --region us-west-1 ``` 2. [apply the Terraform configuration][terraform-apply]. @@ -80,7 +80,7 @@ When running the following commands, replace `USERNAME` and `IP_ADDRESS` with th 1. Run: ``` - aws ssm delete-parameter --name "/prod/bastion/allowed-ips/USERNAME" + aws ssm delete-parameter --name "/prod/bastion/allowed-ips/USERNAME" --region us-west-1 ``` 2. Remove the username from the list in [`terraform/bastion/firewall.tf`][allowed-ips] (local variable From f9caf0598d05c986a241683064485e2a90fca93f Mon Sep 17 00:00:00 2001 From: MarcoIeni <11428655+MarcoIeni@users.noreply.github.com> Date: Fri, 9 Aug 2024 14:45:41 +0200 Subject: [PATCH 22/38] docs(bastion): improve how to login into servers --- src/infra/docs/bastion.md | 42 +++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/src/infra/docs/bastion.md b/src/infra/docs/bastion.md index e9cb02cf6..9ae6c2aa8 100644 --- a/src/infra/docs/bastion.md +++ b/src/infra/docs/bastion.md @@ -1,9 +1,9 @@ # Bastion server -* FQDN: `bastion.infra.rust-lang.org` -* [Ansible playbook][ansible] to deploy this server. -* [Terraform configuration][terraform] to create AWS resources. -* [Instance metrics][grafana] (only available to infra team members). +- FQDN: `bastion.infra.rust-lang.org` +- [Ansible playbook][ansible] to deploy this server. +- [Terraform configuration][terraform] to create AWS resources. +- [Instance metrics][grafana] (only available to infra team members). ## Logging into servers through the bastion @@ -12,20 +12,28 @@ directly to a production server with SSH. Instead, all connections must come from a small server called the "bastion", which only allows connections from a few whitelisted networks and logs any connection attempt. -To log into a server through the bastion you can use SSH's `-J` flag: +To log into a server through the bastion, use one of the following methods: -``` -ssh -J bastion.infra.rust-lang.org servername.infra.rust-lang.org -``` +- Use SSH's `-J` flag: -It's also possible to configure SSH to always jump through the bastion when -connecting to a host. Add this snippet to your SSH configuration file (usually -located in `~/.ssh/config`): + ``` + ssh -J @bastion.infra.rust-lang.org @servername.infra.rust-lang.org + ``` -``` -Host servername.infra.rust-lang.org - ProxyJump bastion.infra.rust-lang.org -``` +- Configure your SSH client to always jump through the bastion when connecting to a host: + + - Add this snippet to your SSH configuration file (usually located in `~/.ssh/config`): + + ``` + Host servername.infra.rust-lang.org + ProxyJump @bastion.infra.rust-lang.org + ``` + + - Use SSH: + + ``` + ssh @servername.infra.rust-lang.org + ``` Please remember the bastion server only allows connections from a small list of IP addresses. Infra team members with AWS access can change the whitelist, but @@ -71,17 +79,21 @@ When running the following commands, replace `USERNAME` and `IP_ADDRESS` with th ### Updating a whitelisted IP 1. Run: + ``` aws ssm put-parameter --overwrite --type String --name "/prod/bastion/allowed-ips/USERNAME" --value "IP_ADDRESS/32" --region us-west-1 ``` + 2. [apply the Terraform configuration][terraform-apply]. ### Removing a whitelisted IP 1. Run: + ``` aws ssm delete-parameter --name "/prod/bastion/allowed-ips/USERNAME" --region us-west-1 ``` + 2. Remove the username from the list in [`terraform/bastion/firewall.tf`][allowed-ips] (local variable `allowed_users`). From f0dd31fd5c12eb0cfc6983828787b5130f8fa8cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20Ber=C3=A1nek?= Date: Sat, 24 Aug 2024 10:07:27 +0200 Subject: [PATCH 23/38] Add link to Rust 2023 annual survey results --- src/community/survey-faq.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/community/survey-faq.md b/src/community/survey-faq.md index 307c77c17..4af1f0510 100644 --- a/src/community/survey-faq.md +++ b/src/community/survey-faq.md @@ -47,6 +47,7 @@ We expect to publish results from the survey within a month or two of the survey ## Where can I see the previous survey reports? +- [State of Rust 2023](https://blog.rust-lang.org/2024/02/19/2023-Rust-Annual-Survey-2023-results.html) - [State of Rust 2022](https://blog.rust-lang.org/2023/08/07/Rust-Survey-2023-Results.html) - [State of Rust 2021](https://blog.rust-lang.org/2022/02/15/Rust-Survey-2021.html) - [State of Rust 2020](https://blog.rust-lang.org/2020/12/16/rust-survey-2020.html) From 130ee3f7a37295f0624f4b125118c94540a882ee Mon Sep 17 00:00:00 2001 From: MarcoIeni <11428655+MarcoIeni@users.noreply.github.com> Date: Mon, 23 Sep 2024 14:51:25 +0200 Subject: [PATCH 24/38] docs(dev-desktops): document available disk space --- src/infra/docs/dev-desktop.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/infra/docs/dev-desktop.md b/src/infra/docs/dev-desktop.md index d435e51c1..0c35a5985 100644 --- a/src/infra/docs/dev-desktop.md +++ b/src/infra/docs/dev-desktop.md @@ -186,6 +186,16 @@ It helps to check first and make sure that the package is available for both `arm64` and `amd64` architectures. The dev desktops are currently running Ubuntu 22.04 LTS. +## Available disk space + +Every user has a quota of how much disk space they can use. +Currently, the quota is set to +[150GB](https://github.com/rust-lang/simpleinfra/blob/8a59e4faeb75a09b072671c74a7cb70160ebef50/ansible/roles/dev-desktop/defaults/main.yml#L7). + +Exceeding the quota will result in the error `Disk quota exceeded (os error 122)`. + +The [remarks on disk space] section of the rustc dev guide contains some tips on how to clean your disk space. + ## How to give feedback and report issues If you experience any problems with the dev desktops, or have feedback and @@ -202,6 +212,7 @@ We might ask you to create an issue in the [rust-lang/simpleinfra] repository. [rust foundation]: https://foundation.rust-lang.org/ [rust-lang/rust]: https://github.com/rust-lang/rust [rust-lang/simpleinfra]: https://github.com/rust-lang/simpleinfra +[remarks on disk space]: https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html#remarks-on-disk-space [visual studio code]: https://code.visualstudio.com/ [certain teams]: https://github.com/search?q=repo%3Arust-lang%2Fteam+path%3Ateams%2F*.toml+dev-desktop&type=code&ref=advsearch [simpleinfra-vars-user-config]: https://github.com/rust-lang/simpleinfra/blob/dbf839ef25155df1f33c18f151283436b0f70f3b/ansible/roles/dev-desktop/defaults/main.yml#L12:L16 From b3ac160172831f6d1135b86dd67d79745b216a6d Mon Sep 17 00:00:00 2001 From: Pietro Albini Date: Sun, 1 Sep 2024 10:45:27 +0200 Subject: [PATCH 25/38] update release process --- src/release/process.md | 176 +++++++++++++++++++++-------------------- 1 file changed, 92 insertions(+), 84 deletions(-) diff --git a/src/release/process.md b/src/release/process.md index f1bd20a71..36e4e701b 100644 --- a/src/release/process.md +++ b/src/release/process.md @@ -2,10 +2,10 @@ Here's how Rust is currently released: -## Bump the stable version number (T-6 days, Friday the week before) +## Bump the stable version number (Friday the week before) Open a PR bumping the version number in `src/version`. `r+ rollup=never` this -PR. +PR (self-approve it). Mark it as `rollup=never`, because if it lands in a rollup as *not* the first PR then other pull requests in that rollup will be incorrectly associated with @@ -14,56 +14,87 @@ the prior release. This is effectively when the beta branch forks -- when beta is promoted, it will be based off of the PR that landed just before this version number bump PR. -## Promote branches (T-3 days, Monday) +## Promote branches (Monday) Both promotions should happen on Monday. You can open both PRs at the same -time, but make sure the stable promotion lands first. +time, but prioritize landing the stable promotion first (to maximize the +pre-release testing period). -### Beta to stable +### Updating the base of the `beta` and `stable` branches -[Obtain AWS CLI credentials][awscli] and run this command from the [simpleinfra] repository: +Run this command from the [rust-lang/release-team] repository[^auth]: ``` -./release-scripts/promote-release.py branches +./scripts/start-release.py update-rust-branches ``` -Once that's done, send a PR to the freshly created beta branch of rust-lang/rust -with two commits: +Remember that `start-release.py` starts a job in the background, and the script +will exit *before* the branches are updated. Watch the logs to see when the +background job finishes before proceeding. -* The changes caused by running `./x.py run replace-version-placeholder` -* An update of `src/ci/channel` to `beta` +### `stable` PR -The version placeholder replacement changes must be in a separate commit so -that they can be cherry picked to the master branch. - -Also send a PR to rust-lang/rust targeting the new stable branch making the +Send a PR to [rust-lang/rust] targeting the new `stable` branch making the following changes: -- Update `src/ci/channel` to `stable` -- Update release notes to the latest available copy - * e.g., `git checkout origin/master -- RELEASES.md` +- Update release notes to the latest available copy: + + - If the release notes PR was merged: + + ``` + git checkout origin/master -- RELEASES.md + ``` + + - Otherwise, manually copy `RELEASES.md` from the pending release notes PR + +- Update `src/ci/channel` to `stable`: + + ``` + git checkout origin/master -- RELEASES.md + ``` + +Self-approve the PR with `r+ rollup=never p=1000`. + +### `beta` PR -Once the PRs are sent, r+ both and give them a high `p=1000` (for stable) and -`p=10` for beta. +Send a PR to [rust-lang/rust] targeting the new `beta` branch with these +changes: -After the PR is merged you'll need to start a **dev** release. [Obtain AWS CLI -credentials][awscli] and run this command from the [simpleinfra] repository: +* Run this command and create a **separate commit** with just its output: + + ``` + ./x.py run replace-version-placeholder + ``` + +* Update `src/ci/channel` to `beta` + +Self-approve the PR with `r+ rollup=never p=10`. + +### Publish the pre-release on the `dev-static` environment + +After the `stable` PR is merged you'll need to start the pre-release. Run this command from the +[rust-lang/release-team] repository[^auth]: ``` -# The date here is of the actual, production, stable release. Used for the blog post. -./release-scripts/promote-release.py release dev stable --release-date YYYY-MM-DD +./scripts/start-release.py publish-rust-dev-stable YYYY-MM-DD ``` -## Master bootstrap update (T-2 day, Tuesday) +You need to replace `YYYY-MM-DD` with the date of the release (Thursday). + +## Master bootstrap update (Tuesday) Send a PR to the master branch to: -- Cherry pick the commit that ran `./x.py run replace-version-placeholder` +- Cherry pick the commit that ran `replace-version-placeholder` from the now merged beta branch PR. Do not re-run the tool as there might have been other stabilizations on master which were not included in the branched beta, so may not be attributed to the current release. -- Run `./x.py run src/tools/bump-stage0` to update the bootstrap compiler to - the beta you created yesterday. + +- Run this to update the bootstrap compiler to the beta you created yesterday: + + ``` + ./x.py run src/tools/bump-stage0 + ``` - Remove references to the `bootstrap` and `not(bootstrap)` conditional compilation attributes. You can find all of them by installing [ripgrep] and @@ -84,78 +115,55 @@ Send a PR to the master branch to: ## Release day (Thursday) -Decide on a time to do the release, T. - -- **T-50m** - Run the following command in a shell with [AWS - credentials][awscli] in the [simpleinfra] repository: - - ``` - ./release-scripts/promote-release.py release prod stable - ``` - - That'll, in the background, schedule the `promote-release` binary to run on - the production secrets (not the dev secrets). That'll sign everything, upload - it, update the html index pages, and invalidate the CDN. Note that this takes - about 30 minutes right now. This will also push a signed tag to rust-lang/rust. +Decide on a time to do the release. Let Mara know of the time, so that she can +be ready to post the release on the project's social media channels. -- **T-2m** - Merge blog post. +As of September 2024 a release takes between 75 and 90 minutes to complete, so +start the release process earlier enough to hit the time you planned. -- **T** - Tweet and post everything! +To start the release, Run this command in the [rust-lang/release-team] +repository[^auth]: - - Twitter [@rustlang](https://twitter.com/rustlang) - - [Users forum](https://users.rust-lang.org/) +``` +./scripts/start-release.py publish-rust-prod-stable +``` -- **T+5m** - Release and tag Cargo. From a rust-lang/rust checkout (script will - checkout the stable branch automatically), run the following script from - [simpleinfra]. +The command will start a background job to invoke [`promote-release`] targeting +the production environment, and it will show the instructions to follow its +logs. - ```sh - ../simpleinfra/release-scripts/tag-cargo.sh - ``` +When the release process completes, merge the blog post PR and inform Mara to +announce the release on social media. Finally, bask in your success 🎉 -- **T+1hr** Send a PR to the beta branch running `./x.py run - src/tools/bump-stage0` to bump the boostrap compiler to the stable you - just released. +## Beta stage0 update (Friday) -[update-thanks]: https://github.com/rust-lang/thanks/actions/workflows/ci.yml +Send a PR to the beta branch updating the stage0 to the stable release you +published: -Bask in your success. +``` +./x run src/tools/bump-stage0 +``` -## Rebuilding stable pre-releases +## Appendix: Rebuilding stable pre-releases If something goes wrong and we need to rebuild the stable artifacts, merge the PR on the `stable` branch of the [rust-lang/rust] repository. Once the commit -is merged, issue the following command in a shell with [AWS -credentials][awscli] on the [simpleinfra] repository: +is merged, [authenticate with AWS][awscli] and run this command in the +[rust-lang/release-team] repository: ``` -./release-scripts/promote-release.py release dev stable --bypass-startup-checks +./scripts/start-release.py publish-rust-dev-stable-rebuild ``` -You'll also want to update the previously published blog post and internals post -with the new information. - -## Publishing a nightly based off a try build - -Sometimes a PR requires testing how it behaves when downloaded from rustup, for -example after a manifest change. In those cases it's possible to publish a new -nightly based off that PR on dev-static.rust-lang.org. - -Once the try build finishes grab the merge commit SHA and run the following -command in a shell with [AWS credentials][awscli] on the [simpleinfra] -repository: +You'll also want to update the previously published pre-release announcement on +the blog and internals with the new information. -```sh -./release-scripts/promote-release.py release dev nightly $MERGE_COMMIT_SHA -``` - -When the release process end you'll be able to install the new nightly with: - -```sh -RUSTUP_DIST_SERVER=https://dev-static.rust-lang.org rustup toolchain install nightly -``` +[^auth]: Publishing releases require authentication, and only authorized + members of the release team can invoke it. The command will prompt you on how + to setup your environment and how to authenticate with AWS the first time you + execute it. -[awscli]: ../infra/docs/aws-access.md#using-the-aws-cli -[rust-lang/rust]: https://github.com/rust-lang/rust -[simpleinfra]: https://github.com/rust-lang/simpleinfra +[rust-lang/rust]: https://github.com/[rust-lang/rust] +[rust-lang/release-team]: https://github.com/rust-lang/release-team [ripgrep]: https://github.com/burntsushi/ripgrep +[`promote-release`]: https://github.com/rust-lang/promote-release From d32174d2fe30d67a9df598dd7a80ad43b81d3d52 Mon Sep 17 00:00:00 2001 From: Pietro Albini Date: Mon, 2 Sep 2024 10:27:15 +0200 Subject: [PATCH 26/38] Apply suggestions from code review Co-authored-by: Urgau <3616612+Urgau@users.noreply.github.com> --- src/release/process.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/src/release/process.md b/src/release/process.md index 36e4e701b..056eef064 100644 --- a/src/release/process.md +++ b/src/release/process.md @@ -47,11 +47,7 @@ following changes: - Otherwise, manually copy `RELEASES.md` from the pending release notes PR -- Update `src/ci/channel` to `stable`: - - ``` - git checkout origin/master -- RELEASES.md - ``` +- Update `src/ci/channel` to `stable` Self-approve the PR with `r+ rollup=never p=1000`. @@ -115,8 +111,9 @@ Send a PR to the master branch to: ## Release day (Thursday) -Decide on a time to do the release. Let Mara know of the time, so that she can -be ready to post the release on the project's social media channels. +Decide on a time to do the release. Let the Social Media coordinator (currently Mara) +know of the time, so that she can be ready to post the release on the project's social +media channels. As of September 2024 a release takes between 75 and 90 minutes to complete, so start the release process earlier enough to hit the time you planned. From e846704ae9bad310304fad0665cba993499b88d3 Mon Sep 17 00:00:00 2001 From: Pietro Albini Date: Mon, 14 Oct 2024 12:35:43 +0200 Subject: [PATCH 27/38] update the release process with boxy's feedback --- src/release/process.md | 46 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43 insertions(+), 3 deletions(-) diff --git a/src/release/process.md b/src/release/process.md index 056eef064..2ebf1698d 100644 --- a/src/release/process.md +++ b/src/release/process.md @@ -2,6 +2,24 @@ Here's how Rust is currently released: +## A note about the `start-release.py` script + +Steps of the release process that require interacting with our production +environment are executed through the `start-release.py` script. The script +requires you to install programs and configure your local environmet, and it +will guide you through the setup. + +The first time you run the script (or when the pre-requisites change), you will +need to invoke the script *multiple times* until everything is setup correctly. + +`start-release.py` will always start a CI job in the background. To know when +it finishes, you have to watch the logs. WHen the build finishes, a line like +this will appear in the logs: + +``` +Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED +``` + ## Bump the stable version number (Friday the week before) Open a PR bumping the version number in `src/version`. `r+ rollup=never` this @@ -51,6 +69,15 @@ following changes: Self-approve the PR with `r+ rollup=never p=1000`. +Note that we need to merge this PR as soon as possible, to maximise the +pre-release testing time. If another PR is being tested by bors, and CI is not +going to finish soon (use your judgement here), you can "yield" +priority to the stable release PR by going into that PR and typing this +comment: + +> @bors retry +> Yield priority to the stable release. + ### `beta` PR Send a PR to [rust-lang/rust] targeting the new `beta` branch with these @@ -79,6 +106,11 @@ You need to replace `YYYY-MM-DD` with the date of the release (Thursday). ## Master bootstrap update (Tuesday) +This step can only be done after the new beta has been released. The release +process for the beta happens automatically at 00:00 UTC every day, so if the +beta PR landed after that you will have to wait another day. You can check +whether beta has been released by installing it with rustup. + Send a PR to the master branch to: - Cherry pick the commit that ran `replace-version-placeholder` @@ -109,11 +141,19 @@ Send a PR to the master branch to: relevant documentation block (or in a new documentation block). - Replace any `#[cfg_attr(not(bootstrap), $attr)]` with `#[$attr]`. + Note that if a PR adds `cfg(bootstrap)` and is merged between the beta PR and + the master bootstrap update, the `rg` invocation will show them even though + they won't have to be removed. The easiest way to handle this is to change + them anyway and let CI show you the failure. + ## Release day (Thursday) -Decide on a time to do the release. Let the Social Media coordinator (currently Mara) -know of the time, so that she can be ready to post the release on the project's social -media channels. +Decide on a time to do the release. You are fully in charge of deciding when +the release happens, pick the time that works best for you. The only constraint +is, the release process must start and finish within the release day (in UTC). + +Let the Social Media coordinator (currently Mara) know of the time, so that she +can be ready to post the release on the project's social media channels. As of September 2024 a release takes between 75 and 90 minutes to complete, so start the release process earlier enough to hit the time you planned. From a383e366b5bbb09059742e7218be53d69dc490e8 Mon Sep 17 00:00:00 2001 From: Yerkebulan Tulibergenov Date: Mon, 14 Oct 2024 15:58:43 -0700 Subject: [PATCH 28/38] deduplicate entries in archive of stable standalone installers --- blacksmith/src/lib.rs | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/blacksmith/src/lib.rs b/blacksmith/src/lib.rs index c75c85602..581faa12f 100644 --- a/blacksmith/src/lib.rs +++ b/blacksmith/src/lib.rs @@ -128,6 +128,12 @@ impl Blacksmith { let latest_stable_version = &blacksmith.stable_version.clone().unwrap(); + // We need to use a map to deduplicate entries and sort them in the correct order. + // There are multiple entries for stable 1.8.0, 1.14.0, 1.15.1, 1.49.0 in MANIFESTS_URL. + // Keys contain (minor_version, patch_version) and values contain (full_version, platforms). + let mut previous_stable_version_map: BTreeMap<(u32, u32), (String, Vec)> = + BTreeMap::new(); + // Go over stable versions in https://static.rust-lang.org/manifests.txt in reverse order. let manifests_content = reqwest::blocking::get(MANIFESTS_URL)?.text()?; let stable_manifest_url_regex = regex::Regex::new( @@ -140,14 +146,19 @@ impl Blacksmith { // Check if it's a stable version. if let Some(captures) = stable_manifest_url_regex.captures(&(manifest_url)) { - minor = captures.get(1).unwrap().as_str(); - patch = captures.get(2).unwrap().as_str(); + minor = captures.get(1).unwrap().as_str().parse::().unwrap(); + patch = captures.get(2).unwrap().as_str().parse::().unwrap(); } else { continue; } let full_version = format!("1.{}.{}", minor, patch); + // Check if we already processed that version. + if previous_stable_version_map.contains_key(&(minor, patch)) { + continue; + } + // Skip latest stable version. if &full_version == latest_stable_version { continue; @@ -169,6 +180,9 @@ impl Blacksmith { let version = rust.version.split(' ').next().unwrap().to_string(); + // Sanity check. + assert_eq!(&full_version, &version); + let platforms = rust .target .into_iter() @@ -181,9 +195,13 @@ impl Blacksmith { }) .collect::>(); + previous_stable_version_map.insert((minor, patch), (version, platforms)); + } + + for (_, (version, platforms)) in previous_stable_version_map.into_iter().rev() { blacksmith .previous_stable_versions - .push((version.clone(), platforms)); + .push((version, platforms)); } blacksmith.last_update = unix_time(); From 2d61c587ff49f9b8d3999b77e9bb03744329abd8 Mon Sep 17 00:00:00 2001 From: Yerkebulan Tulibergenov Date: Tue, 15 Oct 2024 10:02:06 -0700 Subject: [PATCH 29/38] add support for stable versions before 1.8.0 --- blacksmith/src/lib.rs | 56 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/blacksmith/src/lib.rs b/blacksmith/src/lib.rs index 581faa12f..dca7ab703 100644 --- a/blacksmith/src/lib.rs +++ b/blacksmith/src/lib.rs @@ -198,6 +198,62 @@ impl Blacksmith { previous_stable_version_map.insert((minor, patch), (version, platforms)); } + // There are no manifests for stable versions before 1.8.0, + // so we hardcode them instead. + // i686-pc-windows-msvc wasn't supported until version 1.3.0. + for minor in 3..8 { + previous_stable_version_map.insert( + (minor, 0), + ( + format!("1.{}.0", minor), + Vec::from([ + "i686-apple-darwin".to_string(), + "i686-pc-windows-gnu".to_string(), + "i686-pc-windows-msvc".to_string(), + "i686-unknown-linux-gnu".to_string(), + "x86_64-apple-darwin".to_string(), + "x86_64-pc-windows-gnu".to_string(), + "x86_64-pc-windows-msvc".to_string(), + "x86_64-unknown-linux-gnu".to_string(), + ]), + ), + ); + } + + // x86_64-pc-windows-msvc wasn't supported until version 1.2.0. + previous_stable_version_map.insert( + (2, 0), + ( + "1.2.0".to_string(), + Vec::from([ + "i686-apple-darwin".to_string(), + "i686-pc-windows-gnu".to_string(), + "i686-unknown-linux-gnu".to_string(), + "x86_64-apple-darwin".to_string(), + "x86_64-pc-windows-gnu".to_string(), + "x86_64-pc-windows-msvc".to_string(), + "x86_64-unknown-linux-gnu".to_string(), + ]), + ), + ); + + for minor in 0..2 { + previous_stable_version_map.insert( + (minor, 0), + ( + format!("1.{}.0", minor), + Vec::from([ + "i686-apple-darwin".to_string(), + "i686-pc-windows-gnu".to_string(), + "i686-unknown-linux-gnu".to_string(), + "x86_64-apple-darwin".to_string(), + "x86_64-pc-windows-gnu".to_string(), + "x86_64-unknown-linux-gnu".to_string(), + ]), + ), + ); + } + for (_, (version, platforms)) in previous_stable_version_map.into_iter().rev() { blacksmith .previous_stable_versions From 26b7323a008a071120dbc9658e8da26f189f8046 Mon Sep 17 00:00:00 2001 From: Urgau <3616612+Urgau@users.noreply.github.com> Date: Wed, 16 Oct 2024 10:20:57 +0200 Subject: [PATCH 30/38] Include toml files when searching for `#[cfg(bootstrap)]` --- src/release/process.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/release/process.md b/src/release/process.md index 2ebf1698d..16c32abf4 100644 --- a/src/release/process.md +++ b/src/release/process.md @@ -129,7 +129,7 @@ Send a PR to the master branch to: running this command: ``` - rg '#!?\[.*\(bootstrap' -t rust + rg '#!?\[.*\(bootstrap' -t rust -t toml ``` The general guidelines (both for `#[]` and `#![]`) are: From d11a8ac71e6084f841c3c9283b231278e7f8eee8 Mon Sep 17 00:00:00 2001 From: binarycat Date: Thu, 24 Oct 2024 12:34:07 -0500 Subject: [PATCH 31/38] add instructions for triaging ICE issues based off of disscussion in https://rust-lang.zulipchat.com/#narrow/channel/242269-t-release.2Ftriage/topic/Please.20make.20sure.20to.20label.20ICEs.20by.20source.20region --- src/release/issue-triaging.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/release/issue-triaging.md b/src/release/issue-triaging.md index 070604a29..f5b46d2bf 100644 --- a/src/release/issue-triaging.md +++ b/src/release/issue-triaging.md @@ -54,6 +54,16 @@ To see a list of all labels, check out the "labels" page next to the search bar [IRLO]: https://internals.rust-lang.org/ [Discord]: https://discord.gg/rust-lang +### ICE Triage +For [Issues that have both I-ICE and needs-triage](https://github.com/rust-lang/rust/issues?q=is%3Aissue%20state%3Aopen%20label%3AI-ICE%20label%3Aneeds-triage) +* If it an older (like latest stable) version of rust, ask for or check the latest nightly (potentially check for duplicates) +* If it does not have a reproduction, comment asking for one and add S-needs-repro. if there isn't one for around a month it should generally be closed. +* If the reproduction is not minimal, add `E-needs-mcve` or create a Minimal Complete and Verifyable Example yourself. +* Add `A-*` labels based on the code that causes the issue (check backtraces!), + and the nature of the repro (eg. if the repro is a weird trait impl or the backtrace points to `rustc_trait_selection`, add `A-traits`) +* Add `F-*`, `requires-*`, and `regression-*` labels as appropriate. + + ## Further triaging For issues that have been through the initial triaging step (that is, don't have the `needs-triage` label anymore), there are usually From 002a7eae2633821b39c578036d5db43381db423b Mon Sep 17 00:00:00 2001 From: binarycat Date: Thu, 24 Oct 2024 13:42:48 -0500 Subject: [PATCH 32/38] add more info about ICE triage --- src/release/issue-triaging.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/release/issue-triaging.md b/src/release/issue-triaging.md index f5b46d2bf..cfee6c5e4 100644 --- a/src/release/issue-triaging.md +++ b/src/release/issue-triaging.md @@ -56,12 +56,15 @@ To see a list of all labels, check out the "labels" page next to the search bar ### ICE Triage For [Issues that have both I-ICE and needs-triage](https://github.com/rust-lang/rust/issues?q=is%3Aissue%20state%3Aopen%20label%3AI-ICE%20label%3Aneeds-triage) -* If it an older (like latest stable) version of rust, ask for or check the latest nightly (potentially check for duplicates) +* Check that the issue is actually an ICE, and not more accuratly described with `I-crash` or `I-hang`. +* If it is an older (like latest stable) version of rust, ask for or check the latest nightly. +* Check for duplicates, but don't close as duplicate unless you're sure they represent the same underlying issue. + Prefer simply linking to the issue as possibly related/duplicate. * If it does not have a reproduction, comment asking for one and add S-needs-repro. if there isn't one for around a month it should generally be closed. -* If the reproduction is not minimal, add `E-needs-mcve` or create a Minimal Complete and Verifyable Example yourself. +* If the reproduction is not minimal, add `E-needs-mcve` or create a Minimal Complete and Verifible Example yourself. * Add `A-*` labels based on the code that causes the issue (check backtraces!), and the nature of the repro (eg. if the repro is a weird trait impl or the backtrace points to `rustc_trait_selection`, add `A-traits`) -* Add `F-*`, `requires-*`, and `regression-*` labels as appropriate. +* Add `T-*`, `WG-*`, `PG-*`, `F-*`, `requires-*`, and `regression-*` labels as appropriate. ## Further triaging From 71d0cf2061b7a8fa32b868bba48152743490c95f Mon Sep 17 00:00:00 2001 From: Eric Huss Date: Wed, 30 Oct 2024 11:12:55 -0700 Subject: [PATCH 33/38] Remove RFC rendered link step --- src/lang/rfc-merge-procedure.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/lang/rfc-merge-procedure.md b/src/lang/rfc-merge-procedure.md index 48632aa1a..bfe8091ad 100644 --- a/src/lang/rfc-merge-procedure.md +++ b/src/lang/rfc-merge-procedure.md @@ -64,12 +64,5 @@ To track further discussion, subscribe to the tracking issue here: rust-lang/rust#41517 ``` -### Step 4: Update the rendered link - -Update the rendered link in the first post of the PR to the permanent home under -`https://github.com/rust-lang/rfcs/blob/master/text/`. - -(This way future visitors can open it easily after the PR branch is deleted.) - ### That's it, you're done! From 7334e4ceb7ef7ef0cf151081686d3e262e4db95c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=AE=B8=E6=9D=B0=E5=8F=8B=20Jieyou=20Xu=20=28Joe=29?= <39484203+jieyouxu@users.noreply.github.com> Date: Fri, 8 Nov 2024 20:32:35 +0800 Subject: [PATCH 34/38] Update triage issue label docs (#776) --- src/release/issue-triaging.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/release/issue-triaging.md b/src/release/issue-triaging.md index 070604a29..3b1e731e6 100644 --- a/src/release/issue-triaging.md +++ b/src/release/issue-triaging.md @@ -80,10 +80,14 @@ There are many different labels that can be applied to issues. - [`PG-*`]: Specifies the project groups that this issue is relevant to, for example the `PG-exploit-mitigations`. - [`C-*`]: Specifies the category of the label, for example a bug, tracking issue or discussion - `A-diagnostics` issues usually don't have any `C-*` label. - - Also note `C-optimization` for missed compiler optimizations. + - `C-optimization` for missed compiler optimizations. + - `C-defective-hardware` for hardware bugs that are beyond our control. + - `C-external-bug` for software bugs that affect us but we don't have direct control over, but is worth tracking from our side. - [`O-*`]: For target-specific issues, specifies the compile target[^1] or compile target family (most notably the platform, i.e., the architecture or operating system). For example `O-macos`, `O-aarch64`, `O-windows`, `O-windows-msvc`. - [`A-*`]: The areas that the issue is relevant to, for example `A-linkage`, `A-patterns`, `A-diagnostics`. - [`L-*`]: When the issue concerns a specific lint. + - `L-false-positive` if the lint fires on a case that it should not have fired on. + - `L-false-negative` if the lint misses a case where it should have fired on. - [`F-*`]: When the issue concerns a specific (usually unstable, usually language) feature. - [`-Z*`]: When the issue concerns a specific unstable `-Z` compiler flag. - `requires-nightly`: This issue is not relevant to the stable compiler @@ -91,6 +95,7 @@ There are many different labels that can be applied to issues. should be closed in accordance with compiler [MCP 620](https://github.com/rust-lang/compiler-team/issues/620). - [`regression-*`]: Labels for tracking issues that are regressions. - [`D-*`]: Labels for diagnostic issues. + - `D-diagnostic-infra`: This issue is about the diagnostics infrastructure itself. - [`I-*`]: Different labels about the nature[^2] of a bug. For example ICE, slow code, heavy code (binary size), crashes, unsoundness. There are also some other `I-*` labels that don't really fit into this. For triaging, focus on `I-ICE`, `I-crash`, `I-hang`, `I-slow`, `I-heavy`, `I-compiletime` and `I-unsound`. - [`P-*`]: Priority labels. Applied using the [compiler prioritization procedure](../compiler/prioritization.md). @@ -99,6 +104,8 @@ There are many different labels that can be applied to issues. - `E-mentor`: A mentor is available to help with the issue, which makes for good first issues. - `E-needs-mcve`: This issue has a reproduction, but it is not minimal, it should be minimized. - `E-needs-bisection`: This issue needs a bisection, for example using [cargo-bisect-rustc](https://github.com/rust-lang/cargo-bisect-rustc). + - `E-needs-investigation`: This issue needs further investigation to determine root causes and the nature of the issue. + - `E-needs-design`: This issue will require some substantial design effort (exploration, prototyping, discussions, etc.). - `E-needs-test`: The issue has been fixed, but no test has been added for it. After someone adds a test, it can be closed. - `E-{easy,medium,hard}`: Someone has estimated how hard the issue is to fix. This can help with finding good first issues, but is [bound to be inaccurate](https://en.wikipedia.org/wiki/Curse_of_knowledge). From 82525f925bdeffbcdca31290c6a72568e4125f87 Mon Sep 17 00:00:00 2001 From: Jan David Date: Mon, 11 Nov 2024 10:47:50 +0100 Subject: [PATCH 35/38] Remove obsolete page for self-hosted runner The self-hosted runner has recently been decommissioned and removed from our infrastructure, so its documentation is no longer needed. --- src/SUMMARY.md | 1 - src/infra/docs/gha-self-hosted.md | 160 ------------------------------ 2 files changed, 161 deletions(-) delete mode 100644 src/infra/docs/gha-self-hosted.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 4fbdab43d..bb091172d 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -87,7 +87,6 @@ - [Bors](./infra/docs/bors.md) - [CDN](./infra/docs/cdn.md) - [Crater agents](./infra/docs/crater-agents.md) - - [Custom GitHub Actions runners](./infra/docs/gha-self-hosted.md) - [Dev Desktops](./infra/docs/dev-desktop.md) - [GitHub App for dev-desktops](./infra/docs/dev-desktop-github-app.md) - [Discord moderation bot](./infra/docs/discord-mods-bot.md) diff --git a/src/infra/docs/gha-self-hosted.md b/src/infra/docs/gha-self-hosted.md deleted file mode 100644 index 4fe4a6f39..000000000 --- a/src/infra/docs/gha-self-hosted.md +++ /dev/null @@ -1,160 +0,0 @@ -# Custom GitHub Actions runners - -The Infrastructure Team manages a pool of self-hosted GitHub Actions runners, -meant to be used by whitelisted repositories that need to run tests on -platforms not supported by the GitHub-hosted runners. We're currently running -the following machines: - -* `ci-arm-1.infra.rust-lang.org`: AArch64 runners, hosted on [packet] - ([configuration][host_vars-ci-arm-1]). - -The server configuration for the runners is managed with Ansible ([playbook], -[role]), and the source code for the tooling run on the server is in the -[gha-self-hosted] repository. - -Please get in touch with the Infrastructure Team if you need to run builds on -this pool for your project in the `rust-lang` organization. - -## Maintenance procedures - -### Updating the GitHub Actions runner version - -Our self-hosted CI runs on a [custom fork of the GitHub Actions -runner][rust-lang/gha-runner], which improves the security of the setup. The -fork needs to be manually rebased every time a new version comes out though, -and that needs to be done relatively quickly to prevent CI from -stopping[^self-updates]. - -Once a new release of [actions/runner] is out, clone [rust-lang/gha-runner] and -fetch the new tag pushed to the upstream repository. Then, rebase the changes -on top of the latest tag: - -```bash -git rebase --onto ${NEW_TAG} ${OLD_TAG} ${OLD_TAG}-rust${N} -``` - -For example, if the new tag is `v2.275.0`, the old tag is `v2.274.2` and there -were two releases of our fork, the command to execute would be: - -```bash -git rebase --onto v2.275.0 v2.274.2 v2.274.2-rust2 -``` - -The last commit to rebase *will* conflict, as that commit updates the version -number and the release notes. Add the `-rust1` suffix to the new version number -and remove the description of the changes from the changelog (keeping the -*"Fork of the GitHub Actions runner used by the Rust Infrastructure Team."* -sentence). Once the rebase is complete force-push the commits to `main`. - -After you force-push the new commits to `main` you're done! CI will create a -tag, build the release, upload it to GitHub Releases, and automatically push a -commit to [rust-lang/gha-self-hosted][gha-self-hosted] bumping the pinned -runner version to download in the images. The servers will then shortly pull -the latest changes, rebuild the images and restart idle VMs. - -[^self-updates]: The GitHub Actions runner really wants to self-update when a - new release is out, but such updates would prevent our security mitigations. - Because of that, one of the patches in our fork disable self-updates, but - that means the runner just stops working until it's updated. - -[rust-lang/gha-runner]: https://github.com/rust-lang/gha-runner -[actions/runner]: https://github.com/actions/runner - -### Changing the instances configuration - -The set of instances available in each host is configured through the -Ansible configuration located in the [simpleinfra repo][simpleinfra]: - -``` -ansible/envs/prod/host_vars/{hostname}.yml -``` - -You'll be able to add, remove and resize instances by changing that file and -applying the changes: - -``` -ansible/apply prod gha-self-hosted -``` - -### Forcing an update of the source code - -The server checks for source code updates every 15 minutes, but it's possible -to start such check in advance. You need to log into the machine you want to -act on, and run the following command: - -``` -sudo systemctl start gha-self-hosted-update -``` - -If the contents of the `images/` directory were changed, an image rebuild will -also be started. The new image will be used by each VM after they finish -processing the current job. - -### Forcing a rebuild of the images - -The server automatically rebuilds the images every week, but it's possible to -rebuild them in advance. You need to log into the machine you want to act on, -and run the following command: - -``` -sudo systemctl start gha-self-hosted-rebuild-image -``` - -### Managing the lifecycle of virtual machines - -Each virtual machine is assigned a name and its own systemd unit, called -`gha-vm-{name}.service`. For example, the `arm-1-1` VM is managed by the -`gha-vm-arm-1-1.service` systemd unit. You can stop, start and restart the -virtual machine by stopping, starting and restarting the systemd unit. - -Virtual machines are configured to restart after each build finishes. - -### Logging into the virtual machines - -It's possible to log into the virtual machines from localhost to debug builds. -This should be used as the last resort. Each VM binds SSH on a custom port on -the host (configured in the host Ansible configuration), and allows access to -the `manage` user (with password `password`). For example, to log into the VM -with port `2201` you can run: - -``` -ssh manage@localhost -p 2201 -``` - -Note that the VM image regenerates its own host key every time it boots, so -you'll likely get host key mismatch errors when connecting to a freshly booted -VM. - -### Accessing the out-of-band console for Packet servers - -In the event that a bare metal server hosted on Packet becomes unreachable but -is still marked as online, it's possible to access the out-of-band console over -the serial port to get a root shell. - -To access it, retrieve the root password configured on the server with: - -``` -aws ssm get-parameter --name /prod/ansible/HOSTNAME/root-password --with-decryption --query 'Parameter.Value' --output text -``` - -For example, to get the root password of `ci-arm-1`, run: - -``` -aws ssm get-parameter --name /prod/ansible/ci-arm-1/root-password --with-decryption --query 'Parameter.Value' --output text -``` - -Then, log into the [packet console][packet-console], navigate to the server -page and click the "out-of-band console" button at the top right: the SSH -command to use will be shown. Once you run the command you will be asked to -login on the server: use `root` as the username and the password you fetched -earlier as the password. - -To exit the out-of-band console, type a new line followed by `~.`. - -[gha-self-hosted]: https://github.com/rust-lang/gha-self-hosted -[host_vars-ci-arm-1]: https://github.com/rust-lang/simpleinfra/blob/master/ansible/envs/prod/host_vars/ci-arm-1.infra.rust-lang.org.yml -[packet]: https://www.packet.com -[packet-console]: https://app.packet.net -[playbook]: https://github.com/rust-lang/simpleinfra/blob/master/ansible/playbooks/gha-self-hosted.yml -[role]: https://github.com/rust-lang/simpleinfra/blob/master/ansible/playbooks/gha-self-hosted.yml -[simpleinfra]: https://github.com/rust-lang/simpleinfra From 60acd469bb377ffa17b9d6de736caa6d6f595623 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Le=C3=B3n=20Orell=20Valerian=20Liehr?= Date: Wed, 13 Nov 2024 15:19:37 +0100 Subject: [PATCH 36/38] Remove mention of label needs-triage-legacy which is no more --- src/release/issue-triaging.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/release/issue-triaging.md b/src/release/issue-triaging.md index 3b1e731e6..dffddf2bb 100644 --- a/src/release/issue-triaging.md +++ b/src/release/issue-triaging.md @@ -67,9 +67,6 @@ Another useful thing to do is go through `E-needs-mcve` and `E-needs-bisection` (using [cargo-bisect-rustc](`https://github.com/rust-lang/cargo-bisect-rustc`)). When you provide one, you can also remove the label using rustbot (`@rustbot label -E-needs-bisection`). -At the time of writing, there is also the `needs-triage-legacy` label, for older issues that are suspected to not have been triaged. -Triaging them the same way as `needs-triage` is also useful. - ## Labels There are many different labels that can be applied to issues. From 564a2a880da1f8dd5e7a032bd17b48c0c9dee4f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E8=AE=B8=E6=9D=B0=E5=8F=8B=20Jieyou=20Xu=20=28Joe=29?= <39484203+jieyouxu@users.noreply.github.com> Date: Wed, 13 Nov 2024 22:48:10 +0800 Subject: [PATCH 37/38] Remove outdated compiler-contributors team (#779) --- triagebot.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/triagebot.toml b/triagebot.toml index 765c38f29..cd82256f4 100644 --- a/triagebot.toml +++ b/triagebot.toml @@ -18,7 +18,7 @@ fallback = [ "/blacksmith" = ["@ehuss"] "/js" = ["@ehuss"] "/src/community" = ["community"] -"/src/compiler" = ["compiler", "compiler-contributors"] +"/src/compiler" = ["compiler"] "/src/crates-io" = ["crates-io"] "/src/docs-rs" = ["docs-rs"] "/src/governance" = ["ehuss"] From 757eb45d60bbcce1b608b8845d58663b30640067 Mon Sep 17 00:00:00 2001 From: Stanislav Ladyzhenskiy Date: Sun, 17 Nov 2024 18:53:51 +0700 Subject: [PATCH 38/38] Fix a link in process.md --- src/release/process.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/release/process.md b/src/release/process.md index 16c32abf4..475b93439 100644 --- a/src/release/process.md +++ b/src/release/process.md @@ -200,7 +200,7 @@ the blog and internals with the new information. to setup your environment and how to authenticate with AWS the first time you execute it. -[rust-lang/rust]: https://github.com/[rust-lang/rust] +[rust-lang/rust]: https://github.com/rust-lang/rust [rust-lang/release-team]: https://github.com/rust-lang/release-team [ripgrep]: https://github.com/burntsushi/ripgrep [`promote-release`]: https://github.com/rust-lang/promote-release