[new release] xdg, stdune, ordering, fiber, dyn, dune, dune-site, dune-rpc, dune-rpc-lwt, dune-private-libs, dune-glob, dune-configurator, dune-build-info and dune-action-plugin (3.0.0)

[rgrinberg]

XDG Base Directory Specification

• Project page: https://github.com/ocaml/dune
• Documentation: https://dune.readthedocs.io/

CHANGES:

• Remove uchar and seq dummy ocamlfind libraries from dune's builtin
  library database (Remove uchar and seq dummy modules dune#5260, @kit-ty-kate)

• Add a DUNE_DIFF_COMMAND environment variable to match --diff-command
  command-line parameter (@raphael-proust, fix environment variable for diff-command dune#5369, support DUNE_DIFF_COMMAND environment variable dune#5375)

• Add support for odoc-link rules (Rework odoc rules dune#5045, @lubegasimon)

• Dune will no longer generate documentation for hidden modules (Rework odoc rules dune#5045,
  @lubegasimon)

• Parse the native_pack_linker field of ocamlc -config (Parse the native_pack_linker field of ocamlc -config dune#5281, @TheLortex)

• Fix plugins with dot in the name (Dune sites plugin with dots dune#5182, @bobot, review @rgrinberg)

• Don't generate the dune-site build part when not needed (Don't generate the dune-site build part when not needed dune#4861, @bobot,
  review @kit-ty-kate)

• Fix installation of implementations of virtual libraries (Fix #3636 dune#5150, fix Missing -I when building against a virtual library from another project dune#3636,
  @rgrinberg)

• Run tests in all modes defined. Previously, jsoo was excluded. (@hhugo,
  Improve js_of_ocaml support dune#5049, fix Make the tests stanza setup runtest aliases for js dune#4951)

• Allow to configure the alias to run the jsoo tests (@hhugo, Improve js_of_ocaml support dune#5049, Allow to customize runtest aliases for inline_tests and tests stanza  dune#4999)

• Set jsoo compilation flags in the env stanza (@hhugo, Improve js_of_ocaml support dune#5049, Per environment flags with js_of_ocaml dune#1613)

• Allow to configure jsoo separate compilation in the env stanza. Previously,
  it was hard coded to always be enabled in the dev profile. (@hhugo, Improve js_of_ocaml support dune#5049,
  fix Make jsoo separate compilation independent of dev dune#970)

• Fix build-info version in jsoo executables (@hhugo, Improve js_of_ocaml support dune#5049, fix dune-build-info only works when public_name is specified, even if only mode is JS dune#4444)

• Pass -no-check-prims when building bytecode for jsoo (@hhugo, Improve js_of_ocaml support dune#5049, when building js_of_ocaml program: "while linking cmo, the external function [...] is not available" dune#4027)

• Fix jsoo builds when dynamically linked foreign archives are disabled
  (@hhugo, Improve js_of_ocaml support dune#5049)

• Disallow empty packages starting from 3.0. Empty packages may be
  re-enabled by adding the (allow_empty) to the package stanza in
  the dune-project file. (Make dune fail when trying to build empty packages (fixes #2882) dune#4867, fix Raise an error when building a defined but empty package dune#2882, @kit-ty-kate, @rgrinberg)

• Add link_flags field to the executable field of inline_tests (add link_flags field to inline_tests dune#5088,
  fix Linking C libraries for inline test runners dune#1530, @jvillard)

• In watch mode, use fsevents instead of fswatch on OSX (Polling mode - Native FSevents dune#4937, FSEvents: multiple watchers dune#4990, fixes
  Switch to FSEvents dune#4896 @rgrinberg)

• Remove inotifywait watch mode backend on Linux. We now use the inotify API
  exclusively (Remove inotifywait dune#4941, @rgrinberg)

• Report cycles between virtual libraries and their implementation (fix: better cycle detection for virtual libraries dune#5050,
  fixes Strange issue with virtual libraries, non-virtual module ends up with missing implementation dune#2896, @rgrinberg)

• Warn when lang versions have an ignored suffix. (lang dune 2.3.4) or (lang dune 2.3suffix) were silently parsed as 2.3 and we know suggest to remove
  the prefix. (Warn when lang versions have an ignored suffix dune#5040, @emillon)

• Allow users to specify dynamic dependencies in rules. For example (deps %{read:foo.gen}) (Get rid of the last remaning bits of "static evaluation" dune#4662, fixes Support for functions that compute deps in (rule (deps ..)) dune#4089, @jeremiedimino)

• Sandbox infer rules for menhir. Fixes possible "inconsistent assumptions"
  errors (fix(menhir): sandbox infer rule dune#5015, @rgrinberg)

• Experimental support for ctypes stubs (ctypes stanza dune#3905, fixes Rules for ctypes dune#135, @mbacarella)

• Fix interpretation of binaries defined in the env stanza. Binaries
  defined in x/dune wouldn't be visible in `x/*/**/dune. (Test that shows the environment binaries not properly set up for cram tests dune#4975, fixes Adding binaries into the PATH in a hierarchy doesn't work dune#4976,
  @Leonidas-from-XIV, @rgrinberg)

• Do not list private libraries in package listings (Hide private libraries in odoc listings dune#4945, fixes dune build @doc documents private libraries dune#4799,
  @rgrinberg)

• Allow spaces in cram test paths (Fix #4162 dune#4980, fixes Spaces in path -> cram tests -> failure dune#4162, @rgrinberg)

• Improve error handling of misbehaving cram scripts. (Improve Cram Eror Handling dune#4981, fix Poor error reporting in cram tests dune#4230,
  @rgrinberg)

• Fix foreign_stubs inside a tests stanza. Previously, dune would crash
  when this field was present (Fix #4936 dune#4942, fix No longer read installed META files for compiler libraries (-> fix "vmthreads only exists in OCaml < 4.09") dune#4946, @rgrinberg)

• Add the enabled_if field to inline_tests within the library stanza.
  This allows us to disable executing the inline tests while still allowing for
  compilation (Add enabled_if to running inline_tests dune#4939, @rgrinberg)

• Generate a dune-project when initializing projects with dune init proj ...
  (Add dune-project generation to dune init dune#4881, closes Make dune init proj create the dune-project file  dune#4367, @shonfeder)

• Allow spaces in the directory argument of the subdir stanza (Allow quoted strings in subdirs argument dune#4943, fixes
  3.0 bug : subdir does not support directories with spaces dune#4907, @rgrinberg)

• Add a %{toolchain} expansion variable (Add %{toolchain} variable dune#4899, fixes toolchain variable dune#3949, @rgrinberg)

• Include dependencies of executables when creating toplevels (either dune top or dune utop) (Support for executable-only projects for dune top/utop dune#4882, fixes Feature request: Support for executable-only projects for dune top/utop dune#4872, @Gopiancode)

• Fixes opam META file requires entry for private libs (Fix META generation for private libs. dune#4841, fixes Private packages not found in public libraries using private libraries dune#4839, @toots)

• Fixes dune exec not adding .exe on Windows (Test dune exec -- wrt file path, public_name with or without .exe dune#4371, fixes dune exec needs to add .exe on Windows dune#3322, @MisterDA)

• Allow multiple cinaps stanzas in the same directory (Simplify Dune_engine.Build_context dune#4460, @rgrinberg)

• Fix $ dune subst in empty git repositories (Fix #3619 dune#4441, fixes dune subst crashes when there is no commit dune#3619, @rgrinberg)

• Improve interpretation of ansi escape sequence when spawning processes (Add test for 2664 dune#4408,
  fixes Fix dependencies incorrectly dropped when specified with select dune#2665, @rgrinberg)

• Allow (package pkg) in dependencies even if pkg is an installed package
  (Fix dependency on an installed package dune#4170, @bobot)

• Allow %{version:pkg} to work for external packages (Add build_context field to context dune#4104, @kit-ty-kate)

• Add (glob_files_rec <dir>/<glob>) for globbing files recursively (Add glob_files_rec dune#4176,
  @jeremiedimino)

• Automatically generate empty .mli files for executables and tests (Generate empty interfaces for executables and tests dune#3768,
  fixes Implicit empty .mli files for executable modules dune#3745, @craigfe)

• Add ocaml command subgroup for OCaml related commands such as utop, top,
  and merlin (Introduce dune ocaml command group dune#3936, @rgrinberg).

• Detect unknown variables more eagerly (Resolve variable names at parsing time dune#4184, @jeremiedimino)

• Improve location of variables and macros in error messages (Improve location of percent forms dune#4205,
  @jeremiedimino)

• Auto-detect dune-project files as dune files in Emacs (Detect auto-mode for dune-project files in emacs dune#4222, @shonfeder)

• Dune no longer automatically create or edit dune-project files
  (No longer edit or create dune-project files dune#4239, fixes Remove support for creating and editing the dune-project file automatically dune#4108, @jeremiedimino)

• Warn if dune-project is not found (fatal in release mode) (Warn when dune-project does not exist dune#5343, @emillon)

• Cleanup temporary files after running $ dune exec. (Run exit hooks before Unix.exec'ing dune#4260, fixes dune exec leaves empty directory in /tmp/ dune#4243,
  @rgrinberg)

• Add a new subcommand dune ocaml dump-dot-merlin that prints a mix of all the
  merlin configuration of a directory (defaulting to the current directory) in
  the Merlin configuration syntax. ([merlin] Add option to dune ocaml-merlin to dump generic config in dot merlin syntax dune#4250, @voodoos)

• Enable cram tests by default (Enable cram by default starting with 3.0 dune#4262, @rgrinberg)

• Drop support for opam 1.x (Drop support for opam1 dune#4280, @jeremiedimino)

• Stop calling ocamlfind to determine the library search path or
  library installation directory. This makes the behavior of Dune
  simpler and more reproducible (Stop calling ocamlfind to determine the library search/installation directory dune#4281, @jeremiedimino)

• Remove the external-lib-deps command. This command was only
  approximative and the cost of maintainance was getting too high. We
  removed it to make room for new more important features (Remove the external-lib-deps command dune#4298,
  @jeremiedimino)

• It is now possible to define action dependencies through a chain
  of aliases. (get rid of alias stamp files and compute alias expansions dune#4303, @aalekseyev)

• If an .ml file is not used by an executable, Dune no longer report
  parsing error in this file (Narrow ml dependencies of executables dune#4330, @jeremiedimino)

• Add support for sandboxing using hard links (Add support for hard links dune#4360, Andrey Mokhov)

• Fix dune crash when subdir is an absolute path (Fix a dune crash when subdir is an absolute path dune#4366, @anmonteiro)

• Changed the implementation of actions attached to aliases, as in
  (rule (alias runtest) (action (run ./test))). A visible result for
  users is that such actions are now memoized for longer. For
  instance:

  $ echo '(rule (alias runtest) (action (echo "X=%{env:X=0}\n")))` > dune
  $ X=1 dune runtest
  X=1
  $ X=2 dune runtest
  X=2
  $ X=1 dune runtest
  

  Previously, Dune would have re-executed the action again at the last
  line. Now it remembers the result of the first execution.

• Fix a bug where dune would always re-run all actions that produce symlinks,
  even if their dependencies did not change. (Cache actions that generate symlinks dune#4405, @aalekseyev)

• Fix a bug that was causing Dune to re-hash generated files more
  often than necessary (Re-hash generated files less often dune#4419, @jeremiedimino)

• Fields allowed in the config file are now also allowed in the
  workspace file (Allow to set config parameters via the workspace file dune#4426, @jeremiedimino)

• Add options to control how Dune should handle stdout and stderr of
  actions when then succeed. It is now possible to ask Dune to ignore
  the stdout of actions when they succeed or to request that the
  stderr of actions must be empty. This allows to reduce the noise of
  large builds (swallow stdout on success dune#4422, Add actions-stdxxx-on-success options dune#4515, @jeremiedimino)

• The @all alias no longer depends directly on copies of files from the source
  directory (@all: do not depend directly on file copies from source files dune#4461, @nojb)

• Allow dune-file as an alternative file name for dune files (needs to be
  enabled in the dune-project file) (Accept dune-file as dune file name dune#4428, @nojb)

• Drop support for upgrading jbuilder projects (Drop support for upgrading jbuilder projects dune#4473, @jeremiedimino)

• Extend the environment variable BUILD_PATH_PREFIX_MAP to rewrite
  the root of the build dir (or sandbox) to /workspace_root (Extend BUILD_PATH_PREFIX_MAP with the workspace root dune#4466,
  @jeremiedimino)

• Simplify the implementation of build cache. We stop using the cache daemon to
  access the cache and instead write to and read from it directly. The new cache
  implementation is based on Jenga's cache library, which was thoroughly tested
  on large-scale builds. Using Jenga's cache library will also make it easier
  for us to port Jenga's cloud cache to Dune. (Add a new implementation of the shared cache and start using it dune#4443, Remove the old cache library and cache daemon dune#4465, Andrey Mokhov)

• More informative error message when Dune can't read a target that's supposed
  to be produced by the action. Old message is still produced on ENOENT, but other
  errors deserve a more detailed report. (Improve error messages on unreadable targets dune#4501, @aalekseyev)

• Fixed a bug where a sandboxed action would fail if it declares no dependencies in
  its initial working directory or any directory it chdirs into. (mkdir the initial working directory in sandbox dune#4509, @aalekseyev)

• Fix a crash when clearing temporary directories (dune crash when deleting temp files dune#4489, Do not fail due to races when deleting directories dune#4529, Andrey Mokhov)

• Dune now memoizes all errors when running in the file-watching mode. This
  speeds up incremental rebuilds but may be inconvenient in rare cases, e.g. if
  a build action fails due to a spurious error, such as running out of memory.
  Right now, the only way to force such actions to be rebuilt is to restart
  Dune, which clears all memoized errors. In future, we would like to provide a
  way to rerun all actions failed due to errors without restarting the build,
  e.g. via a Dune RPC call. (Start caching errors in Memo dune#4522, Andrey Mokhov)

• Remove dune compute. It was broken and unused (Remove the "dune compute" command dune#4540,
  @jeremiedimino)

• No longer generate an approximate merlin files when computing the
  ocaml flags fails, for instance because they include the contents of
  a file that failed to build. This was a niche feature and it was
  getting in the way of making Dune's core better. (No longer generate approximate merlin files dune#4607, @jeremiedimino)

• Make Dune display the progress indicator in all output modes except quiet
  (verbose output with status line dune#4618, @aalekseyev)

• Report accurate process timing information in trace mode (enabled with
  --trace-file) (Use wait3 to accurately time spawned processes dune#4517, @rgrinberg)

• Do not log live_words and free_words in trace file. This allows using
  Gc.quick_stat which does not scan the heap. (Do not log live_words and free_words in trace file dune#4643, @emillon)

• Don't let command run by Dune observe the environment variable
  INSIDE_EMACS in order to improve reproducibility (Don't let command run by Dune observe INSIDE_EMACS dune#4680,
  @jeremiedimino)

• Fix root_module when used in public libraries (Add a reproduction case for issue #4684 dune#4685, fixes Using (root_module ...) with a public library results in a build failure dune#4684,
  @rgrinberg, @craigfe)

• Fix root_module when used with preprocessing (Add a reproduction case for issue #4682 dune#4683, fixes Using (root_module ...) with a PPX results in a build failure dune#4682,
  @rgrinberg, @craigfe)

• Display Coq profile flags in dune printenv ([coq] Display coq env flags in dune printenv dune#4767, @ejgallego)

• Introduce mdx stanza 0.2, requiring mdx >= 1.9.0, with a new generic deps
  field and the possibility to statically link libraries in the test
  executable. (Mdx stanza rework dune#3956, mdx stanza: record env var dependency dune#5391, fixes [RFC] mdx stanza rework dune#3955)

• Improve lookup of optional or disabled binaries. Previously, we'd treat every
  executable with missing libraries as optional. Now, we treat make sure to
  look at the library's optional or enabled_if status (Do not change set of binaries based on optional dune#4786).

• Always use 7 char hash prefix in build info version (Specify length of git hash prefix used in build info version dune#4857, @jberdine, fixes
  git hash in build-info version is not as stable as it could be dune#4855)

• Allow to explicitly disable/enable the use of dune subst by adding a
  new (subst <disable|enable>) stanza to the dune-project file.
  (dune-project: Add the (subst disabled/enabled) stanza dune#4864, @kit-ty-kate)

• Simplify the way dune discovers the root of the workspace. It now
  stops at the first dune-workspace file it encounters, and fails if
  it finds neither a dune-workspace nor a dune-project file
  (Simplify root detection + require a dune-project or dune-workspace file to exist dune#4921, fixes Require a dune-project or dune-workspace file to exist dune#4459, @jeremiedimino)

• Dune no longer reads installed META files for libraries distributed with the
  compiler, instead using its own internal database. (No longer read installed META files for compiler libraries (-> fix "vmthreads only exists in OCaml < 4.09") dune#4946, @nojb)

• Add support for (empty_module_interface_if_absent) in executable and library
  stanzas. (Add support for (empty_module_interface_if_absent) dune#4955, @nojb)

• Add support for %{bin-available:...} (Add support for %{bin-available:...} dune#4995, @jeremiedimino)

• Make sure running git or hg in a sandboxed action, such as a
  cram test cannot escape the sandbox and pick up some random git or
  mercurial repository on the file system (Bump Dune's own (lang dune ...) to 3.0 dune#4996, @jeremiedimino)

• Allow %{read:...} in more places such as (enabled_if ...)
  (Allow %{read:...} in more places such as enabled_if dune#4994, @jeremiedimino)

• Run each action in its own process group so that we don't leave
  stray processes behind when killing actions (Run each action in its own process group dune#4998, @jeremiedimino)

• Add an option expand_aliases_in_sandbox (Disable expansion of alises in sandboxes by default, and add an option for it dune#5003, @jeremiedimino)

• Allow to cancel the initial scan via Control+C (Simplify Dune_engine.Build_context dune#4460, fixes allow to exit dune via signals (e.g. ^C) during the initial scan dune#4364
  @jeremiedimino)

• Add experimental support for directory targets ([RFC] Formalise support for directory targets dune#3316, Add basic support for directory targets dune#5025, Andrey Mokhov),
  enabled via (using directory-targets 0.1) in dune-project.

• Delete old promote-into, promote-until-clean and promote-until-clean-into
  syntax (Delete [promote-foo] syntax in favour of [promote foo] dune#5091, Andrey Mokhov).

• Add link_flags in the env stanza (Allow to configure link_flags in env dune#5215)

• Bootstrap: ignore errors when trying to remove generated files. (bootstrap.ml: ignore errors when trying to remove generated files dune#5407,
  @damiendoligez)

[smorimoto]

dune-build-info and xdg have the following error:

#=== ERROR while compiling dune-build-info.3.0.0 ==============================#
# context              2.0.10 | linux/x86_64 | ocaml-base-compiler.4.02.3 | pinned(https://github.com/ocaml/dune/releases/download/3.0.0/fiber-3.0.0.tbz)
# path                 ~/.opam/4.02/.opam-switch/build/dune-build-info.3.0.0
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune build -p dune-build-info -j 31 @install
# exit-code            1
# env-file             ~/.opam/log/dune-build-info-2610-d2d111.env
# output-file          ~/.opam/log/dune-build-info-2610-d2d111.out
### output ###
# File "otherlibs/build-info/src/.build_info.objs/byte/_unknown_", line 1, characters 0-0:
# (cd _build/default && /home/opam/.opam/4.02/bin/ocamlc.opt -w -40 -alert -unstable -g -bin-annot -I otherlibs/build-info/src/.build_info.objs/byte -no-alias-deps -open Build_info__ -o otherlibs/build-info/src/.build_info.objs/byte/build_info__Build_info_data.cmi -c -intf otherlibs/build-info/src/build_info_data.mli)
# /home/opam/.opam/4.02/bin/ocamlc.opt: unknown option '-alert'.

I think it is related to ocaml/dune#5245, does anyone have any ideas?

[rgrinberg]

We can just put constraints and move on. If someone is still using 4.02, using this library is the least of their concerns. Rudi.

…

On Feb 13, 2022, 9:59 AM -0700, Sora Morimoto ***@***.***>, wrote: dune-build-info and xdg have the following error: #=== ERROR while compiling dune-build-info.3.0.0 ==============================# # context 2.0.10 | linux/x86_64 | ocaml-base-compiler.4.02.3 | pinned(https://github.com/ocaml/dune/releases/download/3.0.0/fiber-3.0.0.tbz) # path ~/.opam/4.02/.opam-switch/build/dune-build-info.3.0.0 # command ~/.opam/opam-init/hooks/sandbox.sh build dune build -p dune-build-info -j 31 @install # exit-code 1 # env-file ~/.opam/log/dune-build-info-2610-d2d111.env # output-file ~/.opam/log/dune-build-info-2610-d2d111.out ### output ### # File "otherlibs/build-info/src/.build_info.objs/byte/_unknown_", line 1, characters 0-0: # (cd _build/default && /home/opam/.opam/4.02/bin/ocamlc.opt -w -40 -alert -unstable -g -bin-annot -I otherlibs/build-info/src/.build_info.objs/byte -no-alias-deps -open Build_info__ -o otherlibs/build-info/src/.build_info.objs/byte/build_info__Build_info_data.cmi -c -intf otherlibs/build-info/src/build_info_data.mli) # /home/opam/.opam/4.02/bin/ocamlc.opt: unknown option '-alert'. I think it is related to ocaml/dune#5245, does anyone have any ideas? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[smorimoto]

As this happens with OCaml compilers prior to 4.08, I think it's necessary to set the constraint to make it available only above 4.08, but won't that be a problem for people? If not, we can certainly move forward with it.

[rgrinberg]

A 4.08 minimum for these libraries is acceptable to me. If it causes problems, we can always revisit. Rudi.

…

On Feb 13, 2022, 10:32 AM -0700, Sora Morimoto ***@***.***>, wrote: As this happens with OCaml compilers prior to 4.08, I think it's necessary to set the constraint to make it available only above 4.08, but won't that be a problem for people? If not, we can certainly move forward with it. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[smorimoto]

Indeed. I just opened a PR for that on the dune side.

[smorimoto]

Should we move the 3.0.0 tag? Or just publish 3.0.1 instead?

[rgrinberg]

We don't re-publish tags anymore. I could publish 3.0.1 or this PR can be updated with the constraints as well. Rudi.

…

On Feb 14, 2022, 5:48 PM -0700, Sora Morimoto ***@***.***>, wrote: Should we move the 3.0.0 tag? Or just publish 3.0.1 instead? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[dinosaure]

It seems that cram tests is broken on ocamlformat.0.{19,20,21}.0. I can reproduce it locally. One of this error is:

Warning: Ocamlformat disabled because [--enable-outside-detected-project] is not set and no [.ocamlformat] was found within the project (root: ../../../../../.sandbox)

It's fixed when we add into tests the option --enable-outside-detected-project with ocamlformat. (/cc @gpetiot)

[dinosaure]

It seems that we are not able to dune build -p iter @runtest anymore (iter.1.3.0 & iter.1.2.1) (/cc @c-cube). The error is related about the %{deps} variable which is not authorized anymore into the (diff? (is it expected @rgrinberg?). I did this patch which fix the @runtest:

diff --git a/dune b/dune
index 59adf47..7496755 100644
--- a/dune
+++ b/dune
@@ -1,8 +1,8 @@
 
 (alias
  (name runtest)
- (deps README.md)
+ (deps (:readme README.md))
  (action (progn
-          (run ocaml-mdx test %{deps})
-          (diff? %{deps} %{deps}.corrected))))
+          (run ocaml-mdx test %{readme})
+          (diff? %{readme} %{readme}.corrected))))
 

EDIT: the same issue remains for lwt-pipe.0.1
EDIT: the same issue remains for printbox.0.2 & printbox-text.0.6

[dinosaure]

It seems that dune build -p ppx_blob @runtest does not work anymore (/cc @johnwhitington) (hacl_x25519 has the same error but the project is archived). It seems that we need to put the full-path (relative to _build/default) of the file instead of the filename now. A patch like this one fixes the issue:

diff --git a/test/test.ml b/test/test.ml
index 8569593..e74747d 100644
--- a/test/test.ml
+++ b/test/test.ml
@@ -1,6 +1,6 @@
 let suite = [
   ("path relative to source file", `Quick, fun () ->
-    Alcotest.(check string) "file contents" "foo\n" [%blob "test_file"]
+    Alcotest.(check string) "file contents" "foo\n" [%blob "test/test_file"]
   );
 ]
 

[kit-ty-kate]

(previous CI logs for reference: https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/3967e6d85aef5c4001cff44a5e1f2d3fddd0587a)

[johnwhitington]

cc Bertrand Bonnefoy-Claudet @bbc2 the most recent committer (maintainer?) of pxx_blob. I'm afraid it's been years since I touched it or understood PPX.

[kit-ty-kate]

@rgrinberg this looks rather peculiar as well:

#=== ERROR while compiling ojs_base.0.7.0 =====================================#
# context              2.0.10 | linux/x86_64 | ocaml-base-compiler.4.13.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.13/.opam-switch/build/ojs_base.0.7.0
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune build -p ojs_base -j 31 --promote-install-files=false @install
# exit-code            1
# env-file             ~/.opam/log/ojs_base-24569-e7966f.env
# output-file          ~/.opam/log/ojs_base-24569-e7966f.out
### output ###
# File "server/dune", line 7, characters 13-36:
# 7 |  (preprocess (pps xtmpl_ppx lwt_ppx))
#                  ^^^^^^^^^^^^^^^^^^^^^^^
# (cd _build/default && .ppx/0e52ba3df38a95b1b52a2bc615777cd5/ppx.exe --cookie 'library-name="ojs_server"' -o server/tmpl.pp.ml --impl server/tmpl.ml -corrected-suffix .ppx-corrected -diff-cmd - -dump-ast)
# Fatal error: exception Not_found

[rgrinberg]

@rgrinberg this looks rather peculiar as well:

#=== ERROR while compiling ojs_base.0.7.0 =====================================#
# context              2.0.10 | linux/x86_64 | ocaml-base-compiler.4.13.1 | file:///home/opam/opam-repository
# path                 ~/.opam/4.13/.opam-switch/build/ojs_base.0.7.0
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune build -p ojs_base -j 31 --promote-install-files=false @install
# exit-code            1
# env-file             ~/.opam/log/ojs_base-24569-e7966f.env
# output-file          ~/.opam/log/ojs_base-24569-e7966f.out
### output ###
# File "server/dune", line 7, characters 13-36:
# 7 |  (preprocess (pps xtmpl_ppx lwt_ppx))
#                  ^^^^^^^^^^^^^^^^^^^^^^^
# (cd _build/default && .ppx/0e52ba3df38a95b1b52a2bc615777cd5/ppx.exe --cookie 'library-name="ojs_server"' -o server/tmpl.pp.ml --impl server/tmpl.ml -corrected-suffix .ppx-corrected -diff-cmd - -dump-ast)
# Fatal error: exception Not_found

I just confirmed this issue with 2.9.x as well. It seems like an odd bug though so I will have a look. It's not related to 3.0 however.

[rgrinberg]

EDIT: I made a mistake when testing this, this is indeed a 3.0 issue. I'm investigating.

Here's the full trace. This issue is ppxlib related. cc @pitag-ha rings a bell perhaps? I'd also suggest not to leak Not_found exceptions in the public API

File "server/dune", line 7, characters 13-36:
7 |  (preprocess (pps xtmpl_ppx lwt_ppx))
                 ^^^^^^^^^^^^^^^^^^^^^^^
Fatal error: exception Not_found
Raised at Stdlib__List.find in file "list.ml", line 224, characters 10-25
Called from Xtmpl_ppx__Ppx_xtmpl.expand_xtmpl in file "ppx/ppx_xtmpl.ml", line 325, characters 15-32
Called from Ppxlib__Ast_pattern_generated.pconst_string.(fun) in file "src/ast_pattern_generated.ml", line 612, characters 27-42
Called from Ppxlib__Ast_pattern_generated.pstr_eval.(fun) in file "src/ast_pattern_generated.ml", line 2697, characters 27-42
Called from Ppxlib__Ast_pattern.(^::).(fun) in file "src/ast_pattern.ml", line 100, characters 18-33
Called from Ppxlib__Ast_pattern.map_result.(fun) in file "src/ast_pattern.ml", line 163, characters 53-71
Called from Ppxlib__Ast_pattern.parse in file "src/ast_pattern.ml", line 11, characters 6-31
Called from Ppxlib__Extension.For_context.convert in file "src/extension.ml", line 247, characters 14-68
Called from Ppxlib__Context_free.map_node in file "src/context_free.ml", line 220, characters 12-46
Called from Ppxlib__Context_free.map_top_down#expression in file "src/context_free.ml", line 457, characters 12-111
Called from Ppxlib__Ast_traverse.map_with_expansion_context#value_binding in file "src/ast_traverse.ml", line 140, characters 21-61
Called from Stdlib__List.map in file "list.ml", line 92, characters 20-23
Called from Ppxlib_ast__Ast.map_with_context#structure_item_desc in file "ast/ast.ml", line 6717, characters 20-54
Called from Ppxlib_ast__Ast.map_with_context#structure_item in file "ast/ast.ml", line 6703, characters 24-62
Called from Ppxlib__Context_free.map_top_down#structure.loop in file "src/context_free.ml", line 618, characters 36-71
Called from Ppxlib__Context_free.map_top_down#structure.loop in file "src/context_free.ml", line 668, characters 31-60
Called from Ppxlib__Driver.Transform.merge_into_generic_mappers.map_impl in file "src/driver.ml", line 247, characters 17-43
Called from Ppxlib__Driver.apply_transforms.(fun) in file "src/driver.ml", line 523, characters 20-28
Called from Stdlib__List.fold_left in file "list.ml", line 121, characters 24-34
Called from Ppxlib__Driver.apply_transforms in file "src/driver.ml", line 505, characters 4-953
Called from Ppxlib__Driver.map_structure_gen in file "src/driver.ml", line 566, characters 4-259
Called from Ppxlib__Driver.process_ast in file "src/driver.ml", line 928, characters 8-112
Called from Ppxlib__Driver.process_file in file "src/driver.ml", line 969, characters 15-80
Called from Ppxlib__Driver.standalone in file "src/driver.ml", line 1391, characters 9-27
Re-raised at Location.report_exception.loop in file "parsing/location.ml", line 932, characters 14-25
Called from Ppxlib__Driver.standalone in file "src/driver.ml", line 1394, characters 4-59
Called from Dune__exe___ppx in file ".ppx/0e52ba3df38a95b1b52a2bc615777cd5/_ppx.ml-gen", line 1, characters 9-36

[pitag-ha]

The problem in xtmpl_ppx seems strongly related to the problem in ppx_blob, doesn't it? (the Not_found seems to be triggered by an unsuccessful search for a file: https://framagit.org/zoggy/xtmpl/-/blob/master/ppx/ppx_xtmpl.ml#L72)

rings a bell perhaps?

It very broadly rings a bell from an issue reporting a regression in ppxlib: ppxlib passes the path of the current module to the rewriter and it seems like the latest ppxlib version messes up that path in certain circumstances. It doesn't seem to be the same flawful pattern as in the path in the ppx_blob test though. Do you think that can be related anyways?

[pitag-ha]

I'd also suggest not to leak Not_found exceptions in the public API

What are you referring to? That ppx rewriters such as xtmpl_ppx shoudln't leak Not_found exceptions?

[rgrinberg]

Yes, I thought the exception was raised by ppxlib, but it's raised by the rewriter in question. So indeed it should be the rewriter that should be fixed not to emit Not_found.

[bbc2]

About ppx_blob, I would be happy to push a fix and make a release. @dinosaure's suggested fix looks good to me although my (recompiled) ocamllsp doesn't seem to like the new path (unlike the new Dune).

In such a situation, would making a new version of ppx_blob with a constraint like dune >= 3.0.0 make sense? Or is there a smarter way to do that?

[rgrinberg]

Okay, I understand the ppx issue now. The culprit is this PR: ocaml/dune#4466

The issue is that once dune sets BUILD_PATH_PREFIX_MAP, compiler-libs will now mangle paths. The result is that the returned value of Ocaml_common.Location.absolute_path is always junk.

Looking up artifacts this way sounds wrong to me, but we shouuld try and keep old builds working so I will disable this behavior for projects pre 3.0

[rgrinberg]

@kit-ty-kate i don't understand this change, we don't use effects at all in dune. How are we incompatible with it?

[kit-ty-kate]

base-effects is a package that depends on a compiler with the "effect syntax" so everytime a project uses e.g.

let effect = ... in

it will fail with the effect syntax.

[rgrinberg]

Okay, so effect is now a keyword. I think I can just rename our uses of effect then.

[kit-ty-kate]

As you wish but this is basically OCaml 6.0 and the proposal hasn’t really been made yet. It only lives in opam-repository as ocaml-variants.4.12.0+domains+effects

[rgrinberg]

@kit-ty-kate where did this constraint come from? I think I can actually just remove this dependency. We don't use the result package at all anymore.

[rgrinberg]

Oh never mind. We require it because of Lwt.

[kit-ty-kate]

This one was just a guess, given anything below 1.5 doesn’t have compatible types with the standard library (Result.t != Stdlib.Result.t)

[kit-ty-kate]

Note that it won’t be the case in Lwt 5.6.0: ocsigen/lwt#925

[kit-ty-kate]

Superseded by #20758