Add Plugins Support with WebAssembly #2056
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AmmarAbouZor
force-pushed
the
plugins_support
branch
from
a593d0e to
e7fb42c
Compare
AmmarAbouZor
force-pushed
the
plugins_support
branch
2 times, most recently
from
6e1387b to
1edaec3
Compare
AmmarAbouZor
force-pushed
the
plugins_support
branch
from
a42ff83 to
8809d46
Compare
AmmarAbouZor
force-pushed
the
plugins_support
branch
from
7c59290 to
ddde66e
Compare
AmmarAbouZor
force-pushed
the
plugins_support
branch
from
961d6a9 to
358b27b
Compare
AmmarAbouZor
force-pushed
the
plugins_support
branch
5 times, most recently
from
8fcc42b to
9941076
Compare
AmmarAbouZor
force-pushed
the
plugins_support
branch
from
74f20c8 to
194c439
Compare
*** These commits has been squashed for rebase *** *** Changes while rebase included too *** ******************************************************** Change parse function return type Parse function returns how many bytes have been consumed instead the rust of the slice to eliminate the reference and the lifetime Parse returns vector of results (in progress) - Change parse signature to return a vector of values instead of one item at a time. - Basic adjustments for producer logic to accept vector of parse results. - Adjust the current parsers temporally by wrapping their single return value inside a vector - Adjust parsers unit tests assuming that each parser will still return a vector of one value on each call. Plugin: Add simple temporary benchmarking code Assert parsing ends on first error temporarlly - Producer logic assumes that parsing will stop after encountering the first error. This check should be done in unit tests but for now it's done in code for now Plugin Host and API (in progress) - Create library for plugins host - Create wit file for parser on directory where plugins API crate will be initialized Parse return `impl IntoIterator` instead of `vec` - This can give more flexibility for other implementations of pares to return other types than vector - Parsers with a single return value can use `iter::onc` or an array to enable more compiler optimization on the code Move return statement in producer loop - Returning the results should happen after the parse iterator is finished. It was positioned at the end of the first iteration of parse call. Plugin: Wit files definitions & Prototype Clients - Wit files: - Use a directory for each version. - Use a separate interface for shared types. - Create wit file for bytesource - Small changes on parse wit file - Initialize rust lib and created place holder plugins Plugin: Initialize WASM host lib Plugins: Initialize WASM Engine Plugins: Provide general configuration for plugins - Parser and Bytesource plugins should get general chipmunk configuration besides their user-define configurations as string (or path to a file) - Currently the configurations structs are just palce-holders for chipmunk actual configs Plugin: Implementing Parser Host (in progress) - Create Parse Message type for parser plugins supporting only strings - Create Plugin Parser State, but its implementation is still missing - Bindgen Macro Plugin: Implementing Parser Host (in progress) - Implement Initialization of Parser Plugin - Error definitions for plugins - Move shared types and functionality to shared module - Move bindings module to their own file - WasmPlugin trait with basic implementation - Create Bytesoruce Plugin place holder with implementation for input types only Plugin: Parser Host & Versioning (in progress) - Move version specific parts to their own module to have one module for each version for the parts that can be changed - Define shared types and functions among different versions in their separate module - Provide Mapping between bindings types and shared types. - Create semantic version structure with parsing - Implement reading component meta data by loading and perform simple validation and version parsing to initiate the corresponding host version - Use std::OnceLock instead of tokio::OnceCell for creating wasm host since we don't need the function to be async. - Change error type `unsupported` in wit file to provide a custom message too. Plugin: Clippy Fixes Plugin Parser Host: Implement Parser Trait - Implement both parse functions using add method or returning a collection of results - Provide the needed mapping between guest and host results Plugin Parser: Add Unrecoverable Error type - Add Unrecoverable error to parse results to cover panics inside the plugins - Currently we log the error and print it to stderr and stop the parsing Plugins: Parser Integrations (in progress) - Introduce new parser type for the plugins with it's settings data type - Add Error Kind to the native errors for the plugins and map plugin initialization error to it. - Implement plugins parser Integrations in sessions export and observing - Make configurations path optional for parser and source plugins - Set `duplication_if_necessary` to true on parser host bindgen Integrate Plugin parser in CLI interactive tool - Option `plugin` is added to CLI interactive sessions - Plugin path will be defined via environment variables temporally. Plugins: Use interfaces in wit files Interfaces in wit files contain the current version meta data as well which is used to detect the plugin version to select the matching host version. Plugins: Implement missing mapping for parser host Plugin: Fix producer loop logic - Return was called from within the inner loop Producer Stream: Change return type to vector of values - Producer stream will yield all the available items at once instead of caching them Force using plugin parsers via env vars temporally Plugins: Parser Client Defs & Macro (in progress) - Define `Parser` trait which must be implemented by plugins authors - Implementing `export_parser` macro to be used by users - global paths for crate types still missing - Unit tests still missing
- Define `Parser` trait which must be implemented by plugins authors - Implementing `export_parser` macro to be used by users - global paths for crate types still missing - Unit tests still missing
- Define which types from bindings will be exported to the plugins implementers - Export types needed in export macros as hidden - Modify generate! Macro attributes to make its export macro public and set the bindings types for its needed types - Set all types withing export macros with their full path.
Apply fixes and changes for the parser client after trying to use it with real external plugin. Changes include: - Separate parser and bytesource with features to prevent generate! Macro from being called multiple times. - Move add function on the host to it's own interface that will be imported separately to avoid compilation errors because of duplicate definitions of types that occur when using the crate with separate projects as plugins only + Make the needed adjustments on the host side. - Make generated bindings public but hide them in documentations to because they are needed with export Macro - Move Prototyping code to test module temporally.
- Insure Parser trait from Chipmunk is in scope inside export macro so we can call parse method successfully. - Remove prototyping suppressing warning + Corresponding fixes
- Use the crate `TryBuild` to insure that the wrong patterns won't compile - Add successful test case when Parser trait is implemented. - Add failing tests for not implementing Parser Macro or for completely wrong input with trait or expression. - Ignore the comparison of compilation error messages for the tests since we still not sure if we gonna include this comparison on the future, because the comparison is text-based and can be broken on different Rust editions or different environments - Write script to run tests since they are behind inactive features
- Insure that one feature form Plugin-API crate at most is enabled at a time to provide the users with meaningful error message instead of the encrypted linking error if they enable many features at once since doing so will lead to linking errors on release builds only with very weird error messages
- Add test for the case when parser implementation and export macros are in different modules
- Internal modules now start with underscore to make it more clear for the users that they shouldn't use them - A warning have been added for the users if they explored the source code - This idea is taken from the crate 'Log'
- Add logging definitions and functions to wit file - Creating module on the client side to provide clean logging public API while handling log level complexity internally to direct avoid log calls to the host when the log level isn't allowed - Integrating the log level in the macro is still missing
- Provide custom logger that implements `log::Log` trait and register it using boxed logger. - This enables the users from using the general log macros from `log` crate and the message will be validated on the client side then sent to the host. - Added log level to the custom configurations for the parser in the wit file and implement it on the client side only for now.
- Use static logger instead of the boxed one because since the current max log level can be retrieved from log crate itself.
- Add unit tests for plugins parser - Change attributes and module name for prototyping code to make sure this wouldn't be compiled on any use case beside activating it manually with commenting out the attribute
- Implement logging on parser host. - Log Level will be always the current leg level in chipmunk since we use the same log functionality to log plugins' messages
Reexport log crate from chipmunk client crate because it's used in export macro, and the users can use it directly without having import their own version of the same library.
- Use sending parse results to the host instead of using the host add function by default since it's more readable and until know there is no noticeable performance difference between the two approaches
- Add configurations, attributes to show all the features in crate documentation, showing a hint for items available on features only. - Provide a script file to show the documentation with all needed attributes and environment variable to replicate what will be shown on docs.rs once the crate is published - Provide documentations and comments the implemented part of the crate
* Create a service for plugins in client front end and wire up the requests from the service to rust core though application and bindings * Debug code inside plugins manager tab to ensure all requests are working as expected.
Basic UI to show the infos of the plugins in pretty json format wit a reload button
AmmarAbouZor
force-pushed
the
plugins_support
branch
from
ea9a6f8 to
598a838
Compare
* Create typescript types and parse the json string directly in ts-bindings * Use files directly instead of directories for typescript types. * Rust: Set render options in a box for parser * Fix linting issues on typescript
AmmarAbouZor
force-pushed
the
plugins_support
branch
from
e22add1 to
fa45a79
Compare
* Create structure for byte-source inside platform and provide basic implementation with a lot of placeholders * Fix other places in code temporally and add TODO comments there as well. * Some comments about adding parser structure to platform to do next * Program can compile and run on the current state.
* Create new parser type for plugins and define new protocol for plugins as well. * Add parser plugin to all sources types. * Extend the code in client and holder temporally to make the code compile. * Add TODOs to all temporally fixes.
* Use two commands for parser plugin in menu one for files and the other for stream. * Started with implementing the command for stream. * Action for running stdout with plugins. * Postpone byte-source plugin command until we have a prototype for plugin parser
Placeholder for parser in session start view
* Add state for plugin parser configurations UI * Load plugins into a drop down and with reaction to selection changing
* Create modules and components for configuration schemas. * Currently only boolean and string are supported. * Saving the data is still missing. * Components are grouped in one module so it can be used in different views with different plugin types. * A lot of functions are place holders and will be extended or removed.
* Config Schemas are rendered on the screen * Binding the values to the configurations is still missing.
* Bind the value of UI controls to the configuration of the current session * Remove not needed ILC function from config controls. * Provide temp types for the value of the configurations.
Renderer still doesn't read the render info from the plugins.
* Parser plugins for commands is working directly from chipmunk without any workaround. * Change configurations of plugins in the front-end to make it compatible with rust types. This change include delivering the plugin binary path instead its directory and change naming to snake case * Ignore general setting of plugins on rust side temporally.
AmmarAbouZor
force-pushed
the
plugins_support
branch
from
215ba58 to
3a410a0
Compare
* Fix binding on text input control. * Remove saving configuration on Destroy because it's called after session request is sent. * Extend string plugin with prefix field to demonstrate configurations functionality * Demonstrate log and printing to std in parser plugin * Add more debug logs
AmmarAbouZor
force-pushed
the
plugins_support
branch
from
3a410a0 to
59e68b1
Compare
Basic resolving for merge conflicts but solution can't be compiled yet Conflicts: application/apps/indexer/Cargo.lock application/apps/indexer/Cargo.toml application/apps/indexer/indexer_cli/src/interactive.rs application/apps/indexer/session/Cargo.toml application/apps/indexer/session/src/events.rs application/apps/indexer/session/src/handlers/export_raw.rs application/apps/indexer/session/src/handlers/observing/file.rs application/apps/indexer/session/src/handlers/observing/mod.rs application/apps/indexer/session/src/unbound/commands/mod.rs application/apps/indexer/session/src/unbound/mod.rs application/apps/indexer/sources/src/factory.rs application/apps/indexer/sources/src/lib.rs application/apps/rustcore/rs-bindings/Cargo.lock application/apps/rustcore/ts-bindings/src/api/jobs.ts application/apps/rustcore/ts-bindings/src/native/native.jobs.ts application/platform/types/index.ts
* Move plugins types to shared types library `stypes` and extend the current types their with plugins. * Adjust the code in rust core to make it compile, moving some mapping implementations to proper positions in source code. * Write proptests for plugin types and extend proptests for current types after extending them with plugins types, making all proptests pass. * Todos in code to remove serializing to JSON. * Compile proptest library with full optimizations.
* Move more plugins types to shared types library `stypes` and extend the current types their with plugins. * Adjust the code in rust core to make it compile. * Write proptests for the new plugin types.
* Create type for plugins list to be used in communication instead of the serialized json string. * Improve proptests with built-in function for collections and options. * Add proptests for the new plugins list type.
AmmarAbouZor
force-pushed
the
plugins_support
branch
from
729168d to
a971c5e
Compare
* Add missing CommandOutcome with plugins list + proptests. * Use plugins list on rs-bindings instead of strings making it compile.
* Generate typescript types from rust types, and move them to platform, ensuring there is not changes in formatting anywhere. * Apply bincode adjustments on plugin functions in ts-bindings making it compile again. * Remove referencing manually generated types from platform and mark the file as to delete once the transform is finished.
* Adjust generating enum types on rust side to get more infos in typescript world. * Rename deprecated plugins types in typescript to get compiler errors where types needs to be replaced. * Finish the replacing the types in platform
* Generate encoding methods for plugin types. * Include plugin types in protocol test. * Adjust numbers types on wit and rust size to make it compatible with typescript ones.
* Integrating types in client and holder, making code compile, however we still get runtime errors because the generated types doesn't match the serialized ones at runtime.
* Removing code that generates tagged union types on typescript because those values aren't delivered at runtime with the current decoding implementation. * Making adjustments on the front-end with the changes making the front-end compile and work. * Set default for general parser setting on all features. * Added helper method to ensure exhaustive matching.
* Update wasmtime to version 28 and wit-bindgen to version 0.36. * Fix simple breaking changes on after the update, without experimenting with the new features.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR still works in progress...
It will provide a plugin system for Chipmunk using WASM with wasmtime runtime and the component model definitions.
The Communication API between Chipmunk and the plugins is defined via WIT files, and the plugins could be written basically in any language which supports compiling to web assembly and the component model proposal.
For Plugins written in Rust we will provide a crate which will take care of generating the bindings form the
witfiles and provides all the needed functionalities for development (like logging).The plugin system will offer writing plugins in the following areas:
Current State:
Host:
PluginsParserstruct on the host implements theParsertrait, abstracting plugin communication complexity.PluginsByteSourcestruct implements the standard library'sReadtrait and is passed toBinaryByteSourceto provide data as bytes.ParserTypeenum now includes plugin types with configuration settings. These configurations specify the plugin path, general settings for all parser plugins, and the path to the plugin's custom configuration file.Plugins:
.watfiles located in theplugins-apistand-alone crate..wat; instead, they can use theplugins-apilibrary, implement required traits, and export them with the provided macro.Next steps:
Parsertrait in Core to support custom delimiters, headers, and configurations, and deliver these values to the UI for rendering. (Consider making this change universal for all parsers in a separate PR.)