Explore unknowns

[alandefreitas]

Based on the internal discussion held on October 21st, the project will concentrate on the following areas of uncertainty:

• Mrdocs-as-compiler Explore mrdocs as compiler mode #1073
• Scripting extensions Explore Scripting extensions #1128 Explore Javascript Engines #881
• Plugins Add plugins/generators via DLL #58 define how the plugins are found #1044
• JSON-only mrdocs
• Reflection Explore reflection #1114
• Cross-linking support for reading tagfiles #1072

This issue serves as a reference for the list and a platform for more lasting discussions about these topics.

[alandefreitas]

Update: The idea of a MrDocs version that only outputs JSON has been discussed internally this week. I'm adding this to the list of unknowns because it's independent from the idea of MrDocs as a compiler, even though Peter Dimov proposed both at the same time.

This unknown is similar to #678, but the pros and cons are inevitably different because some of the cons in #678 no longer apply, and we have other unknowns that could change the incentives here.

Also, there's some urgency to discussing this one because it completely changes the meaning of the unknown regarding extensions.

[alandefreitas]

Update

@vinniefalco @mizvekov @gennaroprota

After investigating these unknowns for a little longer, here's my assessment of each unknown, ordered by their blocking relationship (and therefore priority):

• JSON-only mrdocs: It makes sense, but it depends on how extensions go. Tools can focus on what they're good at. Extensions become much more powerful post-hooks. Extensions can do anything we think of without bloating the binary. It's much easier to find devs for these extensions because they don't even need to know Clang/LLVM (though they do need to know some basic C++). Modularization/encapsulation improves. Different steps would have their own tests. The performance cons in Should Mr. Docs be XML-only? #678 also no longer apply. It enforces good internal boundaries and encapsulation. If we want to keep the XML generator or even create a binary generator (as it would be useful for a mrdocs-as-compiler feature), that's also fine. Although we're calling this "JSON-only mrdocs", the relevant feature is actually "serializable-structured-data-formats-that-are-low-effort-to-maintain-only mrdocs".
• Mrdocs-as-compiler: This is a good idea if it is seen as a modularization or encapsulation effort to improve MrDocs and, consequently, give users the power to control these internal parts of the process. This is bad idea if we hope we're going to trick CMake into thinking it's a compiler, that we can get rid of the mrdocs "coordinator" tool, or that this will make the user workflow simpler. This didn't work for clang tools (because they're not really a compiler at the end of the day) and won't work for us either. I left a longer comment in Explore mrdocs as compiler mode #1073 (comment), and @mizvekov also made good points about that internally.
• Scripting extensions: What we want to do here depends a lot on what we decide for the two unknowns above. If we have JSON-only mrdocs or mrdocs-as-a-compiler, the extensions (and effectively all generators) become a post-hook event in the workflow. Then the user can use Node, Python, or whatever they want. On the other hand, if the extension engines are bundled into MrDocs, we have different issues to consider. For instance, duktape doesn't provide the features we need. Then I investigated the alternatives and implemented jerryscript in feat: javascript helpers extension #1126. It took me weeks to get this right (for instance, jerryscript doesn't allow more than 1 interpreter instance at a time unless you customize the code extensively - because the maintainer is only concerned with his own use case) and the result is still not great. Every interpreter is either not supposed to be bundled at all (and that is not even desirable, since bundling something like Node.js would make the binary hundreds of MB larger) or it only provides a precarious, poorly maintained, limited set of features that people will complain about. JSON-only mrdocs or mrdocs-as-a-compiler also make things simpler in the sense that we don't have to reinvent the wheel: we can use real handlebars.js instead of alanbars, and so on.
• Plugins: I think the biggest question here is whether plugins are supposed to achieve the same as an extension (but in C++) or something else. In any case, it's impossible to have an informed opinion on this before we decide what to do with the other unknowns.
• Reflection: An uncontroversial technical feature that makes everything above much easier to implement and maintain. By "much easier," I literally mean at least 10x less time spent on any feature issue. Having to manually update multiple competing sources of truth for these data structures (sources, generators, binaries, docs, schemas) is significantly slowing us down and incentivizing people to leave bugs behind (it's much easier and faster to "forget" to update the XML generator and schema no one is using outside tests than to go through multiple rounds of CI until getting this right).
• Cross-linking: An uncontroversial technical feature that is unrelated to everything above.

[alandefreitas]

For reference, this is what I currently understand as an extension: a step between the extraction and the generator that manipulates the corpus, such that:

1. Every kind of manipulation in the corpus is allowed. Nothing would fail in the extension: the user can create new fields for a corpus symbol, remove fields, change symbol types, merge symbols, split symbols, etc.
2. The user would also be able to use system libraries and every feature of the language. For instance, if the chosen language is JS, they can use the fs module to manipulate files, use whatever async features they want, and structure their extension with whatever directory layout they want.

Is this a good description of what an extension is? Are there more requirements? I think there was a requirement related to exposing MrDocs algorithms. That’s still a little unclear to me.

[alandefreitas]

Update after internal discussions:

• Mrdocs-as-compiler: No longer going to be pursued.
• Scripting extensions: Will be executed between extraction and the generator. Will provide scripts with objects that maintain corpus invariants.
• Plugins: Will be executed at the beginning of the process. A "main" plugin function should be called to manipulate MrDocs and register new generators, etc.
• JSON -only mrdocs: It will be one more generator. This should be a plugin, but it can be implemented as a regular generator if the plugin infrastructure is not ready yet; it can be converted to a plugin later.
• Reflection: Remains reasonably uncontroversial.
• Cross-linking: The technical issue remains reasonably uncontroversial, but we have to discuss how dependencies can share their tagfiles so that they don't have to be listed manually.