Ongoing refactoring

[typhonrt]

This issue tracks the progress of the initial ongoing refactoring occurring with TJSDoc.

Just dropping a notice for those w/ early access. I suppose watch this repo to get updates on progress.
@dandv, @raveclassic, @ratson, @skurger, @connor4312

[ratson]

Looking forward to see the code~

[typhonrt]

It's coming. I hope to have things wrapped up with the initial effort committed by Wednesday or earlier. There have been a lot of changes that have benefited from doing this work offline. Everything has been examined and hardened.

[typhonrt]

@ratson, well, you know software.. right? Ah, it's going to be a few more days as a few great features recently got added to the TODO list that I finished yesterday. I don't expect any more items to be added, so I'm working down the final list which is now mostly getting to an 100% documented code base and other housekeeping like alphabetically sorting all methods in files, etc. So a few more days, maybe through the weekend, but likely not longer than that.. Gotta 💯% this.. ;)

[typhonrt]

OK... So I decided to spend a few more days to finalize what I think is a solid initial public release of tjsdoc. I'll switch the repo from private to public on initial commit and publish to NPM, so that everyone can start using this effort.

Some of the cool features I just finished (in addition to dozens more I haven't mentioned yet):

• It's possible to directly archive (zip or tar/gz) generated docs. By default AST and docData dump is not included unless specified with config params. Also it's possible to split the archived output for docs, AST, and docData as separate archive files. This will be handy when I also add escomplex analysis in the future. This feature enables a CI tool I plan to create for TJSDoc which will generate docs and compress them for automatic upload to a separate docs repo and an associated container based web app which pulls down newly committed data to host docs live as commits occur.

• Support for extending configs like eslint.

There are many more enhancements as well. My remaining TODO list is to separate out the single page output to module scope overview for functions / typedefs, convert and / or fold in my previous ESDoc plugins and glob matching for sources / multi root repos. Switch to material-components-web for mobile responsive template usage + useful components with zepto/velocity integration. Then finish up documentation and create a website for TJSDoc. So a little more to do, but if I keep up the hustle things should launch next week or sooner. A bit more of a delay, but I think launching publicly out of the gate is a good thing and all of the above definitely will separate TJSDoc from ESDoc in terms of look and feel.

[typhonrt]

Ugh... It keeps going on and on... ;P You know that final 10% that takes longer than the 1st 90.... I'll give a little update on current progress at the end, but do have some musings on possibly creating 4 main repo / NPM modules for release. 1 common runtime, 1 common publisher, 1 babylon, 1 typescript.... I have everything separated into a common runtime and babylon runtime which extends things that is all plugin driven such that you set "config.runtime": "tjsdoc-babylon" in the config and the babylon runtime is loaded and chugs along. For typescript, yep, tjsdoc-typescript. Everything is handled with custom config overrides for default file extensions along with anything else pertinent for the particular runtime.

One of the cool things is that the common runtime tjsdoc could still be installed separately, but you must also include the runtime of choice and set the config parameter. Where I am thinking the 3 separate runtime repos come into play is being able to release tjsdoc-babylon as a CLI enabled module which includes the runtime pre-configured. A simple Object assign would do it.

import TJSDocCLI     from tjsdoc/src/TJSDoc.js
import TJSDocBabylon from './TJSDocBabylon.js'

/** 
 * Provides an overidden TJSDoc CLI implementation setting the runtime to TJSDocBabylon.
 * @inheritdoc
 */
export default class TJSDocBabylonCLI extends TJSDocCLI
{
   /* @override @inheritdoc */
   exec(runtime) { super(TJSDocBabylon); }
}

That's it.. TJSDocBabylon has a simple override for the generate method:

import TJSDoc from 'tjsdoc';

/**
 * Provides an overridden version of TJSDoc with a default runtime assigned to `tjsdoc-babylon`.
 * @inheritdoc
 */
export default class TJSDocBabylon
{
   /* @inheritdoc */   
   static generate(config) { TJSDoc.generate(Object.assign({ runtime: 'tjsdoc-babylon' }, config)); }
}

Likewise something as simple as that for the typescript version. Having separate CLI runtimes for vanilla tjsdoc where the user manages the runtime and even plugs in a complete 3rd party one if so desired or the two specific CLI / modules both could be installed even globally at the same time and used independently.

This brings up testing... There will be a separate GitHub repo that holds all the tests shared between the runtimes. The runtimes only having to produce the DocDB and the plugin system provides ample opportunity to mock and load the minimum runtime which can be data driven (tests work for babylon & typescript transparently). The static HTML publisher is also a separate module with all of the HTML tests accompanying it.

The alternative and lazy way is to just run everything in the same repo and assume technical debt for a quick / lazy win. I think the above strategy is good to get to before public release as any short term ramp up speed for other developers will be worth it. I think it will more than make up for things being a more user-friendly experience as well for not that much more work which also benefits from doing this out of the gate; IE no downstream change to multiple repos / user confusion.

Any thoughts? I'm still reasonably confident for a launch before 2 weeks, but maybe 10 days. Hard to say. If all goes well from here that'd be great. I'd really like to get this done before GDC for a public release.

---

Other cool stuff... I did finally get the config resolution / extension / validation loading perfect and into a separate repo. It works hand in hand with the plugin manager with custom resolution for plugin deduplication in all extension scenarios and has extensive pre / post validation steps along with setting any missing default values between pre / post. It also is a plugin and while a class is the default export it is consumed in TJSDoc as purely a plugin.

This gives some insight into how the rest of TJSDoc is structured. Everything is event based over a shared eventbus which is injected via the plugin system. There are ~85 specific event bindings when completely configured and there are no dependencies at all between even reasonably fine grained plugin breakdown. The plugin system actually sends out a proxy so that all event binding registrations made from the plugin are transparently managed. Unload a plugin and automatically all event bindings are unregistered without the plugin authors intervention.

Here are the two points that registers new ConfigResolver plugin instances. First internally in an instance L296-L326 and then there is a module scoped plugin method L467-L477.

A bit of a mouthful I suppose especially the snippet below... I'll probably just quit it after the next code block.. 8D

In typhonjs-babylon which just exports a module scoped onPluginLoad method along with the default class with the runtime preset thus loads all the Babylon specific plugins including kicking off the common runtime. Here that is in entirety:

import path       from 'path';
import TJSDoc     from 'tjsdoc';
import ConfigData from 'tjsdoc/src/ConfigData';

/**
 * Provides an overridden version of TJSDoc with a default runtime assigned to `tjsdoc-babylon`.
 * @inheritdoc
 */
export default class TJSDocBabylon extends TJSDoc
{
   /* @inheritdoc */   
   static generate(config) { TJSDoc.generate(Object.assign({ runtime: 'tjsdoc-babylon' }, config)); }
}

/**
 * Adds all Babylon runtime plugins.
 * @param {PluginEvent} ev - The plugin event.
 */
export function onPluginLoad(ev)
{
   const eventbus = ev.eventbus;
   const dirPath = path.resolve(__dirname);

   eventbus.trigger('plugins:add', { name: 'tjsdoc' });
   eventbus.trigger('plugins:add', { name: `${dirPath}/doc/` });
   eventbus.trigger('plugins:add', { name: `${dirPath}/parser/` });

   // Adds an instance of typhonjs-config-resolver with overridden common TJSDoc config resolver data for usage with
   // Babylon / JS source code.
   eventbus.trigger('plugins:add', { name: 'tjsdoc-config-resolver', target: 'typhonjs-config-resolver', options:
   {
      eventPrepend: 'tjsdoc',
      resolverData: ConfigData.createResolverData(eventbus,
      {
         defaultValues:
         {
            'includes': ['\\.(js|jsx|jsm)$'],
            'pathExtensions': ['.js', '.jsx', '.jsm'],
            'test.includes': ['\\.(js|jsx|jsm)$']
         }
      })
   } });
}

[typhonrt]

So I almost made the GDC "artificial goal" of getting TJSDoc fully shipped, but one module remains to be finished; that is the static HTML publishing module. I've certainly got my upgraded version of the ESDoc version and template working, but I need to get things shifted over to the new material-components-web based publisher in addition to adding the ability for plugins to add menu items with static html destinations, roll in of my enhanced left hand navigation, and new addition of a file / module overview output for functions / typedefs, etc.. I certainly plan to get TJSDoc launched next week, so things will continue to progress, but the new publisher might take a few days. There is some exciting news to follow the module overview...

So... Things ended up with the following modules for the main ES6+ / Babylon distribution:

• tjsdoc
• tjsdoc-babylon
• tjsdoc-docs-babylon
• tjsdoc-docs-common
• tjsdoc-runtime-common

missing / not checked in currently (pending update):

• tjsdoc-publisher-static-html

with test and test utils for all modules with direct github dev dependencies for:

• tjsdoc-test-utils
• tjsdoc-tests-ecmascript

The exciting news is that I came up with a good way to work locally for development using Babel environment options for .babelrc, the dev-<x> NPM scripts, and a Babel plugin (babel-plugin-module-resolver). If all of the repos are checked out locally one can run TJSDoc against the local modules due to the module resolver including full test support across all modules with the dev-<x> NPM scripts. This allows development to occur in a reasonably transparent process across all modules locally.

Though at this time checking out the modules manually is the general situation I'll be creating in the next month or so a tjsdoc-dev CLI which will automatically checkout or fork depending on desired setup a developer environment including WebStorm project files.

The main runtime specific parts of the code for Babylon which need to have Typescript implementations are the following:

• parser dir in tjsdoc-babylon
• all of tjsdoc-docs-babylon

I'll certainly be creating an architecture document to describe how everything fits together.

[typhonrt]

So the rewrite of the static HTML publisher with material-components-web works well. I still have a bit to finish up before publishing this module, but an interesting issue was posted for file watching / incremental doc regeneration. I have taken this on as it really allowed the opportunity to make sure the TJSDoc runtime is robust and fine tuned. Of course it's tricky and took a while, but tjsdoc-plugin-watcher is complete and well tested including various stress tests.

Now tjsdoc-plugin-watcher doesn't actually do the regeneration of docs, but sets up file watching on the source and test files in the onComplete plugin callback altering the control flow of TJSDoc. It keeps TJSDoc alive and posts events which any other plugin can pick up when changes occur. There is a terminal option for user input and options for headless / silent operation. I'll be starting tjsdoc-plugin-watcher-doc-regeneration soon and this plugin which depends on tjsdoc-plugin-watcher completes very fast incremental doc regeneration. I have to work out the TaffyDB merging and resolution of doc data, but I expect things to take about a second or less to regenerate a single changed file making it suitable for live regeneration.

Interestingly enough I started with gaze, but it had some silly bugs and fell down in rapid regeneration of all docs / starting / destroying gaze repeatedly. It turns out that chokidar didn't have these issues.

So yes... A bit more of a delay on the launch of TJSDoc, but well worth it IMHO. It may still be 7-10 days before things are fully published and live on NPM.

[typhonrt]

Alright an update is certainly due. The live doc regeneration feature is well under way at this point though it really took some time to restructure the internal data handling of TJSDoc. Now everything works via the DocDB and there is no separate AST storage. Having separate data sources caused complications when it came to deleting old doc data and merging new data. The main DocDB is also now available at all times during execution along with event bindings. The doc generation process also has now been thoroughly overhauled with some initial performance (memory) optimizations.

All doc data generation now happens by static classes instead of creating new objects just for generating data. All generated doc object data is immediately inserted doc by doc into the DB when they are processed versus storing everything in an array with one bulk insert into the DocDB (note: for ESDoc there is no DocDB and the TaffyDB instance is only available in the publisher subsystem with additional access methods in an abstract class DocBuilder. This changes the plugin callback for processing doc objects before insertion as there is now onHandleDocObject which is invoked for every doc object before insertion versus sending one array of bare doc data to a plugin callback. There is a major benefit to this as the AST and specific AST node for the doc object is included along with the already processed doc data which makes it possible for plugins to intelligently provide further processing of doc data with the connected AST node. This should allow Flow typing support for instance to be added as a plugin versus building it into the standard default processing as the specific node associated with the doc object is available for processing which contains the additional flow type nodes. Right after onHandleDocObject is invoked the doc data is filtered to remove the AST data and node unless outputASTData is true in the target project TJSDoc config. By removing AST data flowing through the system this greatly reduces the memory overhead as the AST data is usually rather huge.

Another massive improvement that was necessary was restructuring the old DocFactory which is now DocGenerator. One of the biggest hacks / inefficiencies / and near unintelligible code sections of course non-commented as well of ESDoc is DocFactory. To process the AST in one pass ESDoc rewrites the AST copying / synthesizing class and variable nodes via JSON.parse(JSON.stringify(node)) that are associated with export nodes and placing them at the end of the AST body. This allows processing to occur in a single pass, but destroys the AST and makes the whole aspect of outputting AST moot as it is no longer properly formatted. It took ~10+ days or so to work out a two pass algorithm for processing export nodes in place that didn't require destroying the AST. This is possible because of the incremental addition of doc data to the DocDB such that any updates necessary for the 2nd pass on export nodes modifies the existing doc data via queries to the DB. Removing all of this copying and modification of the AST was a huge memory optimization win and the code is a lot clearer and will greatly aid supporting TypeScript. I'm still adding in comments as this was / is the most complex part of the system. This whole process was possible by unifying on an always persistent DocDB. I hope to get memory usage much lower, but it's now about half of the overhead of ESDoc.

Other interesting updates is that there is a very clean integration of live-server via a wrapper typhonjs-live-server which provides the capability for graceful shutdown of live-server without hanging the process. An accompanying TJSDoc plugin tjsdoc-plugin-live-server starts and shutsdown live-server with the control flow of TJSDoc. A browser is opened automatically for the doc destination source and when docs are regenerated the current page is updated immediately, so this is going to be a very nice feature when complete.

For the latter part of the week I'm finishing the doc regeneration process with final merge support and incremental publishing of sub sections of the DocDB. The nice thing is that the merge process updates the target source file changed and any dependent files as well which certainly will be pertinent once @inheritdoc support is added.

So things are moving along. I'm certainly glad that this feature was taken on now as it really required cleaning up the internals, how data is stored / accessed, the plugin callback mechanisms and initial performance concerns. This also means that all technical debt assumed from ESDoc has been reversed. There is now no part of the architecture that relies on what I'll label as bad hacks.. Everything has been thoroughly vetted and re-architected as necessary.

[typhonrt]

I know it's been awhile since the last update. I had to take a break as my mind was melting a bit with the live doc regeneration work. The good news is that initial support is working and the results are very nice. For a single file changed without dependencies via extension (child classes) live doc regeneration from saving the file to results showing in the browser is less than a second! Each additional dependent file regenerated adds about a 1/3 of a second each mostly due to markdown to HTML conversion in the publishing module. This is also on my 2011 MBP, so a more powerful box should really be quick. I hope to finish up the live regeneration functionality tomorrow with a bunch of tests and then get back on working on the new material-components-web publisher module.

I do have some questions though to anyone following this thread:

For file additions / deletions a full regeneration of all docs is necessary to catch all corner cases without putting in a great deal of effort to root out all of the cases. This is still reasonably fast as TJSDoc is spun up and ready to regenerate all files. Is it reasonable to assume when adding and deleting files that a full regeneration of docs is acceptable? My answer is yes right now and that reduces quite a bit of complexity.

The standard use case is changing an existing file adding or modifying docs. I have forward / child relationships (re: future @inheritdoc support) updating and likely will add reverse / parent relationships for when say the extends keyword is removed from a class such that parent docs update as well which is mostly just updating the indirect / direct subclass relationships at the top of the class HTML. Anything beyond that IMHO should require a full regeneration.

It's easy to do a full regeneration with the watcher functionality as there is a terminal and one just types in 'regen'.

Also for the time being I'm not supporting modification of the published manual output if the related markdown files change or the index / README. This will require a full regen. I may update this, but I'm on the fence on these cases. I may very well support it once I finish off final publisher module efforts.

I think it's important to support single source and test source live doc regeneration where the user is incrementally adding docs to a single file with the expectation that for more complex use cases a full regeneration of all docs may be necessary.

[typhonrt]

Once again a long overdue update. I'm about ~80%+ done with the completely rearchitected publisher module which no surprise has turned out to be a lot of work. Like the rest of TJSDoc everything is plugin based now and various sections of the publisher separated into actions, components, and content (file, class, module) plugins along with internal implementation plugins.

One of the nastier "god objects" of ESDoc is DocBuilder. If you are trying to create a 3rd party plugin which outputs new doc pages you must extend it which makes things rather fragile. Of course all existing ESDoc plugins that output new doc pages break with the v1.0.0 release of ESDoc. All of the doc data / taffy DB access found in DocBuilder has been moved to DocDB which is persistently available across the TJSDoc runtime. Rather than requiring fragile extension of DocBuilder like ESDoc the rest of the general functionality of DocBuilder is now available via events over main plugin eventbus. So any 3rd party plugin authors don't need to extend anything to create plugins which output new doc pages as all the functionality to do so is accessible by events.

Another small, but pertinent change is a runtime pattern to dynamically create the top navigation menu. This is and remains to be a huge PITA for 3rd party plugin authors creating plugins which output new doc pages. It's hard coded in ESDoc and any plugin which outputs new pages which needs to add a menu link in the top nav menu needs to rewrite all HTML output. This is very fragile, but TJSDoc dynamically creates the top nav allowing 3rd party plugins to insert links in an organized manner not requiring any bulk rewriting of all HTML pages output.

Things also have a refined structure moving the HTML, images, CSS locally to each content component versus a bulk copied over template directory. This was easy with HTML / images, but handling CSS in a modular manner turned out to be quite problematic, so it has taken up a good deal of recent effort. With ESDoc there is a static all inclusive style.css that with v1.0.0 has a minor improvement of breaking out the different sections, but still all imported in a static manner in style.css. If you are a 3rd party plugin author that needs to provide extra styling for any doc pages perhaps you could try to hook into the user scripts system, but this is fragile with the only other option being to rewrite all HTML pages depending on what is being added. For instance this was necessary for esdoc-plugin-enhanced-navigation. I guess it should be noted that 3rd party plugins can easily replace internal components like the navigation as it's composed by events versus being hard coded.

In TJSDoc the solution that I'm well onto the way of completing is a separate utility plugin which integrates PostCSS in an event driven manner available over the plugin eventbus. Like other utilities it is separate of TJSDoc though not published on Github yet. One can create a grouping for a CSS file to be composed via appending / prepending directly various subsections along with optionally running PostCSS against a particular subsection with particular PostCSS processing plugins. On final output of the composed CSS file once again it can be processed by PostCSS. So now all CSS is localized to the component that it depends upon and it's possible for 3rd party plugin developers to integrate CSS into the main style.css. I'm also thoroughly going through the existing style.css with an eye to which PostCSS plugins like autoprefixer and precss which will greatly aid theming and creating 3rd party theme oriented plugins to style TJSDoc output with the minimum of fuss. The nice thing for development is that source maps are tracked across all the CSS components which make up a final file, so it's possible to enable source maps for development purposes in Chrome, etc. I'm also quite pleased to announce a light and dark theme for TJSDoc as well; the dark one being my default / favorite!

All material-component-web integration has been complete for a while, but I'm also going to be taking the time to fully complete and update all of my extensive ESDoc plugins verifying that they work with TJSDoc seamlessly, so I have plenty of complex 3rd party plugins as a test bed to verify that integration for plugin authors is considerably smoother and non-fragile compared to ESDoc.

Right now the publisher module is still not committed along with the final watcher / doc regenerator plugin. I've been working on these offline due to the many changes involved. So my commit log on Github is a bit deceptive over the past month or so. Work continues and TJSDoc will get out there. All of this baking is a good thing, but is going on a bit.

I suppose I should comment that I have reduced the amount of time that I've been working on TJSDoc as I did hit a burn out moment near the end of April. I was pushing ~80+ hours a week on TJSDoc from the beginning of the year until then... I'm hitting ~30-40 hours a week now and have started deliberate pool / billiards training which has been a hobby I've neglected for a while as far as getting better. I've finally made the jump to the big tables (9') leaving the bars behind and wake up around 3-4am to work on TJSDoc until 11am then head out to the pool hall for 5 hours of practice (membership hours). I'll be switching to a different pool hall to practice in July which thankfully has membership hours between 3pm and 8pm, so I can get up around 6am which is much welcome, so this month has been a bit crazy. Hopefully in 3 months or so I'll have a 9' table at my home which will help even more w/ time management. With any luck within 6 months or so I'll be at the semi-pro level and plan to travel around to all regional tournaments across California and further coming up, so just not banging away at TJSDoc for 80+ hours a week; perhaps staying more sane... That is debatable though... ;P

[typhonrt]

OK... It may have seemed like my commits died again over the last couple of weeks, but after much gnashing of teeth and offline work a major architecture overhaul is complete! Now I didn't mention in my last update the quandary introduced by incorporating PostCSS into TJSDoc. For the astute you'll know that ESDoc is and hence TJSDoc (was) a synchronous runtime and for those with experience with PostCSS you'll know that it's an asynchronous API and that the synchronous API has been deprecated. While the sync API is still there it will only work with PostCSS plugins that support it. So PostCSS was the straw that broke the camels back so to speak though going async was on my mind for a while... Of course this allows full PostCSS integration and I'll mainly be focusing on the cssnext plugin for the main supported plugin for adding modern CSS features to TJSDoc.

TJSDoc is now a fully asynchronous architecture and has thoroughly adopted async / await! In addition to modifications to typhonjs-plugin-manager which now fully supports async plugin lifecycle operation all plugin loading and the majority of control flow of TJSDoc is asynchronous ready. I'll list a bunch of the async callbacks below, but any method appended with Async is awaited upon where it is invoked / events dispatched. So any 1st or 3rd party TJSDoc plugin may utilize asynchronous operations via using async / await itself or by returning a Promise in the callback.

This is super cool as it opens up so many more possibilities for plugins to interface with the outside world. In particular I'll now be supporting a plugin that queries the Github API (and potentially other SCM systems in the future) to find the exact links to commits when documentation is generated versus providing links to just the master branch which can change over time for all source and navigation linked in generated documentation. Additionally the future CI plugin to automatically upload generated docs to a web app and / or a separate repo can now be completed entirely as a TJSDoc plugin versus more heavyweight / external system. I'm sure the larger community will also come up with many creative ideas on how to integrate documentation generation via plugins with the outside world as well which will be exciting to see!

One should note that ESDoc likely will never become asynchronous or if it does it will once again have to up the Node requirement to v8.x and be a major breaking change to all users / plugin authors. The ESDoc maintainer removed usage of Babel and thus is now dependent on Node support for language functionality. IMHO that was a misstep as documentation generation is a development utility and as such it should run as widely as possible across as many versions of Node as possible. TJSDoc embraces Babel and all stage 2 functionality for implementation and doc generation out of the box.

Here is a mostly comprehensive list of all TJSDoc asynchronous callbacks / plugin hooks:
onHandleConfigAsync
onHandleDocDBAsync
onHandlePostPublishAsync
onHandlePrePublishAsync
onHandlePublishAsync
onHandleSearchIndexAsync
onHandleVirtualAsync
onRuntimeCompleteAsync
onRuntimePreGenerateAsync
onRuntimeRegenerateAsync
onRuntimeShutdownAsync
onRuntimeStartAsync

...and from the plugin manager: onPluginLoad / onPluginUnload

I have to say though that the gnashing of teeth was real and I almost threw in the towel on the async / await conversion as it was fairly tricky. The one bug that got me was using await with a ternary operator. This doesn't do what you may think it does:
await typeof TJSDoc.default === 'function' ? TJSDoc.default.generate(config) : TJSDoc.generate(config);

indeed parenthesis are necessary:
await (typeof TJSDoc.default === 'function' ? TJSDoc.default.generate(config) : TJSDoc.generate(config));

This is from a utility for testing, so all tests that generate docs which there are many did not work correctly and control flow exited before the end of the TJSDoc runtime completed. I spent days trying to find the problem and this was the last place I looked after copious divide and conquer elimination!

So work should now be progressing again and I'll be back on finalization of the publishing module tomorrow and hopefully TJSDoc will be out sooner than later. I really don't foresee any more major stalling architecture overhauls, so it's just crossing off everything else on the TODO list. Hang in there folks!