Rewriting the bundler in the Lua

[ThistleSifter]

I've had it on my mind for a while now that it would be a good idea to rewrite the bundler in Lua. Having a significant part of our functionality written in a language with which most contributors aren't familiar is less than ideal. By my count only @asherber and myself are familiar with it, but I haven't been able to contribute for a while, so that really only leaves one person.

So I'm opening this to plan and discuss any issues around this, such as:

• Is there any reason that it can't be in Lua? There are extensions for Lua for things like PCRE, etc
• Can we run Lua with extensions installed from an action?
• ... and whatever else I might have missed

Cc @asherber @rpatters1

[asherber]

I agree with the general idea of making the ecosystem accessible, but I'm not in favor of taking this action, for a number of reasons. Off the top of my head:

• I am reluctant to rewrite something that's already working.
• The JS ecosystem is bigger, easier to find answers in, easier to navigate, easier to set up and test. It seems to me that the major selling point of Lua as a language is its easy interoperability with C -- a significant factor in using it (rather than a better-known language) to interface with Finale, but not relevant for this project's supporting infrastructure.
• I don't think that having the bundler in Lua will really make it any more welcoming to the community in general than it is now, because of what I expect to be the complexity of the code.
• It's reasonable to expect that some complicated parts of the ecosystem (the website, the build process, the mixins, RGPLua itself) are provided and maintained by a subset of the community for the benefit of the whole.

To answer @ThistleSifter's question, yes, it would be possible to run this from an workflow. There's an action that installs Lua on the runner (with luarocks), and then you would use a run statement to invoke your Lua file.

[ThistleSifter]

I wasn't look at it from the perspective of making the repo more welcoming or inviting. I don't expect that a bundler written in the same language as the rest of the repo will attract an influx of contributions. Rather, I was looking at it from the point of view of the bus factor. You're right that, for example, you wouldn't expect everyone to add to or modify the mixin library. However, I have documented it as best as I could (I'm always happy to take suggestions for better docs) and should the need arise for someone else to make changes, the only hurdle is understanding the code. Websites aren't built with Lua and it's also a separate entity built on top of this repo, which could still function without it as a place for people to get scripts, whereas the bundler is part of this repo. The JS ecosystem is only useful if you already know JS, or TS in this case.

As somebody in another team once said to me (in good humour) "now it's okay if you get hit by a bus."

[rpatters1]

I am wondering if converting the build over to Lua would buy us much. The thing about Lua is that where expertise matters is in the domain it's running under. To do anything useful with it, you have a domain-specific library that it is interacting with. (In our case, it's the PDK Framework.)

Whatever domain library a Lua bundler is running under is not going to be any more common knowledge within the team than TypeScript. I think mixin is actually a good example. It is a fundamental part of the project now, but until recently only @ThistleSifter was able effectively to modify it. In this last project (with his help) I think I finally understand how it works well enough to make at least superficial changes.

I've never used TypeScript except in this project, but it's just a computer language and those are basically all the same. The syntax is not what's hard. What's hard is understanding the structure of the program and the ramifications of changes.

This project will survive only as long as there are team members interested in learning new aspects of it. And usually that is driven by the Mother of All Invention.

[asherber]

@ThistleSifter makes good points about documentation. The bundler has reasonable general documentation, but it could use some more specific docs about program structure, and warnings about certain gotchas; I'm happy to take on those improvements.

FWIW, as it is used in this repo, TypeScript is really just JavaScript with type checking.

[rpatters1]

As long as @asherber is volunteering to update documentation, expanded documentation about how to build, test, and deploy the jw-lua-scripts-docs repo would be a very helpful addition to the README.md there. The current instructions are minimal.

This probably goes without saying, but any new docs for this repo should be at or clickable from the README.md here.

[asherber]

(The bundle and get-metadata actions, which live in this repo, actually have their own READMEs. I think it's probably good to keep them separate, since the info there will only be relevant to people working on those parts of things -- don't want to pollute the main README.)

[rpatters1]

It might be at least worth adding a link to the readme files for each from the top-level readme. Links can be relative to the top of the repo, and they will work in any branch that way.

I did this, for example, in the readme for luaosutils.