Use node-releases from npm?

[dominykas]

Just spotted https://www.npmjs.com/package/node-releases. Would it work better than fetching (and dealing with caching, etc) the data directly from GH?

[wesleytodd]

Interesting. There are for sure some benefits from this approach, that said it adds another layer between us and the upstream data. What if we supported both via a preferOffline option? The reason I would hesitate to make it default is that you might accidentally get stale data when you do not expect it via an old package version in a lock file.

[dominykas]

Or maybe just provide a way to pass in a custom schedule/dist list file (buffer or parsed JSON)? Would also beat having to Nock() the requests made by nv in unit tests...

[wesleytodd]

Oh, I like that idea. So we would then also document using this new feature with the node-releases package and then use it in our tests.

[wesleytodd]

I was just using this lib in a new thing and so circled back on the issues. I think we should take a two part approach to this:

1. Allow passing a pojo of the json data (fixes your nock issues)
2. Publish a secondary package (from this repo or another) which builds a nightly package like node-releases

So the point of 2 is that we can get the same end benefit, but we can fully pre-process at build time. Then with this secondary package as a dep we can add add an option like bundled: true or bundled: <pacakge_name> where it will not make the http requests or do any processing at all, just used the pre-built values.

Since it would be a nightly (or maybe since the project owns it we could hook it's build into the node release process directly) and imported with a loose(ish) semver range it would be "fresh" on install but lockable along with other deps for reproducible builds.

Thoughts @dominykas?

[wesleytodd]

So what brought me back around to this issue is this use case:

We use @pkgjs/nv as a source in a daily job which generates new container images for node versions. Since it is daily, if you run @pkgjs/nv as a part of a separate toolchain to create containers on top of our nodejs base container, there is a race between when the new versions being published by node vs published by our internal container job.

The reason I think the approach about with bundled: <package_name> is better is that we can add to that job a step which publishes an internal version of the bundled data and point all our internal usage to that. We never want people using a node version which does not have a corresponding container base image yet anyway.

Just wanted to expand on why I think the above is a good solution.

[dominykas]

Is a nightly necessary? It only needs to be re-published when there's a new release.

Thing is I figure that we don't want to publish nv (or anything else, for that matter) each time there's a new node, hence why I figure that using something that is npm installable ¹ is worth a shot (rather than something bundled), unless that's what you mean by bundled: package-name, in which case yeah, it's not a bad option.

And then yeah, passing in an explicit schedule as an override would also be quite useful (if I understood the intent behind how you use it at daily job), even if it's a starting building block.

Footnotes

1. I wonder if we can convince folks to add a package.json (even if it's private: true) in https://github.com/nodejs/release, so that it becomes npm installable and use it as a git-based dep directly here? And then have an optional refresh: true to fetch latest (or a refresh: false)? ↩

[wesleytodd]

unless that's what you mean by bundled: package-name, in which case yeah, it's not a bad option

Yeah that is what I meant by that. The package you started the conversation with is a nightly build, so that is why I mentioned doing the same. The Main "win" I think we can get by managing it ourselves is doing all of the processing into aliases at build time instead of runtime in this package. So instead of the first run of this package building the alias mapping data structure from scratch we would have it prebuilt.

I wonder if we can convince folks to add a package.json

I really really would like to avoid proliferating git installs of things. There are so many things complex about analysis when the underling state is mutable. But maybe we get the best of both worlds. If we integrate a trigger on nodejs publish to rebuild and publish it to the registry, we can avoid the nightly but get an immutable artifact to reference.

[wesleytodd]

Years later my opinion on a nightly build has 100% shifted. I think that is a bad idea because it will just result in a huge amount of dependabot/renovate/whatever spam. I am thinking that we should just "just provide a way to pass in a custom schedule/dist list file" as you said originally. I am going to pick up #34 in the next month and will try to land this before that if no one opposes this approach.