Make classpath scanning optional

[dsyer]

There's a discussion here spring-projects/spring-framework#27619 about how some common use cases do not require classpath scanning (specifically versionless URL path construction). Myself and @vpavic have both volunteered to contribute code here, but we need some guidance. One option would be to extract the "core core" into yet another Jar file (what would it be called?). Or we could make the classpath scanner an optional dependency here (use it to compile and test but not include it transitively by default). Or we could keep the dependency and simply make the classpath scanning lazy (only do it if it is needed).

UPDATE: added a third option.

[vpavic]

Regarding naming, if new library is the preferred route, maybe we could take some inspiration from the name of the component we have in Spring (WebJarsResourceResolver) and name it webjars-resource-resolver or simply webjars-resolver.

[dsyer]

Actually, maybe we should start by asking a simple question: why does webjars need classpath scanning? What are the use cases that supports?

[vpavic]

I haven't inspected webjars-locator-core at depth, but from my experimentation in vpavic/spring-framework/tree/gh-27619, I suspect that resolving WebJars of Bower GitHub variant cannot be done as easily as for Classic and NPM variants and is probably (one of the things) reliant on classpath scanning.

This is because Bower GitHub variant does not have as static path to pom.properties as Classic and NPM variants do (META-INF/maven/org.webjars/{webjar}/pom.properties and META-INF/maven/org.webjars.npm/{webjar}/pom.properties, respectively) but is rather dependent on the source GitHub org and repo and is like META-INF/maven/org.webjars.bowergithub/{org}/{repo}/pom.properties. Similarly, the path to assets themselves (resources/webjar/...) does not contain version unlike with the other two variants.

You can observe these differences by taking a look at, for example, Bootstrap WebJar in different variants:

• org.webjars:bootstrap:5.2.2
• org.webjars.npm:bootstrap:5.2.2
• org.webjars.bowergithub.twbs:bootstrap:5.2.2

[vpavic]

Another thing to consider, I originally learned about spring-projects/spring-framework#27619 after opening spring-projects/spring-boot#31560 because Spring Boot wasn't picking up webjars-locator-core updates due to versioning policy.

That led me to open #86, and I think that if what we're discussing here end up being part of webjars-locator-core, those concerns will remain and Spring Boot will continue to pick up updates at a very slow pace unless changes occur somewhere (either this repo adopts SemVer, or Spring Boot makes exception to its policy).

[dsyer]

I suspect that we may only support “npm style” webjars, but that’s OK by me - if it’s documented it should be fine for everyone IMO.

If we want Spring Boot users to be able to take advantage, I think @vpavic’s argument suggests that the best way to implement this is to make the classpath scanning lazy in the existing library - only use it if direct lookup fails. I haven’t looked at the code yet but that seems like a feasible solution.

[vpavic]

I suspect that we may only support “npm style” webjars

Hmm, why do you think it's only NPM variant we can support?

The way I see it, Classic and NPM variant are basically equals in terms of the way they are resolved from the classpath. Bower GitHub seems like the only one that isn't straightforward (I'm ignoring Bower Original variant as webjars.org says it's deprecated).

[dsyer]

You’re probably right. I suppose I should rephrase what I said as “I don’t care if we can only support NPM packages”. It’s a bonus if some other layouts work but I don’t think it’s worth a lot of additional effort.

[dsyer]

Here's an attempt at a small step that delivers something useful: https://github.com/dsyer/webjars-locator-core/tree/version-locator. It adds a new utility WebJarVersionLocator with enough features to be used by Spring (and anyone else who simply wants versionless paths) but no classpath scanning. Add feedback here or on my fork.

[dreis2211]

After the interim rise of activity to resolve this there seems to be some loss of traction again.

I can't stretch enough how much of an impact the classpath scanning has. Especially on bigger projects and bigger test suites I see a major portion of allocations being caused by this. Which in turn triggers GC again, which has an impact on CPU and timings.

Here's a (large) project where 17% of allocations are spent on only classpath scanning during tests. I've shared one in the original Spring-Framework issue that even has up to 60%.

What's especially bad in the Spring case it's that it's doing the classpath scanning multiple times. (Basically once per registration inside ResourceHandlerRegistry.

I can workaround a few places where I have control over the code, but it would be really better if this would be solved for good in the underlying libraries.

Cheers,
Christoph

[jamesward]

Sorry this took so long. I've been trying to wrap my head around everything that exists today and why. Here are some initial thoughts...

1. Classpath Scanning:

   For user ergonomics we don't require a WebJar artifact ID or a full path to the file (after the version). So the user can just write: webJarAssetLocator.getFullPath("bootstrap.min.js") and they get the resolved artifact (WebJar and subdir), like: /META-INF/resources/webjars/bootstrap/3.1.1/js/bootstrap.min.js. This resolution requires doing some sort of classpath scanning. I think we used to manually get directory content listings but switched to using the Classgraph library. We could maybe switch to PathMatchingResourcePatternResolver in a Spring implementation. Does that work with GraalVM Native Image correctly?

   As @dsyer proposed, we could offer an API that requires a WebJar artifact ID and a full path but I think the DX will take a negative hit. Today there is a similar method (but it is implemented internally with Classgraph): public String getFullPathExact(@Nonnull final String webJarName, @Nullable final String exactPath)

   Having a library that just has that method would definitely be an option but that doesn't address the ergonomic issues. Maybe those issues aren't a problem for Spring as it seems maybe most users don't use the looser getFullPath(partialPath) method.

2. Version Caching:

   Even if we get rid of scanning we will probably want to figure out some way to cache the version lookup as it is something we don't want to do every time a WebJar asset is resolved.

3. Bower GitHub WebJars

   For a variety of "good reasons" we decided not to put the version in the Bower GitHub WebJars paths. Whatever change we make should be compatible with those WebJars or users will have a hard time.

4. Spring & Versions in Paths:

   As we look at making changes to this in Spring we really need to look at a few other things which don't fit the usual WebJars paradigm. First, it is intentional that we put the version in the path so that good HTTP cache headers can be put on the WebJar requests. It looks like we break that with the way Spring currently removes the path. Second, for Play we add an easy config option to switch to using the WebJars CDN. Both of these could be accomplished with some HTML templating utils which we should investigate for doing in Spring.

Overall I think the best way to deal with all of this would be to do a lot more at build-time. If we have any opportunity to do that, I think that is the best place to invest the energy but requiring another build plugin is probably a non-starter so may not be an option.

If we just want to plug the current hole and ignore the DX & HTTP cache issues, we can take @dsyer's changes and publish a webjars-locator-core-base (or whatever) with the non-scanning logic but we will still need something to address the version caching and I'm not sure the best place or way to handle that.