Module encapsulation of C++ core library code

[zhztheplayer]

Idea derived from #12808 (reply in thread).

This is to suggest a folder structure change, mainly by moving the top level CMakeList.txt to folder velox to make the folder velox can be individually compilable to build a C++ core Velox library.

By doing this, for new language bindings, e.g., Java bindings, we can rely on a folder-to-folder dependency relationship from the language bindings to the core C++ code. This says, we will have

velox (repository root)
  |- velox (cpp library code)
      \- CMakeList.txt
  |- java (java library code)
  \- python (python library code)

So the language bindings (java, python) code does not have to access the repository root, but rather directly rely on the core C++ library velox/velox as the dependency.

This will make it easier for language bindings to handle the C++ Velox dependency, for example, when the Java library tends to pin C++ Velox code on a fixed Velox commit, it could selectively download the source code in velox/velox from that commit. So recursive content under velox/java in a different commit could be avoided.

[zhztheplayer]

@pedroerp A minor refactor like this will help moving the Java bindings work forward (monorepo). But it's a breaking change for dependent projects (CMake root for Velox is relocated).

cc @assignUser

[assignUser]

This is how apache/arrow is structured as well e.g. the R bindings are in r/ and reference through either tooling or system installed libraries depending on the development scenario Arrow C++ in cpp/

[assignUser]

For what it's worth we are currently moving implementations out of the mono repo (java, C#, go, ...) but these are independent implementations for the most part and not bindings and arrow is in a much more stable place regarding development velocity and API.

[zhztheplayer]

but these are independent implementations for the most part and not bindings

Agreed. Arrow provides a stable ABI so the libraries are not strictly depending on each other in code. While Velox Java code will tightly bind on C++ code and its APIs. So it's very possible to me that Java code pins the C++ Velox dependency onto a fixed commit / version at a time to keep the overall development efficiency. This will be pretty much like separate code bases - although the folders are just organised under the same git repository.

[assignUser]

So it's very possible to me that Java code pins the C++ Velox dependency onto a fixed commit / version at a time to keep the overall development efficiency.

In that case what is the advantage in having the java bindings in /velox ? In that case it would be better to keep it in a separate repo as you will have an easier time merging things without the need for meta employees to merge things into the mono repo

[zhztheplayer]

In that case what is the advantage in having the java bindings in /velox

I will take Velox4J as example, there are 3 possible options for managing Velox4J's code in the future:

1. Keep the GitHub repositoryvelox4j/velox4j as it is now;
2. Contribute to Meta / Velox community as an individual repo, e.g.,facebookincubator/velox4j;
3. Contribute to the main Velox repo as a subfolder, e.g., under folder velox/java.

I actually enjoy hosting the project by myself because it's way easier to iterate the code. However this model will concern the project's users as the work may be considered a personal effort. Another reason is the community power - more users / contributors will be attracted if it's part of Velox. So option 1 was excluded.

I will be fine with option 2 or option 3. But it's not controllable by myself because 2 will involve more Meta internal process. (kindly ping @pedroerp for more comments here)

I used to prefer 2 over 3, but considering 3 will make contributors feel better (credits will be kept in Velox repo), I don't have a strong inclination at the moment.

[pedroerp]

If we want to provide these as part of Velox's repo, my preference would be 3. Ideally these things could be reasonably decoupled, but my sense for now is that this will use a large internal API surface, so making sure these don't run out of sync in separate repos could be a headache.