Module Units
C++20 introduces a module system. To be exact, the system includes core language wording, a set of standard facilities, and a subtle expectation of how we do the tooling. Whether it’s a reasonable expectation is out of the scope of this post.
However, in the C++20 module system, we can divide C++ translation units into 3 kinds:
-
non-module translation units,
-
module interface units,
-
module implementation units.
We’re not talking about the module partition units, as they don’t make a difference here. We’re also not talking about the file extensions of them, which is a mess.
What we really care about is how to build them. Luckily, module implementation units build just like regular translation units. The only trouble is the module interface units. These units have an module interface (abbreviated as MI), and have to be dealt with separately.
Dependency Scanning
Before anything, you should know your enemy. We mentioned that the extensions for the units are a mess, so how do we identify them?
Lucky for us, P1689R5 describes a standard dependency scanning facility. All 3 major compilers support it (though at different extents). It can take any valid translation unit, and output a json file describing its exporting MI and imported MIs.
We still don’t know which kind of unit the file exactly is, but knowing whether it exports an MI or not is enough, from a building point of view. So from now on, "files with an MI" and "files without an MI" would be our main terminology here.
CMI
How do we deal with them, then? The first new concept is the Compiled Module Interface (CMI), often called Built Module Interface (BMI) in Clang world, or IFC in MSVC world. For files with an MI, besides the normal object files, we can also generate a CMI file that represents its MI. The CMI files are completely intermediate. They’re just cached, serialized information that the compiler needs to know about an MI. This makes them:
-
heavily implementation-defined,
-
non-portable, not even compatible between different sets of driver options of the same compiler,
-
don’t really contribute to the final codegen, thus not affecting ABI, etc.
-
not intended to be used by other tools.
How we deal with CMIs is a bit messy.
Generation
There’re 2 ways to generate CMIs. The compiler can generate them as a by-product of object file compilation, or we can ask it to generate them separately. A table for compiler support:
| Compiler | Emit CMIs as by-product | Emit CMIs separately |
|---|---|---|
GCC |
|
|
Clang |
|
|
MSVC |
|
|
Consumption
There’re multiple ways to consume CMIs. One idea is to simply "dump all CMIs to a directory and search them like header files (by name)".
- Pros
-
-
All CMI files would follow the same naming convention, which makes it easy for searching and managing.
-
Really easy to implement.
-
- Cons
-
-
An interface might need to have multiple CMIs (due to the compatibility issues) which cause conflicts in this scheme.
-
If we dump all CMIs to one directory, we can’t control the visibility of CMIs between different groups of source files.
-
The cons are pushing us to complicate this design. "Use more directories", the devil whispers. To solve the conflicts, we might ultimately need to have a unique directory for each source file. For the visibility problem, we’d also want to have a directory for each visibility domain. If we go down this path, the "really easy" advantage is easily destroyed. But, it’s still a working design.
The second way is to specify the CMI paths and names of the MIs they represent manually in the driver options.
- Pros
-
-
Precise, complete control.
-
If we perform a 2-stage build, where the 1st stage is the dependencies scanning, and the 2nd stage is the actual compilation after some collating, the final build graph output would still be static, which often leads to better performance.
-
The build system knows and does everything. It’s easier for build systems to do anything based on their knowledge, like caching and debugging the compilation.
-
- Cons
-
-
It’s complicated and hard to implement.
-
I admit the cons are really just my own fears. But they’re reasonable fears. To walk this way, we need to really consider everything and have a good plan beforehand, just in case we do something wrong and find ourselves hard to turn back.
There’s still the third way, which is to use a "module mapper".
A module mapper is a client-server protocol and a server implementation. The client is the compiling process, sending requests every time it needs to import an MI. The server, provided by users, receives the requests and answers with the CMI path. One such protocol is described in P1184R2.
- Pros
-
-
The mapping takes place at compiling, not before it. This enables us to map modules on the fly, offers dynamic-ness during the build process.
-
The server is an independent program. We can do anything we want with it while the compilation is running/hanging, even including downloading the module source from the Internet. Even the server itself can be a remote service. Its independence makes it a possible place to extend the build.
-
All the state and functionality mentioned can be maintained by the implementation of the server. It might improve modularity and reusability of components of the build system.
-
- Cons
-
-
Do we really need the dynamic-ness? A 2-stage static build can do everything it can do.
-
It might be handy in some scenarios, but it’s also error-prone and potentially bad for performance (hard to parallelize, etc).
-
The dynamic-ness makes it hard for existing tools, like ccache and build system debuggers, to fit in.
-
All in all, the 1st approach becomes the 2nd when it scales. Choosing between the 2nd approach and the 3rd are really "2-stage static build" versus "1-stage dynamic build". This is the main decision point.
Now something to kill the game. The compiler support table:
| Compiler | 1st way | 2nd way | 3rd way |
|---|---|---|---|
GCC |
can be simulated using a module mapper |
can be simulated using a module mapper |
|
Clang |
|
|
not supported |
MSVC |
|
|
not supported |
Now it’s clear that we’ll have to adopt the 2nd approach.
Header Units
You must be wondering, what about the header units? Well, they’re a big problem, but not on the build system part. They can introduce preprocessor states into the importing file but not taking any. Except for that, they’re no different from a file with an MI. We generate a CMI out of a header unit, and feed it to the compiler just like a normal CMI.
They are a big problem just because it makes it several times more painful for compiler devs to implement P1689R5 support. The only "correct" implementation I see so far is MSVC’s. GCC’s needs to know where the CMI is to output a correct json file. Clang’s doesn’t output about them at all. (MSVC’s implementation is actually debatable, too.)
So, yes, it’s a problem. But no, it’s not a problem for us.
To Be Continued
This is the first part of the series, just providing context on terminology and concepts. In the next part, I’ll write about the actual building model and the decisions behind it. And the 3rd part would be its implementation in SCons.
Designing the model is painful. Some of you may know the C++ module system is an analog of the Fortran module system, in terms of building. It is, but just 100x more annoying and complicated. C++'s preprocessor and other conditional compilation facilities cause all kinds of problems. I’ll try to cover the details in the next part.