Hi,
joelonsql .
I'm architecting the compiler to serve as a kind of meta-compiler, with the normal Lux compiler simply being a sub-component to it.
The idea being that if someone wants to add their own compiler to it, they'll be able to define it, and they'll be able to tell the meta-compiler which files/modules should be compiled with their own compiler, and which files/modules should be compiled by other compilers.
The way most compilers work, is that they have an implicit assumption that all files in a program must be compiled the same way.
So, if you have a C compiler, then the assumption is that all files in your program contain C code.
The same goes for a Java compiler, C#, Haskell, or any other language.
I'm going for a different approach in Lux.
The idea is that the compilation unit for a compiler should not be an entire program, but rather individual modules (i.e. files containing code).
So, if I have a Lux compiler that compiles .lux files, there is no assumption that all modules in my program are .lux files.
It may be the case, but it doesn't necessarily have to be the case.
The idea then is that you may potentially have multiple compilers being used at the same time, compiling different modules based on their file extension, or where in your program's directory structure they are located, or other criteria.
So, that handles the part of having multiple compilers, but how do they talk to one another?
Well, I added a special data-structure in the compiler which I call "
the archive".
Basically, every time a compiler compiles a module, it generates a data-structure which I call a "document".
The document contains information about the module, such as what sorts of things are defined in it, meta-data, types, etc.
Documents can be of different types, since compilers are at liberty when it comes to choosing their preferred representation.
That means the archive must be a heterogeneous data-structure, since it must contain documents of all sorts.
The way I group all those different documents together in a safe way, is by limiting access to the documents.
I have an abstraction which I call a "key".
Each compiler gets its own key, and keys are unique and impossible to confuse with one another.
Keys have a type-parameter that stands for the type of the document associated with a particular compiler.
The way it works is like this:
Compiler C1 compiles module M1. Then it stores its document D1 inside the archive, using its key K1.
Then, let's say a different compiler C2 is compiling its own module M2. And let's say M2 wants to import something from M1.
Since M2 is being compiled by C2, and M1 was compiled by C1, we can presume that they might be completely different languages, and yet one of them wants to interact with the other.
Well, if the C2 compiler knows how to work with stuff that was compiled by C1, then it can just use the K1 key to access the D1 document, which would tell it everything there is to know about the M1 module. Then, with that information, the import can be done, and the C2 compiler can wire things properly so M2 can use M1's functionality.
This means that compilers must have the means of working with the stuff produced by other compilers they might want to interact with, and that interoperation is not just a given.
This must necessarily be so, since compilers are at liberty to compile modules and everything in them however they please, with nothing binding them or constraining them, so coming up with a structure that enables uniform/homogeneous interop is pretty much impossible without sacrificing that freedom.
But I think this way of doings things is a solid way, and might serve as a foundation for even easier interop in the future.
Also, compiler code in the Lux stdlib is being geared towards high modularity and re-usability, so compilers will be able to share a lot of stuff with one another, reducing the amount of necessary work by compiler-writers, but also providing a shared foundation that should make some aspects of doing cross-language interop easier.
I also have aspirations of doing some sharing for the runtime layer of the language (the sorts of primitive functionality that gets packaged on each program to enable the most fundamental operations). This would further enable having a shared foundation for program execution which could make cross-language interop even easier.
I hope this answers your question.
I know I have mentioned a variety of ideas which are not common among programming languages or compilers, but these are some of the design ideas I've had to come-up with in order to solve some of the challenges involved in achieving Lux's goals.