Technical design decisions

This page provides the elements that helped us shape Chamilo 2.*

Database structure

The Database structure of Chamilo 1.* has been modified over time but essentially matches the structure of the Claroline database back in 2001 with a major renaming in 2005-2006 (to make table names more meaningful) and a big refactoring in 2012 around the release of Chamilo 1.9 (to remove the one-database per course logic).

iid and a lacking implementation

One of the flaws Chamilo 1.10 and 1.11 (and all their minor versions) have is that we have defined, for all course-tool-related tables, a field called "iid" that uniquely identifies any resource in the whole table (meaning across all courses) whereas previous versions relied on a combination of the fields c_id + id to uniquely identify them (the id was specific to the course and the same "id" value could be found several times in the same table, only to be differentiated by the c_id).

This, in itself, was a good move: we can now identify each resource more efficiently through tables indexing of the iid field. However, that implementation was not completed, in that we didn't get rid of the use of the "id" field everywhere in the code: some code snippets still depend on id + c_id to find a resource. This has led to numerous issues being reported and some breaking changes when upgrading from one version to another. Why we didn't completely get rid of the "id + c_id" logic is entirely due to the lack of human resources to do all the changes needed before we released 1.10.0 and 1.11.0. Obviously, Chamilo 2.0 will get definitively rid of the id + c_id logic, to avoid pulling this issue along much longer.

iid rather than uid

If we wanted to use a new field as "unique id", why would we call it "iid" instead of "uid"? The answer boils down to the fact that we want Chamilo 2.0 (or later versions) to support many database systems, and that UID is a reserved keyword for Oracle.

Files management

Files structure

Files management has always been a bit of an issue in Chamilo. For efficiency and ease of use, we never wanted to put files as blobs inside the database. We believe the database should be fast even on small servers, and as such that we prefer to let the filesystem deal with files reads and writes than adding the extra stress to the database of delivering high weight files to the remote user.

As such, files are stored on disk. However, our implementation has been... evolving over the years (Chamilo exists since 2010 and several in our team have worked on its ancestor Dokeos for several years before that, making our team's total history inside the project more than 15 years old already).

For example, when we developed the social network inside Chamilo, we were faced with the issue that other files were always linked to a course, while files uploaded on the social network were not. As such, we decided (because course files were initially stored in a /courses/ directory at the root) to store them somewhere else. They ended up in /main/upload/users (main/ is where the main code of the software lies), and then moved to /app/upload/users.

The same can be said about the files used to illustrate the homepage, which initially ended up in the /home/ directory and later on in /app/home.

We did succeed in re-ordering it more cleanly over time, though (most user-uploaded files are now in /app/), but we still had issues requiring other directories to be writeable (for example CSS stylesheets uploaded afterwards are stored in /web/css/ in version 1.11).

In 1.11, for different reasons, several directories have to remain "writeable" by the web server: /app/, /web/css/, /main/default_course_document/images/ and optionally /main/lang/

This, in turn, led to impracticalities when cloud offerings started to appear, because they tend to complicate things when you have to access several different directories when using your application. For example, Amazon Web Services would require you to declare several block storage devices to store each of those separate directories in order for multiple EC2 instances to share them.

Files access method

Also, having a standardized method for files access (whereas currently we use several custom-developed methods to read or write files on disk) makes it easier to move to cloud-based infrastructures. Using a files management layer that supports REST-type services helps us deploy large scalable cloud systems that offer object files storage (like S3 in the case of AWS).

This is why we decided to use VichUploaderBundle starting from Chamilo 2.0. This uploader considers several naming strategies, called "Namers". Chamilo can sometimes have considerable amounts of files in a specific directory. For example, some of our customers have more than 80K courses, meaning the /app/courses/ directory gets filled with more than 80K entries, which is commonly considered a bad idea (because of inodes management limitations). Some others have more than 600K student directories, meaning that (if it wasn't for our planning around the release of 1.10) we could have 600K entries in the /app/upload/users/ directory. Luckily, we thought about that early and spread those directories by prefix of the user ID (only one level, so we can still have about 200K entries in one single directory in some scenarii).

As such, we need a strategy that will allow us to have an ideal spread of files so as not to clutter the filesystem. Using hashed is a practical way to do that, as files will can be distributed randomly (and thus ideally) on the basis of the first characters of their hashes. However, we know for a fact that many Chamilo administrators still get onto the filesystem manually from time to time to find files or clean things up, and that having files replaced by hashes make it very complicated to find them based on what their name shows in Chamilo. This would suggest keeping part of the name inside the final converted filename.

As such, having a naming convention that contains both a hash and a file name would be even better.

Also, we want to avoid special characters in the original file name that would make it less manageable (Windows has some invisible spaces that are not the default ASCII invisible space, for example, and we've already had issues with several crazy stuff like that).

And this is exactly what VichUploaderBundle's SmartUniqueNamer namer offers.

Plugins as bundles

Coming soon(-ish)...