Open Source Software Primer: The Development Process

Involving yourself in the world of open source can be a bit daunting. "Open Source" doesn't just mean that the source code is open for all to see and contribute to, It also  means that the development process is open for all to see and contribute to. For closed source projects this process happens behind closed doors and if you've never been part of it before, open source might be your first exposure to the development process. And well... it can be a lot. Development is not an easy process and companies have all sorts of structures & supports in place to assist developers through it, and open source has none of those resources (typically). In particular open source projects don't have "Project Managers" or "Product Owners" in the typical sense of the word. In a closed source project these would be the people who decide which fixes and features to prioritise, as well as when things are being released. They are where the buck stops when it comes to big project decisions. In open source, these people essentially don't exist and it is the community who decides. That being said it is normal for there to be a few primary maintainers who sort of fill this role, but they do not wield the same absolute authority that you so often see in closed source development. 

This article will provide a (very) general overview of the entire development cycle with references to how it looks in both, closed and open source projects.

Dramatis Personae

There is a cast of characters in the development cycle that fill various roles. This cast differs between the two source types.

Closed Source

• Project Manager/Product Owner (PM)
• Team Lead
• Developers
• Quality Assurance Testers

Open Source

• Repository Maintainer/Admin
• Contributors

Part 1: Planning

The first step in the development process for both closed and open source is the planning stage. In this stage it is decided what needs to go into the next release. This typically consists of the highest authority on the project dictating what customers want, as well as the direction they think the project should go. There is usually come consulting with the developers on the project as well so that they aren't pushed into an impossible situation with no recourse. In closed source work there will often be direct requests from outside the team, maybe even from higher up the hierarchy. Sometimes customers will have paid extra for a very specific feature that you have to add, for example. The precise policy of how much influence external factors can have on the project can differ between organisations, and PMs.

In Open Source, the situation is very similar, except that it is the maintainer that makes these decisions and they are are influenced primarily by the community as a whole, which can include "customers" (users) and developers, who are usually also users. For example, if a contributor has already implemented a new feature, without having been asked to, and it works, the maintainer may feel a strong urge to put it in the next release because it's an easy win. This type of event is very uncommon in closed source development, but in Open Source where the actual number of developers is unknown at any one time, things like this can happen.

For both source types, the planning stage ends with a list of bugs to fix, a list of features to add, a timeline for achieving all of this and a set of people who will be doing the work. The big list of stuff is usually organised by priority as well, with issues (as they are known on GitHub) ranked from most important (Blocker) through to least important (Minor). Issues (or "tickets" that are to-do but not included in the kanban board are said to be in "the backlog".

Part 2: Writing Code

Time to write the code! At this stage, developers assign themselves to the issues and get to work fixing things and adding features as per the priority list. It is common that teams employ the "Agile" process and that some kind of Kanban Board is used by the PM and the developers to keep track of who is working on what and how the project is coming along. This looks like a series of columns that represent the stages of the development cycle. Note that a key difference between closed and open source projects is that in a closed source environment, someone will eventually be assigned to work on an issue. All issues remain accounted for. If something isn't done by the time of the release, then this will need to be addressed and devs will be on the line to answer why. Whereas, in open source if no one steps up to tackle an issue, it doesn't happen. Everyone is a volunteer, and that means sometimes things won't get done that need to. The dynamic release schedule is necessary to accommodate this.

The PyMAPDL Public Roadmap snapshot below shows what a Kanban-like board might look like. Issues are present like post-it notes that are moved across the board (left to right) as they are completed.

This part of the process does not differ significantly between closed and open source, although in open source it can be a lot more flexible as you usually are not working to a rigid deadline.

CI/CD also occurs at this stage, but that's a whole other article. Read it here: 

Introduction to CI/CD: GitHub Actions

Part 3: Quality Assurance

Once issues have been completed, how do you know the bug was actually fixed or the feature added properly? Enter the Quality Assurance testers. These guys test out the result to make sure everything is working as it should and nothing else has been inadvertently broken with the change.

Quality Assurance is a vital role in any software development cycle. In closed source development there are often dedicated engineers whose jobs are just this. They identify bugs and then they check that every bug that's been fixed has actually been fixed. They use the product in new and inventive ways to ensure it's robust, reporting any bugs they find. They do their level best to break the product and take it to its operational limits. They are the wind tunnels of software.

In many open source projects it is unusual to have any dedicated QA testers. The QA testers are typically the product users! Those of you who understand the value of QA testers may be sweating a bit at this revelation but trust me... it's (sort of) ok because of the dynamic development cycle.

It's OK (mostly) because open source projects are not on a fixed development cycle. If someone finds a bug today, then a fix can be implemented tomorrow and a new release with the fix patched in can go out the day after. This dynamic correction means there isn't as much pressure to check everything in the code before a release because anything you miss can be fixed and released within a week or so. Plus, if a user finds something that doesn't work/they don't like, they can raise an issue directly on the project and, depending on the severity, they don't have to wait several months until a fix is released. In fact, they can even go in and fix it themselves!

However, whilst the consequences of missing bugs aren't as severe as in a commercial development cycle, the need for human testers hasn't gone away. It just usually gets done by the existing developers. 

Part 4: Release

Finally we have the product's release. This is the same everywhere: the final version of the software is created and it is shipped to its final destination. In open source communities this can be a public package manager like PyPI for Python packages, and in closed source projects it usually means the installer is updated on the company website or bundled into a larger installer with other products.

Closing Comments

This is the development process/cycle but it can also refer more generally to the actual cycle that a single developer goes through. Typically the developers are all present for the planning and release parts, which is to say they happen simultaneously for all developers. However, each individual developer will go through the writing code and QA  steps at different rates Some issues take hours, some take multiple days. Issue completion happens asynchronously between developers.

And that's it! The development cycle is complete.