Posted on

Be Open From Day One, Not Day N.

Note: This is an updated version of an article I first wrote in 2011. The original site went offline for a while, and although it was later restored, thanks to heroic efforts by Philip Ashlock, I felt the article needed a new home, and I wanted a chance to update it anyway.

Over the years we’ve watched software projects of all sizes make the transition from closed-source to open source. The lesson we consistently draw from them is this:

If you’re running a software project and you plan to make it open source eventually, then just make it open source from the beginning of development.

Waiting will only create more work.

The longer a project is run in a closed source manner, the harder it will be to open source later.

(Note that being open source from the start doesn’t force you to immediately take on the extra responsibility of community management. People often think that “open source” means “strangers distracting my programmers with questions”, but that’s optional — it’s something you might do down the road, if and when it makes sense for your project. It’s under your control.)

Projects seem to follow the rule pretty convincingly: the longer they run closed-source, the more difficult they are to open up later. Other people we’ve talked to have noticed the pattern too.

Why does it seem to hold so consistently?

I think there’s one underlying cause:

At each step in a project, programmers face a choice: to do that step in a manner compatible with the future open-sourcing, or do it in a manner not compatible with the future open-sourcing. Each time they choose the latter, the project gets just a little bit harder to open source.

The crucial thing is, you can’t help choosing the latter occasionally — all the pressures of development propel you that way. It’s very difficult to give a future event the same present-day consequences as, say, fixing the incoming bugs reported by the testers, or finishing that feature the customer just added to the spec. Also, programmers struggling to stay on budget will inevitably cut corners here and there (that is, they will incur technical debt), with the intention of cleaning things up later.

Thus when it’s time to open source, you will suddenly find:

  • Customer-specific configurations and passwords checked into the code repository;
  • Sample data constructed from live (and confidential) information;
  • Bug reports containing sensitive information that cannot be made public;
  • Comments in the code expressing perhaps overly-honest reactions to the customer’s latest urgent request;
  • Archives of correspondence among the developer team, in which useful technical information is interleaved with personal opinions not intended for strangers;
  • Licensing issues with dependency libraries whose conditions might have been fine for internal deployment (or not even that), but aren’t compatible with open source distribution;
  • Documentation written in the wrong format (e.g., that proprietary internal wiki your department uses), with no easy translation tool available to get it into formats appropriate for public distribution;
  • Non-portable build dependencies that only become apparent when you try to move the software out of your internal build environment;
  • Modularity violations that everyone knows need cleaning up, but that there just hasn’t been time to take care of yet;
  • Need I go on? Do some of these sound familiar?

The problem isn’t just the work of doing the cleanups; it’s the extra decision-making they require. For example, if sensitive material was checked into the code repository in the past, your team now faces a choice between cleaning it out of the historical revisions entirely, so you can open source the entire (sanitized) history, or just cleaning up the latest revision and open-sourcing from that (sometimes called a “top-skim”). Neither method is wrong or right — and that’s the problem: now you’ve got one more discussion to have and one more decision to make. In some projects, that decision gets made and reversed several times before the final release. The thrashing itself is part of the cost.

Waiting Just Creates an Exposure Event

The other problem with opening up a completed code base is that it creates a needlessly significant exposure event. Whatever issues there may be in the code (modularity corner-cutting, security vulnerabilities, etc), they are all exposed to public scrutiny at once — the open-sourcing event becomes an opportunity for the technical blogosphere to pounce on the code and see what they can find.

Contrast that with the scenario where development was done in the open from the beginning: code changes come in one at a time, so problems are handled as they come up (and are often caught sooner, since there are more eyeballs on the code). Because changes reach the public at a low, continuous rate of exposure, no one blames your development team for the occasional corner-cutting or flawed code checkin. Everyone’s been there, after all; these tradeoffs are inevitable in real-world development. As long as the technical debt is properly recorded in FIXME comments and bug reports, and any security issues are addressed promptly, it’s fine. Yet if those same issues were to appear suddenly all at once, unsympathetic observers might jump on the aggregate exposure in a way they never would have if the issues had come up piecemeal in the normal course of development.

The importance of avoiding a needless exposure event is especially true for government projects, even more than for private-sector code. Elected officials and those who work for them are understandably sensitive to negative publicity, and are therefore risk-averse. Even if your team has been very conscientious, a worrying cloud of uncertainty will surround everything by the time you’re ready to open up hitherto closed code. How can you ever know you’ve got it all cleaned up? You do your best, but you can never be totally sure some hawk-eyed hacker out there won’t spot something embarrassing after the release. The team worries, and worry is an energy drain: it causes them to spend time chasing down ghosts, yet at the same time can cause them to unconsciously avoid constructive steps that might risk exposing real problems.

(For private-sector code, there are sometimes competitive reasons to stay closed until the first release, even if the project is intended to be open source in the long run. This is not an exception to the advice given here, it is simply a countervailing factor that should be weighed along with every other strategic consideration related to the project. One temptation to be suspicious of, however, is the notion that creating an exposure event is actually desirable — that dramatically open sourcing the code can itself provide useful publicity. In almost all cases, the better way to achieve that publicity would be by announcing the software’s “1.0” release later on. The feature set and product stability that that event represents is what users and partners actually care about, and that announcement gives potential contributors another natural moment to take interest, or renew their interest, in the open source code base.)

The Good News

The good news is that these are all unforced errors. A project incurs little or no extra cost by avoiding them in the simplest way possible: by running the project in the open from Day One.

“In the open” means the following things are publicly accessible, in standard formats, from the first day of the project: the code repository, bug tracker, design documents, user documentation, wiki (if any), and developer discussion forums. It also means the code and documentation are placed under an open source license, of course. It also means your team’s day-to-day work takes place in the publicly visible area (except for sensitive configuration data and the like — that of course stays behind your firewall).

“In the open” does not have to mean: allowing strangers to check code into your repository (they’re free to copy it into their own repository, if they want, and work with it there); allowing anyone to file bug reports in your tracker (you’re free to choose your own QA process, and if allowing reports from strangers doesn’t help you, you don’t have to do it); reading and responding to every bug report filed, even if you do allow strangers to file; responding to every question people ask in the forums (even if you moderate them through); reviewing every patch or suggestion posted, when doing so may cost valuable development time; etc.

Think of it this way: you open source your code, not your developers’ time. One of those resources is infinite, the other is not. You’ll have to determine whether engaging with outside users and developers makes sense for your project or not. In the long run it usually does, when done with care — different types of projects want different types of engagement. But the important thing is, it’s all under your control. Developing in the open does not change the degree of control you have over the project, it just ensures that everything you do is, by definition, done in a way that’s compatible with being open source. And you get that for free.

If you want your software to be on-time, on-budget, feature-complete, and open source, then just develop it the way you normally would but with everything open source from the start. You’ll still only get two of the first three, of course — there’s no escaping the Project Management Triangle. But you’ll be taking the most efficient and effective route to being open source, and the project will be better off for it.

Posted on

Test Sandbox Post — Please Ignore.

If you don’t work for Open Tech Strategies, but you somehow landed on this blog post, then, uh, welcome to the Bat Cave! Yeah! This is where the magic happens! Few are called, but many are chosen… or something like that.

We just needed a sandbox for testing things like annotation plugins etc. This post is that sandbox. It was published using Stealth Publish, so it should never show up on our front page or in our RSS feed.

I guess three paragraphs is enough for a test post. We’ll leave it at that for now.

Posted on

Open Source Code of Conduct for Commercial Entities (DRAFT)

Note: This is a draft of a Code of Conduct meant to help a specific open source project give guidance to its commercial participants. The project is already in production use, and is successful enough that some commercial entities have become involved, offering support, hosting services, customization, etc. However, those companies need some guidelines about how to conduct themselves, in relation to the project as a whole and in relation to each other.

The first half of the draft is aimed at commercial entities. The second half of the draft contains guidelines for the open source project itself — healthy commercial participation being a two-way street.

Once the text is finalized, we plan is to post this in generic form, as a template that other projects can use, while delivering a customized version to that project.

In the meantime, comments welcome! You can simply leave regular blog comments, but we’ve also enabled the open source WordPress annotation plugin to allow sidebar annotations of selected text. To use it, just mouse-select any passage of text, as though you were going to copy it to the clipboard, and then wait for the annotation action buttons to pop up right under your selected text. There should be two of them: “Annotate” and “Highlight”. Choose “Annotate”, and then, at the top of the right-hand sidebar that should now open up, create a free account at, or sign in if you already have an account. (We need the authentication step to help prevent spam annotations.) Once you are signed in, you can leave a sidebar comment associated with a specific passage of text — the user interface should be pretty clear from this point on. Please note that your comments will be publicly visible by default; you can also make an annnotation that’s private to yourself, but then we wouldn’t be able to see it either of course.

Open Source Code of Conduct for Commercial Entities (DRAFT)

The [Your Project Here] Project welcomes involvement from commercial entities. The availability of multiple sources of commercial support is important to the project’s long-term health, and to making [Your Project Here] a persuasive choice for organizations around the world. Commercial support can include but is not limited to hosting, on-site deployment and installation, advocacy, PR and event funding, administrative services, customization, and even implementation of new features.

This code of conduct provides guidance for commercial entities wishing to participate in the [Your Project Here] ecosystem. While the specific items in the code may be revised from time to time, overall the code is based on a simple principle:

Commercial entities should observe the distinction between public community functions and private commercial functions, and ensure that their activities support (or at least do not harm) the public community.

The first half of these guidelines is meant for commercial entities, but the second half is meant for the open source project as a whole, since a commercially healthy open source environment depends on everyone doing their part. Some of the guidelines are proscriptive, being based on situations that have arisen in the real world — in [Your Project Here] and elsewhere — but we also include some positive recommendations that can help commercial entities participate successfully.

Guidelines for Commercial Entities

  • Treat the project’s name with respect, and follow the project’s trademark and branding policy.

    Don’t use the project’s name for things which are not related to nor supported by the project.  In general, don’t introduce confusion about what the project’s name means and refers to. Also, don’t seize the project’s name in other public namespaces, e.g., in domain names, Twitter, GitHub, Slack, etc. (It is fine, however, to register the project’s name in a new namespace and then immediately donate control of that account to the project — that’s simply helping the project maintain its identity.)

  • TBD: [Your Project Here] needs a trademark policy, for this item to be effective. See the item Have and enforce a formal trademark and branding policy in the section Guidelines for the Open Source Project for more about this.

  • Don’t replace community infrastructure; instead, improve it.

    For example, use the project’s discussion forums as specified in their charters, instead of setting up duplicate forums in a privately-controlled space that could potentially siphon off people from the project’s existing forums. I.e., don’t set up “walled gardens” that detract from public spaces. The same principle applies to other kinds of community infrastructure too, e.g. wikis, but we’ll use discussion forums as an example here.

    It is fine to set up private forums that are specific to commercial offerings, especially where those offerings differ from what is available in the open source project (though see below about avoiding the “enterprise edition” trap). Discussion that is specific to a company’s commercial offering is only useful to that company’s customers. But do not try to discourage customers from making use of the project’s public forums when doing so would serve their needs.

    A good way for a company to ensure compliance with this guideline is to link to the relevant project forums from the same places — and in comparable ways — that the company links to its product-specific forums. That way customers will be aware of all the options available to them and can choose the appropriate forum for their needs. It also means that customers’ questions, and the company’s or others’ responses to them, can contribute to the public store of knowledge about the project, which is an important way that commercial activity can support the long-term health of the project. Company representatives should include their affiliation in the support responses they make to customers or to anyone else; work the company performs publicly is still work that it did, and the company should get credit for its contributions.

  • Label the company’s offerings in a way that makes their provenance clear, and that does not denigrate or diminish the open source project.

    Specifically, do not label a proprietary offering as the “Commercial Edition” or “Enterprise Edition” of the project, especially not in contrast to a so-called “Community Edition” as a euphemism for the project’s open source code. That kind of marketing implies that the open source edition is somehow not commercial (which would be untrue, as the open source license explicitly allows commercial activity by anyone) or not enterprise-ready (unlikely to be true, but in any case ill-defined).

    Instead, give the company’s offerings their own distinctive names. If those names incorporate the name of the project in some way, that may be fine so long as it complies with the project’s trademark guidelines. It is also fine to offer fact-based comparisons between the company’s proprietary offering and the stock open source version, or between this company’s offering and others’ proprietary offerings. Or if the offering is also open source software, but differs from the project’s offering in some significant way, explain that, and again make sure the company’s offering has a name that clearly distinguishes it from the code released by the project.

    The purpose of this guideline is to help the company communicate clearly about what it offers, and to help potential customers understand what differentiates this company’s offerings from what is available elsewhere. There is nothing wrong with offering features not present in the project’s open source releases, or with making different configuration choices from the project’s defaults. Just describe the differences accurately and objectively. Mislabeling, whether accidental or deliberate, causes confusion and is bad for the project’s health.

  • Do not attempt to convert unofficial influence into claims of official control.

    Even if the company is a major contributor to the open source project, do not conflate the project’s identity with the company’s. If a company with a well-established position in the project casts too large a shadow, that may discourage involvement from others.

    For example, the company might have a number of employees as core committers in the project, and thus exercise a significant de facto influence over the project roadmap, simply by doing a lot of the work and participating actively in project development discussions. This is generally not a problem, as long as the company does not claim some official preferential position in project governance that it does not in fact hold.

  • In public forums, the company’s employees should behave like any other project participants.

    While management hierarchy may be important internally to the company, it is irrelevant to the public open source project. For example, if the company assigns an employee specifically to work on the project, expect that employee’s contributions to still go through the usual review procedures, and expect that employee to gain commit access by the same route as anyone else would. Similarly, it is good practice for the company’s employees to hold design discussions or other technical discussions in the project’s public development forums, even if the employees normally sit in the same room and could discuss those things in person. The more they participate visibly in the project, the stronger the ties between the company and the project will be.

  • Contribute to public activity, and avoid converting public conversations to private ones.

    When commercial representatives are active in an open source project’s public forums, there will be many opportunities to turn public conversations into sales opportunities. This can be a good thing — often a potential customer will benefit from being contacted about a commercial offering, and commercial entities should feel free to establish such contact so long as they avoid harassing or spamming people. In general, if some user indicates needs that would be met by the company’s product, it is reasonable to contact them about that directly, as long as the company is respectful of the charter of the forum in which it encountered the user and of any preferences they give about unsolicited commercial communications.

    However, it is important to avoid shunting conversations out of public forums, where inquiries and responses will remain visible to others, and into private forums, where the initiator would be isolated from other sources of information. The best way to avoid this anti-pattern is to distinguish carefully between answering questions and offering commercial services. The former should always be done via a public followup in the original forum where the question was asked, while the latter should happen via a separate private communication. A topic that started in public should remain public, and should not be interfered with or subsumed by private conversations.

  • Improve project documentation, don’t fork it.

    Help make the project’s public documentation better, rather than duplicating and extending that documentation elsewhere. Even if the duplicate documentation is publicly accessible even to non-customers, it will still detract by its mere existence from the project’s own documentation, among other things by causing confusion in Internet search engine results.

    It is fine for the company to maintain separate documentation for functionality that is specific to its product or service. But as much as possible, avoid duplicating material already present in the project’s documentation. Instead, refer to the project’s documentation, and, as much as necessary, participate in improving it and making it easier to refer to, so that the maintenance burden is reduced for the company and everyone else.

  • Any restrictions in non-compete agreements should only affect business activity, not project activity.

    If the company requires employees or contractors to sign non-compete agreements, those agreements must not prevent the individual in question from participating in the open source project in any way, whether during or after the term of their employment.

    This does not mean that all non-compete agreements are incompatible with this code of conduct. A company may restrict an employee’s ability to solicit the company’s customers, for example. The key is that any restrictions should be about things other than project participation itself. An individual should not be blocked from any form of technical or social participation in the project, including the implementation of particular features.

    The accumulation of experience and expertise in individual persons, who are ultimately free to direct their energy and attention as they decide, is one of the most important drivers of progress in open source projects. A company that limits this freedom can damage the open source project very seriously.

  • Security vulnerability information should always be promptly disclosed to the project.

    If a commercial entity learns about a security vulnerability in the open source code, it should always promptly disclose that information to the project using the project’s designated channel for reporting such vulnerabilities. (Of course it is acceptable to pre-patch the company’s own offerings, as long as that patching does not significantly delay the reporting of the vulnerability.)

    Vulnerability information should never be used for unilateral commercial advantage. Vendors may legitimately compete on the speed and reliability with which they deploy security fixes, but withholding vulnerability information damages everyone in the long run by risking harm to the project’s reputation and the security of all users.

  • Company staff should be forthright about the company’s commercial interests.

    There is no need to hide motivations in an open source project. It is very normal for developers to propose that a certain capability be added to the code and justify the proposal on the grounds that it would help their employer’s commercial interests. The best situation for the project is one in which developers communicate their priorities clearly and straightforwardly. As long as the proposed change goes through the project’s usual decision-making procedures, the fact that it would serve specific commercial interests is not only not bad for the project, it may often be good: after all, continued commercial interest in the code is usually a good thing.

    Of course, it would be odd to make a technical proposal that is wildly divergent from the project’s current roadmap and from the interests of all other parties participating in the project, but in that case the proposal is likely to be rejected anyway. In general, there is no need to pretend to purity of motivations. Finding sustainable resolutions to the tensions between various parties’ needs is a long and well-established tradition in open source projects; as long as the company participates in those discussions in good faith, and compromises where appropriate, the influence of the company’s commercial motivations is likely to be an overall benefit to the project.

  • Consider having a designated contact person or ombud, to represent community concerns internally.

    Depending on the size of the company and the degree of its involvement in the open source project, it may wish to officialize a role for representing community concerns internally. In practice, this role is often assumed informally anyway, usually by an engineer or community manager who works for the company but who is active in and invested in the project. What the company can do is make sure this person feels comfortable raising project concerns internally.

    Establishing that level of comfort is a matter of company culture, and obviously these guidelines cannot require nor enforce a particular corporate culture. All they can do is point out that the best way for executives to make informed decisions is by giving those closest to the project a chance to represent the project’s interests honestly and accurately.

    When this role is formalized, it is sometimes known as “Community Representative” or “Ombud” (or “Ombudsperson”, “Ombudsman”, etc). Whether this person’s identity is known publicly or not, they should be reachable via a public contact address, with a prior commitment of confidentiality, so that others in the project have a way to raise concerns privately with the company when they wish to do so.

Guidelines for the Open Source Project

  • Have and enforce a formal trademark and branding policy.

    Specify when and how the project’s name and logo can be used without explicit permission, versus what uses require permission from the trademark holder. Ensure that the trademark holder is an institution that can be trusted to act in the project’s long-term interests and that is not beholden to any particular commercial entities.

    Some examples of open source trademark policies are:


  • Maintain discussion forums.

    Maintain discussion forums for developers and users of the project, make sure each forum has a written charter, and take whatever steps are necessary (e.g., moderation) to ensure that each forum remains useful for the purposes stated in its charter. All public forums should be publicly archived and have open membership.

    The project may also maintain non-public forums, such as a security vulnerability reporting and discussion group, but these non-public forums should still have public written charters that state their membership policy. Membership in non-public groups should be individual, but may take affiliation into account: for example, someone may be granted access to a security group in part because of their role at a commercial hosting provider that wishes to stay up-to-date with pre-release security patches. However, the potential member’s discretion and trustworthiness should also be a factor, and they may retain their membership even if their affiliation with that commercial provider ends.

  • Have a written explanation of how commit access is granted.

    There should be no mystery about how a developer becomes one of the official maintainers of the project. Maintain a written explanation of how new committers are chosen in the project. The process can be as simple as “current committers vote privately to invite a new committer”; the important thing is to document it so that contributors whose work is commercially sponsored understand that the system is fair and that the potential for them to be invited to become a committer is dependent primarily on the technical merits of their work and on the constructiveness of their interactions with the rest of the project community, rather than on their commercial affiliations.

    A few projects do place a limit on the fraction of core maintainers that may be affiliated with the same organization at the same time; in general, this kind of limit should only be imposed as a last resort, when the community has tangible concerns about dominance of project direction by one entity. Remember that open source projects are voluntary social constructs — a project can always be forked, even by its founders. Also, if a trusted entity controls the project’s trademarks, then that is a powerful incentive to commercial interests to behave well.

    A good example of a simple but documented procedure for gaining commit access is this one from the LibreOffice project:

  • Maintain documentation, and make it easy for everyone to contribute to it.

    One of the most common commercial anti-patterns in open source projects is the phenomenon of a company maintaining a separate set of project documentation for its customers, often of higher quality than the project’s official documentation. The company should instead contribute to improving the project’s public documentation — but this implies a reciprocal obligation on the part of the project as a whole to being receptive to such documentation improvements. The project should view commercial support providers as a valuable source of information about deficiencies in project documentation: a company whose sales depend on customers having a smooth experience will be highly sensitive to the quality of documentation, and the project should try to leverage this sensitivity for everyone’s benefit.

  • Support and enforce this code of conduct publicly.

    The project should publicly recognize companies that abide by this code of conduct, and (after making a good-faith attempt to remedy any misunderstandings or problems) publicly chastise companies that violate the code.

    One method of publicly supporting companies who formally agree to abide by the code is to publish a list of such companies on the project’s web site. The same entity that enforces trademarks for the project is usually the right entity to maintain that list. Being listed on the project’s official site, as a commercial support provider who is certified as abiding by the project’s commercial guidelines, is a valuable advertisement; the potential for removal from that list then becomes a powerful incentive for companies to live up to their pledges.

    If a commercial entity is violating the code of conduct, representatives from the project may choose to contact that entity privately at first, to see if the problem can be resolved quickly — often such problems result from misunderstandings, particularly with companies who are unaccustomed to working in an open source environment. However, if the problem cannot be resolved to the representative’s satisfaction, then it is important to publicly state the code of conduct violation, documenting it thoroughly but without rancor, and emphasizing that the project welcomes participation from that entity so long as the entity complies with the code of conduct. This statement should be posted to the appropriate project discussion forum, so that the statement and any followup posts (either from the commercial entity or from other participants in the project) are visible and indexable by Internet search engines. The representative may, of course, choose to remove the commercial entity from the list of entities that abide by the code of conduct; such removals should be announced too.

  • Maintain a private contact address for reporting code-of-conduct violations.

    The project should maintain a private contact mechanism by which anyone may report violations of this code of conduct. It is up to the project to decide who receives and acts on such reports. There is obviously potential for conflicts of interest if representatives of commercial entities are among the recipients, but compliant commercial entities are also those most motivated to be responsive to such reports, since it is strongly in their interests to detect and stop an incipient arms race of bad behavior. One way to alleviate such conflicts is to either allow no representatives of commercial entities, or to have representatives from mulitple entities, so that there are enough participants to prevent any one participant’s interests from being overrepresented.

Posted on

OS IV&V: Independent Verification and Validation for Open Source

When you hire a development shop to build an open source product, you want to make sure the result is truly open source. You want to guarantee that:

  • The end product is independently deployable by others.
  • There are clear instructions for how to get involved.
  • Commercial third parties are welcome (because that’s usually where new development energy comes from).
  • There are no unexpected proprietary dependencies.
  • The developers respond constructively to bug reports.
  • There are procedures in place (as there should be for any software) for receiving sensitive security vulnerability reports.
  • The project is poised to become a multi-participant and even multi-vendor community.

However, often first forays into open source do not meet these goals — not because of bad intentions, but because vendors who are new to open source need some help.

Open Source IV&V provides vendors that help. An independent vendor specializing in open source works alongside the development vendor, playing the role of open source community from the start of the project. The IV&V vendor works with the development vendor out in the open, just as third-party participants would. By the time the first production release is ready, the development vendor knows how to navigate an open source project, technically and culturally.

OS IV&V helps expand the range of vendors you can consider hiring to do open source development, and it ensures that by the time the project reaches beta, there are at least two vendors who have technical and deployment knowledge of the code base.

The long version:

Traditionally, “Independent Verification and Validation” (IV&V)1 is a method of independent oversight for technical projects, in which a disinterested third party is brought in to test that a project’s results fully meet the specifications as set out by the client. It is most commonly used with private-sector contractors delivering technical work to a government agency. For example, NASA has been using IV&V arrangements for many years to ensure that its contractors — and its own engineers — are meeting safety and cost-effectiveness requirements.

Open Source IV&V” is designed specifically for open source software projects, and its purpose is to not only ensure that the end product is legitimately open source software, but that the project is created and managed in a sustainably open source way, which sometimes can include changes in how the customer and contractor relate to each other.

We have gradually designed the OS IV&V process through the repeated experience of being brought into projects after development is already under way with a primary development vendor — often one that does not have extensive experience with open source practices — and then attempting to retroactively transform the work into a real open source project, with all the benefits open source can bring.

In open source software, the result of a project is not merely the software code itself. It’s also the ecosystem around the code: all the publicly-visible discussions between developers and users, all tickets in the bug tracker (each ticket recording not just a technical fact about the software, but also a conversational dynamic between different people at different organizations), all the feedback from both automated and manual testing, contributions from interested third parties (some of these contributions may be already incorporated, others may be still pending review), a project roadmap reflecting input from multiple sources, documentation not just about using the software but about how to contribute to the software, etc.

The health of this larger ecosystem is crucial to the long-term success of the project, and to reaping the benefits of open source. What OS IV&V verifies and validates is not just the technical components of a project — code, licensing, documentation, open source collaboration infrastructure, etc — but also the contractor/customer relationship: how communication happens, how they bring in third-parties as the project grows, how they handle user feedback, etc. The goal of OS IV&V is help the customer and the development contractor structure their working relationship in a way that is compatible with the project flourishing as an open source ecosystem.

Why is OS IV&V necessary?

When a software contractor who has little or no experience running open source projects agrees to create an open source product for a customer (say, for a government agency), the contractor usually fails to adopt standard open source processes from the beginning of the project. For example, the contractor often does not ensure that development happens in the open from the start, is not prepared to integrate commercial or volunteer third-party involvement, does not use typical open source tools and procedures to receive and handle user feedback publicly (nor have proper procedures in place for receiving security vulnerability reports), does not post clear instructions for how to join the development effort, does not encourage its developers to be responsive to genuine technical inquiries, etc.

This is because most primary development contractors are accustomed to doing software development entirely in-house, and simply continue to use in-house techniques even after taking on an open source contract. They use an internal bug-tracking system from which tickets cannot be made public, they use internal code repository servers because that’s what their developers are set up to use, they write documentation in proprietary formats aimed at a primarily in-house audience, and so on. Most importantly, they and their customers tend to communicate almost exclusively in private about the projects they work on together. A typical workflow is that the customer submits feedback by sending private email to some designated contact point at the contractor, which results in a private ticket being filed in the contractor’s bug-tracking system; the customer may know the bug ticket number but often not even have read-only — let alone read-write — access to ticket itself.

To get the full benefits of open source, the contractor needs to adopt practices that may be unfamiliar, and the customer and contractor need to use an open source approach to their own communications around the project. A project that is run in a truly open source manner will be much better able to engage third-party interest, which in turn makes innovation by other companies much more likely, and naturally lowers the customer’s risk of vendor lock-in. It also, in our experience, results in higher-quality, more easily maintainable software.

The role of the OS IV&V vendor is as much to assist both sides in making these adjustments, as it is to verify and validate that the adjustments have been made. A successful OS IV&V engagement results in a new way of working that is repeatable for future projects.

Elements of OS IV&V

Although OS IV&V is as much an art as a science, we have done enough of it now to extract some key ingredients. An OS IV&V vendor should ensure that:

  • The project is conducted in the open from day one:

    • Source code checked into a public repository (e.g., at GitHub)

      • Corollary: The public repository has to be the master repository. Make sure there is not some internal private repository where the real development is happening, with tranches of commits occasionally exported in a throw-it-over-the-wall fashion to the public repository.

    • Public bug-tracker

      • As with the repository, the public bug tracker must be the primary bug tracker.

    • Public documentation

      • There is often a wealth of necessary documentation that the contractor considers to be private deployment documentation, but that is actually necessary for anyone trying to deploy the project, even just for development purposes. The OS IV&V vendor needs to sniff that out and help that stuff see the light of day.

  • Communications forums are available and configured according to open source norms (e.g., publicly archived, publicly subscribable via email or web, etc)

  • The appropriate people (from the contractor and the customer) are responsive in those forums

    • Corollary: Ensure that conversations don’t get silently sidetracked into private forums where they are inaccessible to the rest of the project; make an exception for when privacy is actually necessary, of course.

  • The project does not acquire any proprietary dependencies except those explicitly required (if any) by its specifications, and the code is compatibly licensed for outbound distribution under an open source license of the customer’s choice.

  • The customer is the copyright owner of the code, unless otherwise specified, and is named as such in the copyright notices in the source tree.

  • Development procedures are properly documented, and follow open source norms

    • A third-party developer should be able to deploy the software, for testing or for development sandboxing, using only the instructions and sample data provided by the public project.

    • There is a documented path for receiving and reviewing incoming third-party contributions, whether commercially solicited, volunteer, or due to some other motivation. This includes establishing DCO (developer certificate of origin) and/or CLA (contributor license agreement) procedures as needed.

    • Automated testing and continuous integration are set up in such a way that they are available to all developers, not just to the primary development contractor.

  • Deployment procedures are thoroughly documented and follow open source norms.

    • Sample data is available, in the formats developers would expect.

  • Secure coding practices and procedures for handling vulnerability reports are in place and are followed.

  • The primary development vendor and the customer use standard OSS communications channels for as much of their communications as possible.

  • The project is compatible with any relevant open standards, or at least acknowledges the relevance of those standards and has a long-term plan for compatibility.

  • The project has the governance policies it needs at every stage. This includes both ensuring that governance structures are not baked in too early, which would risk stifling potential early contributors with too much bureaucracy, and making sure that reasonable governance structure is added as more participants join the project.

  • Normal open source practices for receiving, recording, and handling user feedback are documented and followed.

    • This includes feedback from the primary customer: it should be submitted and processed using the same channels as any other feedback. The development contractor may prioritize it differently from other feedback, of course, but that does not change where and how the customer’s feedback is submitted.

  • Potentially adverse public events are anticipated (or swiftly detected) and handled.

    • Assistance with planned events, such as launch announcements, to ensure that they keep the audience of potential contributors in mind.

    • Assistance with unplanned events, such as the discovery of a “zero day” security vulnerability, a data leak, etc.

  • Hackathons and other events that invite the public to become more involved in the project are conducted effectively. The exact assistance an OS IV&V vendor provides here can vary widely, depending on the experience and inclination of the primary development vendor, and can extend as far as the OS IV&V vendor being the chief organizer for such events, though of course the primary development vendor should still be included in whatever way they can be.

  • There is effective, high-quality mentoring provided if the project participates in the Google Summer of Code program2, the Outreachy program3, and other similar formal third-party contribution programs. Again, if the primary development vendor is inclined to take an active role in these programs, then the OS IV&V vendor can step back and merely provide assistance where needed; but the OS IV&V vendor can also directly arrange and mentor participation in the program if the customer and the primary development vendor agree.

  • The project publishes estimates of deployment costs and monthly marginal costs, for hosting of typical datasets. These estimates help facilitate involvement by other commercial entities, and are difficult for the primary development vendor to provide, both because the development vendor often designed the software around skills or hosting configurations they already had in place (and are thus unsuited to estimate a newcomer’s adoption of), and because their potential conflict of interest is too strong in any case.

  • Deployed code matches public code. That is, when the primary development vendor is also the hosting provider, as is commonly the case, their hosting SLA must not serve as a de facto justification for a divergent internal fork. Any deployment-specific customizations must be propagated back to the public, “upstream” version in an appropriate fashion, except for traditionally non-distributed minor customizations such as configuration file parameters, passwords, etc.

  • Structuring an OS IV&V Relationship

    The OS IV&V vendor reports to customer, not to any of the development contractors, and only the customer decides whether the OS IV&V vendor is fulfilling their responsibilities. This is crucial: if the OS IV&V vendor were instead responsible to a development contractor, it could not provide the independent verification and unbiased advice that is the core of OS IV&V’s value.

    This does not affect contracting relationships. It is fine, for example, for an OS IV&V vendor to contract through the development vendor simply as a prime contracting vehicle, or for both to subcontract through some other vendor as the prime, as long as the reporting relationship is clear: OS IV&V reports to the customer and no one else.

    The OS IV&V vendor is also best positioned to assist the customer in higher-level planning related to the project, such as drafting extensions to the development contract, writing future RFIs and RFPs, and budgeting for post-deployment maintenance and enhancement, because the OS IV&V vendor has far fewer conflict of interests in these matters than the primary development vendor.

    The OS IV&V vendor can also assist with the presentation of the project to potential partners, including to the local development community (particularly useful in large municipalities), or to relevant open source or civic technology organizations. This also includes assistance with staffing and contractor evaluation, since the OS IV&V vendor’s work remains largely the same no matter what the balance is between customer-side in-house technical staff and the staff on the development contractor’s side. In general, OS IV&V may be thought of as an independent body whose role is evaluate other participants in the open source ecosystem.

    OS IV&V As Backup Support Vendor

    It is okay for an OS IV&V vendor to have the potential to become an independent provider of commercial support in the project themselves4. The best OS IV&V vendor is not just a management-consulting shop with the enough in-house development knowledge to provide technically competent oversight, but rather a software development shop in its own right, with direct experience performing the kinds of contracts that it now being asked to provide oversight for. This has several advantages. First, it’s easier to communicate credibly with the primary development vendor when one understands their line of work intimately. Second, it’s easier to anticipate the needs of potential third-party contributors when one has been in that position oneself. Third, from the customer’s point of view, part of the purpose of OS IV&V is that by the time of product launch, there will be two independent vendors who could deploy and support the software on a commercial basis. If the original development vendor is performing satisfactorily, then their contract can simply be extended. But if they are not, then there is an alternative readily at hand.


    1 See for more discussion of IV&V in general and its history.

    4 Though it is of course advisable to drop their OS IV&V role with a customer once they start providing direct project-related services to that customer.