OS IV&V: Independent Verification and Validation for Open Source

When you hire a development shop to build an open source product, you want to make sure the result is truly open source. You want to guarantee that:

However, often first forays into open source do not meet these goals — not because of bad intentions, but because vendors who are new to open source need some help.

Open Source IV&V provides vendors that help. An independent vendor specializing in open source works alongside the development vendor, playing the role of open source community from the start of the project. The IV&V vendor works with the development vendor out in the open, just as third-party participants would. By the time the first production release is ready, the development vendor knows how to navigate an open source project, technically and culturally.

OS IV&V helps expand the range of vendors you can consider hiring to do open source development, and it ensures that by the time the project reaches beta, there are at least two vendors who have technical and deployment knowledge of the code base.

The long version:

Traditionally, “Independent Verification and Validation” (IV&V)¹ is a method of independent oversight for technical projects, in which a disinterested third party is brought in to test that a project’s results fully meet the specifications as set out by the client. It is most commonly used with private-sector contractors delivering technical work to a government agency. For example, NASA has been using IV&V arrangements for many years to ensure that its contractors — and its own engineers — are meeting safety and cost-effectiveness requirements.

“Open Source IV&V” is designed specifically for open source software projects, and its purpose is to not only ensure that the end product is legitimately open source software, but that the project is created and managed in a sustainably open source way, which sometimes can include changes in how the customer and contractor relate to each other.

We have gradually designed the OS IV&V process through the repeated experience of being brought into projects after development is already under way with a primary development vendor — often one that does not have extensive experience with open source practices — and then attempting to retroactively transform the work into a real open source project, with all the benefits open source can bring.

In open source software, the result of a project is not merely the software code itself. It’s also the ecosystem around the code: all the publicly-visible discussions between developers and users, all tickets in the bug tracker (each ticket recording not just a technical fact about the software, but also a conversational dynamic between different people at different organizations), all the feedback from both automated and manual testing, contributions from interested third parties (some of these contributions may be already incorporated, others may be still pending review), a project roadmap reflecting input from multiple sources, documentation not just about using the software but about how to contribute to the software, etc.

The health of this larger ecosystem is crucial to the long-term success of the project, and to reaping the benefits of open source. What OS IV&V verifies and validates is not just the technical components of a project — code, licensing, documentation, open source collaboration infrastructure, etc — but also the contractor/customer relationship: how communication happens, how they bring in third-parties as the project grows, how they handle user feedback, etc. The goal of OS IV&V is help the customer and the development contractor structure their working relationship in a way that is compatible with the project flourishing as an open source ecosystem.

Why is OS IV&V necessary?

When a software contractor who has little or no experience running open source projects agrees to create an open source product for a customer (say, for a government agency), the contractor usually fails to adopt standard open source processes from the beginning of the project. For example, the contractor often does not ensure that development happens in the open from the start, is not prepared to integrate commercial or volunteer third-party involvement, does not use typical open source tools and procedures to receive and handle user feedback publicly (nor have proper procedures in place for receiving security vulnerability reports), does not post clear instructions for how to join the development effort, does not encourage its developers to be responsive to genuine technical inquiries, etc.

This is because most primary development contractors are accustomed to doing software development entirely in-house, and simply continue to use in-house techniques even after taking on an open source contract. They use an internal bug-tracking system from which tickets cannot be made public, they use internal code repository servers because that’s what their developers are set up to use, they write documentation in proprietary formats aimed at a primarily in-house audience, and so on. Most importantly, they and their customers tend to communicate almost exclusively in private about the projects they work on together. A typical workflow is that the customer submits feedback by sending private email to some designated contact point at the contractor, which results in a private ticket being filed in the contractor’s bug-tracking system; the customer may know the bug ticket number but often not even have read-only — let alone read-write — access to ticket itself.

To get the full benefits of open source, the contractor needs to adopt practices that may be unfamiliar, and the customer and contractor need to use an open source approach to their own communications around the project. A project that is run in a truly open source manner will be much better able to engage third-party interest, which in turn makes innovation by other companies much more likely, and naturally lowers the customer’s risk of vendor lock-in. It also, in our experience, results in higher-quality, more easily maintainable software.

The role of the OS IV&V vendor is as much to assist both sides in making these adjustments, as it is to verify and validate that the adjustments have been made. A successful OS IV&V engagement results in a new way of working that is repeatable for future projects.

Elements of OS IV&V

Although OS IV&V is as much an art as a science, we have done enough of it now to extract some key ingredients. An OS IV&V vendor should ensure that:

Structuring an OS IV&V Relationship

The OS IV&V vendor reports to customer, not to any of the development contractors, and only the customer decides whether the OS IV&V vendor is fulfilling their responsibilities. This is crucial: if the OS IV&V vendor were instead responsible to a development contractor, it could not provide the independent verification and unbiased advice that is the core of OS IV&V’s value.

This does not affect contracting relationships. It is fine, for example, for an OS IV&V vendor to contract through the development vendor simply as a prime contracting vehicle, or for both to subcontract through some other vendor as the prime, as long as the reporting relationship is clear: OS IV&V reports to the customer and no one else.

The OS IV&V vendor is also best positioned to assist the customer in higher-level planning related to the project, such as drafting extensions to the development contract, writing future RFIs and RFPs, and budgeting for post-deployment maintenance and enhancement, because the OS IV&V vendor has far fewer conflict of interests in these matters than the primary development vendor.

The OS IV&V vendor can also assist with the presentation of the project to potential partners, including to the local development community (particularly useful in large municipalities), or to relevant open source or civic technology organizations. This also includes assistance with staffing and contractor evaluation, since the OS IV&V vendor’s work remains largely the same no matter what the balance is between customer-side in-house technical staff and the staff on the development contractor’s side. In general, OS IV&V may be thought of as an independent body whose role is evaluate other participants in the open source ecosystem.

OS IV&V As Backup Support Vendor

It is okay for an OS IV&V vendor to have the potential to become an independent provider of commercial support in the project themselves⁴. The best OS IV&V vendor is not just a management-consulting shop with the enough in-house development knowledge to provide technically competent oversight, but rather a software development shop in its own right, with direct experience performing the kinds of contracts that it now being asked to provide oversight for. This has several advantages. First, it’s easier to communicate credibly with the primary development vendor when one understands their line of work intimately. Second, it’s easier to anticipate the needs of potential third-party contributors when one has been in that position oneself. Third, from the customer’s point of view, part of the purpose of OS IV&V is that by the time of product launch, there will be two independent vendors who could deploy and support the software on a commercial basis. If the original development vendor is performing satisfactorily, then their contract can simply be extended. But if they are not, then there is an alternative readily at hand.