All your data are belong to u̶s̶ you
A story about how not to get fucked over by your software contractors
In the late 80’s there was a game exported from Japan to the US. As part of this export the game was translated from Japanese to English… Mostly. In one particularly famous point in the game the phrase, something like “We’ve taken all your bases!” had been translated “All your base are belong to us”
Kind of a funny mistake, but recently we at Sitewards started hearing stories of actual situations in which peoples eCommerce software and infrastructure were being held hostage by companies over some sort of contract dispute or other. These disputes can hold hostage millions of dollars and years worth of labour, in some cases entire business hostage to the contractors demand. This is an extremely difficult position to negotiate out from as a merchant, even to move to another provider to continue the contract.
The inherent information asymmetry
The problem that faces you as the merchants is the inherent difficulty in understanding exactly what is being delivered when contracting web software development. By nature if you are contracting out this service to a provider then you do not have the necessary time or expertise to fully implement the required service. You trust the provider to implement a set of reasonable services considering the merchants budget, but have little way to verify if these decisions are reasonable.
Once the provider is in control of the service they are in a position of tremendous power over you, particularly if the online revenue stream is a significant part of your business. The provider can fail to live up to their obligations under the contract, force changes to the contract or simply dodge questions and construct understanding to ensure that you understand “this project is simply expensive” and not “this provider is incompetent”.
A relationship spirals out of control
When the relationship between a provider and your business is collaborative and there is a healthy conversation back and forth about requirements, costs and so forth software development is a fairly straight forward process. Software development can be a little tricker and less predictable than some other disciplines, but mostly it’s pretty stable.
However, when deadlines start slipping, or when a service is not performing as expected or when budgets have been far exceeded and communication has not been as up front and honest as it needed to be, both parties in this relationship will start to hold the other accountable, and attempt to bring the situation in some form of control.
It is here where the power asymmetry between the provider and you as the merchant makes it extremely difficult for you to negotiate successfully. You can withhold funds for a service, but in many cases a software provider can switch off that service, and refuse to allow the access required to transport the service from one provider to another. Additionally, even if they do provide some level of access, it may not be enough to have the service performing as well as it was before, and thus makes it more expensive for another provider to manage this service.
Forewarned is forearmed
Perhaps the best thing you can do when calculating the risks of using a given provider is to determine what level of control over the application is delegated to the provider, and what level of control remains with you. By systematically reducing the amount of power a provider has over the content you can negotiate from a much more powerful position in the case there is some ambiguity about whether a software provider is living up to their expectations.
There is many different kinds of data produced to make a web based software system work. Below are some of the different things that you should have the ultimate control over, delegating only the control required to run it to the software provider.
Compute infrastructure is the underlying hardware (or virtualised hardware) on which your service operates. This could include things like:
- Virtual machines
- DNS services
As well as many other types of operational tooling. This is what actually powers the service, and is generally the “ultimate control” over the service.
A fundamental part of modern software development is the use of version control tooling, the most common being a tool called “git”. This allows many people to work on a project over very long periods of time despite the complexity of working over a million lines of constantly changing code.
It allows developers go back and forth through the history of development to understand and make decisions, as well as to coordinate their development for releasing the next application version.
Both the application itself and the underlying services (such as those that bring up the compute infrastructure) are often in version control. It is good to have at least an always up to date copy of this, however it is also possible to use a centralised service such as GitHub to remain in 100% control of the version control.
Continuous delivery pipeline
If the project is sufficiently advanced it will have an automated release process that takes the content from version control and builds from that your fully functioning software application. While not critical, control of this pipeline allows you to veto automated changes to any systems that the software provider might modify to restrict an application in the event of a disagreement.
Any sufficiently complex software project will include documentation to explain its principles to colleagues who come and go from the team. Such documentation can considerably ease the burden when transferring from one agency to another, and is usually packaged with the version control but can be withheld in the case of a transfer.
Project or issue management data
Lastly, while version control records notes from the developers perspective (if the developers have troubled themselves to write these notes down), it does not include the project managers perspective or any understanding of the current state of the project.
That data is more commonly stored in issue management systems such as Jira, GitHub issues or other such tooling. A history of such things is useful when trying to understand the context of a project after a project has been transferred between providers.
Control means responsibility
The safety implications
There is a significant catch to being in the ultimate control of a software project. The primitives used to create these projects now are extremely powerful, and can run anything from a small business eCommerce store to a billion user real time game.
If you are in control of this tooling you are responsible for the billing associated. Additionally, you immediately become an extremely attractive target for those who would seek to steal that access both to steal your data and to make use of the resources that you have attached to your credit card.
This means that the credentials that you have that give you this access must be kept extremely safe. While they provide you the ultimate access over a given account, you almost never want to use them directly; instead, they’re used only to grant access to others to manage your account. In the case those “ultimate” credentials are compromised there are audit logs that will detail who did what and how, and there is little recourse for you as a merchant having lost 10,000 EU on stolen compute time.
The services that build and manage software defined above are not free, and by being directly in control of them you are also in control of any costs associated. Additionally, should any of those tools lapse payment due to expired credit cards or other problems, development will grind to a halt but developers will likely still bill for the time lost.
How we at Sitewards balance this issue
At Sitewards we have a long history (~20 years) of managing software projects. We’d like to be working with our merchant partners because they find us a suitable partner and are happy to continue retaining our services to help them build their projects.
Accordingly, by default the most important aspects (the compute infrastructure) are in the control of the merchant, with limited access delegated to us to manage the resources as we deem necessary. Additionally, we make available version control at the merchants request. All of our documentation, delivery and other tooling is expressed in version control — accessing version control means accessing everything that we have in the project. Lastly, we are continually refining how we can make our development process more transparent, which in limited cases includes access to the project management tooling.
We are not perfect at managing this trade-off yet. But, as I hope this post demonstrates, we understand the risks that merchants are faced with and we’re trying to offset this risk as best we can.
Software development is a complex discipline, and it makes sense to contract some aspects of that out to experts who specialise in this. However, while undertaking this process it is also worth investigating what control will need to be delegated to the provider, and what recourse that you have if the relationship does not proceed as planned.
If you are in doubt, ask us at Sitewards.
- Francis M. Gallagher for their patient editing; I did a bad job here. ❤
- Pascal Brouwers, Daniel Nettleton for early reviews.
- Tomasz Kaplonski for review and feedback.
- Cipriano Groenendal for review and feedback.