27 October 2014
Many of Mozilla’s web projects have adopted concepts from continuous delivery. Once a code change lands on a master branch it is shepherded to production with little-to-no human intervention. Quickly getting new features out to users.
While the implementation details vary from project to project, the core concepts tend to remain the same between projects. It is important for project stewards; project managers, developers, testers, UX, community, etc to have a shared vocabulary.
The purpose of this document is a starting point for this conversation.
We find that being able to move quicker allows us to create higher quality software with limited resources. This fits our goal of creating features that our users want, working in the open, and experimenting with our development practices.
Continuous delivery has worked for us because of the culture of our community. We accept shared responsibility for the quality of our work, are strong collaborators, and have a culture that is supportive of learning from mistakes.
Over time we've become more confident in the quality of our work. Moving quickly allows us to take a more proactive stance towards the quality of the products we create as well as mitigating risk when creating new features.
The basic concept is that any code that lands on the master branch should be in a shippable state. New feature work should be categorized by the amount of risk the feature or change to the code base poses.
The team discusses and agrees upon a risk metric that they are comfortable with assuming. Assumed risk isn’t a static conversation, but it is good to build a skeleton that the team can agree on and enhance over time.
A product should be broken into user stories so a team can apply a hierarchy of risk. It is important to identify and engage as many stakeholders as possible, each individual will have a unique and important point of view.
The list of stakeholders often include; project managers, developers, testers, UX, IT, DBAs, the security team, etc. Sometimes the product has high enough visibility in the organization that a proxy for the CEO is beneficial.
Not a dogmatic rule, in general new features or defect fixes should be manually tested and verified. Exploratory testing is the rule. Depending on the risk assigned to a feature, projects have the option of using waffle flags. The use of feature flags allows us to hide features from the general public while allowing a team or a group of beta testers to test on production with real data.
If a feature is deemed as low risk and it's functionality is easy to verify, anyone on the team is empowered to test and verify it. If it is a feature that requires deeper investigation and poses high risk, the QA team is the group who verifies it. If a feature set is big or user workflow is changed it is important to engage the test team to flush out defects and usability concerns.
Feature work that falls into the category of “better never break” should see a heavy amount of exploratory testing before it is exposed to the public. Once the feature is baked and code churn has decreased, we seek an appropriate level of coverage using Selenium, Python’s Requests API, load testing, and fuzzers.
A component of the QA team automation strategy should also includes reviewing the amount of test coverage that follows a developers pull request as well as taking part in the code review. Often times the test team is able to influence and enhance the depth of coverage by simply asking for more developer level tests [Unit + Integration].
Sites should always have an easily discoverable method for users to submit feedback and file bugs. The team must be studious about responding to requests for help in a timely manner and do so with empathy.