Testing is unavoidable aspect of development, marriage of development and testing is where Quality is achieved. --Anonymous
Introduction
We all might wonder how does one of the biggest technology company called Google does testing. As they are solutions from operating system, to web apps, to enterprise, to commerce and social apps, etc. Specifically having much of their source code open.
You might be surprise to hear Google doesn't have a Testing Army to have the projects testing rather fewer dedicated testers in the entire organization. The testing team is part of a centralized organization and so called as Engineering Productivity Team (interesting name).
Google engineers prefer quality over features.
Let's see the actors who take responsibilities of the testing, types of tests done on the software, process and tools used. And finally the culture in the organization for testing.
Roles and Responsibilities
It's important to know the characters involved in the testing of the software.
SWE – Software engineer
Create design document
Choose data structures
Overall architecture
Write and review functional code that ships to users
Write lots of test code
SET – Software engineer in Test
A developer role – 100% of their time writing code
Review the designs and look closely at code quality and risk
SETs are partners of SWE codebase, but are more concerned in quality and test coverage
Write code for SWE to have them easily build TDD
TE - Test Engineer
Most of the time developer cum tester role
Its a role that puts testing on behalf of the users first and developers second
Write automation scripts and code that drives usage scenarios and mimics the user
Organize overall quality practices, interpret test results, drive test execution and build end-to-end test automation
Security, privacy, performance, reliability, usability, compatibility, globalization and other concerns are looked on
Types of Tests
Small Tests
Cover single unit of code in a completely faked environment
Unit tests, that verifies the behaviour of single class or function
Medium Tests
Cover multiple or single units of code in real or faked environment
Integration tests, validation of integration of one or more modules
Large Tests
Cover multiple number of units of code in the real environment
System or End-to-end tests, operate at high level and verify the application as a whole is working
SDLC Process at High Level
Idea is converted into POC/Google branded project
Team Structure –
SWE
SET
TE (added in later stage of the project)
Explore the existing libraries, interface and protocols
Development and create unit test cases
Automation planning and increase the testability
CICD ( some projects works based on: Feedback based development / Bug based development)
Tools used and built internally
Unit Test Dashboard
Allow anyone to define their own project, with a set of build and test targets with a set of maintainers
The system would run all the tests for each project every day
Code Coverage
Tool for measuring whether a project's tests have a healthy mixture of small, medium and large tests
The general thumb rule: 70% to small / 20% to medium / 10% to large tests
Test Certified (TC)
A system of testing challenges that if a team completes, will earn the “certified” designation
To start all the projects will be Level 0 and the Levels are from 1-5
TC Level 1
Continuous build with test coverage bundles (small, medium, large)
TC Level 2
Requires smoke tests to pass before a submit
TC Level 3
Requires tests for all non-trivial changes
TC Level 4
Automate running of smoke tests and total test coverage should be at least 40%
TC Level 5
Activity use available analysis tools and total test coverage should be at least 60%
Google Test Analytics (GTA)
GTA is a simple web application that makes the data entry and visualization of risks little easier.
GTA supports the ACC model for Risk Analysis
ACC accomplishes through 3 views of the product
Attributes
Are the adjectives of the system. Qualities and characteristics that promote the product and distinguish from the competition
Eg: Google+
Social: Empowers the users to share information and what they're up to
Expressive: Users can express themselves through the features
Easy: Intuitive. Easy to figure out how to do what you want to do
Components
Are the nouns of the system. Core components or chunks of code makes to know what the product is.
Eg: Google+
Profile: Information and preferences of the logged in user
People: Profiles the users are connected with
More components are : Circles, Notifications, Interests, Posts, Comments, Photos
Capabilities
Are the verbs of the system. Actions the system performs at the command of the user.
Eg: Google+
Profile
Social: Shares profiles and preferences with friends and contacts
Expressive: Users can create online version of themselves
Expressive: Personalize your experiences with Google+
Easy: Easy to enter the update information
Like such all the components are mapped with appropriate attributes
Google Test Case Manager
To make it simple to write tests, provide a flexible tagging format that can tailored to any project
Makes test easily searchable, reusable and trackable
Buganizer
Bug tracking system
Simple to use and provides summary charts and reports
Quality Bots
Running regressions tests in virtual environment.
BITE Experiment
BITE stands for Browser Integrated Test Environment
This experiment is bringing as much as the testing activity, testing tools, testing data and reporting issues into the browser and cloud as possible
Reporting bugs (with recordings) and viewing bugs in chrome itself make it very simple to use
Diagnostic Utility
Issues submitted by Google Users, requires some technical data about the state of the Google software is needed
This utility, Google-signed executable is sent and executed to diagnose a user's machine
Gathering these details helps to fix the issues
Culture
The openness in the source code.
All engineers must reuse existing libraries, unless they have a good reason, as it benefits in time which will take to build and test.
Twenty percent of time is what called side projects, weekly one day is dedicated to work with the peer team or innovate startup POC project. Everyone enjoys this, Gmail and ChromeOS have started from these ideas.
Common infrastructure, linux distributions for engineering workstations and production deployment machines, centralized source code and modules.
Code Review is considered as most important and eagerness to do.
Dogfood, the term denotes to internal adoption of the software that are yet to release. Meant to convey the idea that if you make a product to sell to someone else, you should be willing to use it yourself to find out if it's any good.
Crowd sourcing, is a new phenomenon on the testing scene. Enter the crowd; a set of power users who are tech savvy and willing to help for reasonable compensation. Have programs in place to pay the best bug finders.
Culture of sharing ideas, supporting bottom-up experimentation and organization flexibility creates a fertile world.
It's okay to fail, Google's culture of supporting experimental projects which has led to many innovations and large junk of failed experiments. Learnings and improvements from the failure matters the most.
In-house developed tools don't need a lot of selling, weekly demos of the tools are done. Engineers come the way to adopt the tools.
Success is a collective issue.
Conclusion
Overall it was good reading experience with many examples from different google teams. Some of the points would seem to be repetitive, felt it is to emphasize the concepts. The concepts may not be new to us, but the process and making it meticulously followed is very interesting and we see now where they stand.
Note: The book I read was little old 2013 impression.
Reference
Book “How Google Tests Software” - James Whittaker * Jason Arbon * Jeff Carollo
No comments:
Post a Comment