Microsoft Corporation Xbox Live and SQL Hadoop Speed Service Updates with Automated Software Engineering

Team Foundation Server 2012 provides gated check-in and continuous integration out of the box. We can also customize things to implement the exact processes that we need.  Joe Chiu Senior Software Development Engineer in Test Xbox Live Microsoft Corporation

A worldwide leader in software, services, and Internet technologies, Microsoft has developed an advanced set of internal tools for its Build Deploy Test processes, which include tools for source code repositories, build management, lab management, and deployment.

These tools were designed for the company’s traditional development model of releasing boxed software every few years. With the move into cloud-based services, development teams striving for shorter release cycles wanted to improve their Build Deploy Test processes. “We needed a better engineering system for Apache Hadoop-based Services for Windows Azure if we were going to meet our goals of releasing updates once per week,” says Aviad Ezra, Senior Software Development Engineer in Test, SQL Hadoop Team, Microsoft Corporation. “We had a gap between our development and test processes, and it could take two days to realize that the build was broken.”

Another area that needed improvement was lab management. The tools the teams were using had a lot of capabilities for lab management, but setting up a lab was difficult and the tools did not support virtual machines. “We had one lab environment and we were very cautious with it because it took three to four days to rebuild and was never exactly the same,” says Joe Chiu, Senior Software Development Engineer in Test, Xbox Live, Microsoft Corporation. “That limited the amount of testing we could do and limited our partners’ ability to do integration testing. The single lab environment also caused problems because our automated testing and manual testing efforts were interfering with each other."

The teams didn’t have to look far for an answer: they already used Visual Studio to code. In 2010 the Xbox Live team and the SQL Hadoop team had adopted Microsoft Visual Studio Team Foundation Server 2010 and Microsoft Systems Center Virtual Machine Manager 2008 R2 to automate their Build Deploy Test processes and implement gated check-in, daily builds, and continuous integration. By using Team Foundation Server and Virtual Machine Manager, the teams radically changed their processes to meet shorter release cycles.

Team Foundation Server 2012 stores all of the data and produces great reports that keep the team informed of the progress of the sprint in addition to any issues that need to be addressed.  Aviad Ezra Senior Software Development Engineer in Test SQL Hadoop Team Microsoft Corporation

Gated Check-in –Developers found this easy to set up, and they can choose from multiple levels of validation. An example of this flexibility comes from the SQL Hadoop team, which set up two gated check-in scenarios. The default scenario runs unit tests on a build agent. The second scenario for riskier check-ins performs unit tests on a build agent and deploys to Windows Azure and a Hadoop cluster for additional testing.

Lab Management –The teams can use Lab Manager to create virtual environments to run tests on virtual machines managed by Systems Center Virtual Machine Manager. “Now we can create as many lab environments as we need from a template in a few hours,” notes Chiu. Team Foundation Server 2012 supports standard environments, which combine physical and virtual machines in the lab. With a standard environment, teams can install agents and prepare an environment that is ready for build verification with a few clicks. “We have to deploy Hadoop clusters in physical machines to accurately simulate the production environment, which meant we couldn’t manage all of our tests in Lab Manager before. With Team Foundation Server 2012 we use the standard environment in Lab Manager to manage these tests,” says Ezra. Using Virtual Machine Manager, the teams can auto-install agents and use snapshots to create their environments.

Test Management –Microsoft Test Manager handles all of the testing requirements for the teams. “Microsoft Test Manager provides a great unified test management experience for the teams,” says Chiu. “The testers own the coded UI and web testing. We’ve been able to implement a clear division of labor, which helps to ensure that were doing high quality testing at each step.”

Team Foundation Server provides the capabilities for fully automated code-coverage data collection. “Measuring unit-test code coverage that runs as part of a build is exceedingly easy,” says Chiu. “We can enable it by specifying a test setting file that has code-coverage enabled in the build definition. The code-coverage data is collected and archived in Team Foundation Server data store for reporting and viewing. For a new project in the team, we even start gating check-ins with code coverage criteria, so we can make sure our unit test code coverage starts high and stays high throughout the project cycle.”

Continuous Integration –The teams found it simple to implement continuous integration. “The build system in Team Foundation Server 2012 is built on top of Windows Workflow Foundation and is a powerful general purpose workflow system,” says Chiu. “A ‘build’ process can encompass compiling the binaries, deploying the build to a test environment, running tests, and making test results available in Test Manager for viewing and reporting.”

A lot of manual customization was required to achieve continuous integration with earlier versions of Team Foundation Server. Team Foundation Server 2012 makes is easier by providing additional flexibility to create and manage workflows for the daily builds. “By using the Team Foundation Server 2012 build process template, we can easily produce a build deploy test workflow by simply putting ready-made building blocks together,” says Chiu.

By using Team Foundation Server 2012, the SQL Hadoop and Xbox Live development teams can achieve the high-quality rapid release cycles they require. They have improved check-in quality and implemented continuous integration so they can deliver frequent releases with a high level of confidence in the code. The SQL Hadoop team has achieved is goal of weekly releases and one-day hot fixes, and the Xbox Live team can release code every four weeks.

With gated check-in, the developers can ensure that their code passes a wide variety of unit tests prior to adding it to the build. The developers are happy to get immediate feedback on their code changes rather than waiting one or two days. Another benefit of gated check-in for the developers is that they learn of issues with their code before it gets added to the build so nobody else is impacted or even has to know there was a problem. Now the teams can successfully compile their daily builds without fail.

“Because Team Foundation Studio takes care of all of our lab management, we have reduced the time to set up a lab environment from days to hours,” says Chiu. “Now we can create lab environments as we need them both for internal use and for partners. That flexibility has helped us implement coded UI testing and better support our partners with their integration testing.” The SQL Hadoop team can also easily set up labs that are local or in the cloud on Windows Azure.

The teams have seen additional benefits from Systems Center Virtual Machine Manager 2012. “It’s great for us that Lab Manager leverages Systems Center Virtual Machine Manager 2012, so we can take advantage of all of the new features,” says Chiu. “It is a full private cloud management solution. Deployment of applications, such as SQL Server, that were difficult to install in the lab before are now fully automated through the application templates.”

The team has implemented a process with clear ownership and scope of the tests, which produces faster and better results. The developers and testers receive continuous integration test results every couple hours.

Because all testing takes place within Team Foundation Server 2012, the teams can generate complete reports on the test results and code coverage. “Team Foundation Server 2012 stores all of the data and produces great reports that keep the team informed of the progress of the sprint, in addition to any issues that need to be addressed,” says Ezra.

Chiu adds, “We use the test data to help us focus our efforts. Things we look for are failed tests indicating potential issues, high churn rates indicating potentially high risks, and low pass rates without corresponding product bugs indicating potential low test quality. The data also helps to track improvements, define goals and calculate cost/benefit ratios.”

The services teams within Microsoft are extremely pleased with the Team Foundation Server 2012 solution, and are actively spreading the word to other teams within Microsoft. As the company moves more and more of its development efforts into services the Team Foundation Server build deploy test solution will continue to provide immediate benefits to Microsoft and its customers.

For more information about Microsoft products and services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Information Centre at (877) 568-2495. Customers in the United States and Canada who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to
www.microsoft.com

Visual Studio
Visual Studio 2012's Application Lifecycle Management (ALM) capabilities provide your teams with powerful tools that span the entire modern application lifecycle to ensure quality and reduce cycle times while delivering continuous value from design to deployment. Whether you’re developing applications for SharePoint, the web, Windows, Windows Phone, or beyond, Visual Studio is your ultimate all-in-one solution.

For more information go to:
www.microsoft.com/visualstudio orwww.msdn.microsoft.com/vstudio