How to conduct a real Microsoft 365 Pilot evaluation

Microsoft 365 is consists of many products and tools. It combines productivity tools as well as security tools. Therefore, while evaluating it we must conduct an integrated Microsoft 365 Pilot.

Here are the guidelines:

Microsoft 365 Pilot guidelines

These are not step-by-step instruction. It is a set of items you must consider while planning and executing your pilot. You can customize the exact evaluation as per your organizational needs.

1. Evaluate the entire Microsoft 365 E5 product offering
   • Whether you are going to purchase the full product set or not, evaluation must be done with the entire platform
   • This is because all components in the M365 E5 stack are designed to work in an integrated manner.
   • If you do the evaluation on a subset of tools, you will learn about the benefits in a fragmented manner.
   • This type of Microsoft 365 Pilot evaluation is flawed, inadequate and misleading.
2. Form a team consisting of representatives from the following roles:
   1. IT
   2. Security
   3. Compliance
   4. Business
3. At least ONE business user must be from senior management (SLT, CXO, leadership team)
   1. This way, we get leadership visibility from day 1
   2. From a security point of view, the leadership team is the weakest point. Including them in the pilot increases their awareness about security issues.
   3. It also ensures that all features are evaluated without compromising the convenience and productivity. In the absence of leadership involvement, the evaluation tends to be IT centric – which leads to expectation mismatches during actual deployment and adoption.
   4. Including a decision maker also helps in future adoption of the platform.
4. Ensure that you cover the following aspects during the Microsoft 365 pilot
   1. Entire Microsoft 365 E5 stack
   2. Desktop with entire Office 365 Pro Plus apps
   3. Mobile devices with all Office 365 mobile apps installed
5. From a functional point of view, ensure that you cover :
   1. Document creation on desktop, mobile and browser
   2. Identity protection with MFA on desktop, browser and mobile
   3. Threat protection on desktop as well as mobile
   4. Devices which are outside the corporate network
   5. Document classification for Office as well as OneDrive and Teams
   6. Email on mobile, desktop, web access and external parties
   7. Collaboration for internal as well as external users
   8. Compliance for all types of data – Office, Teams, OneDrive, Emails, Chat, Conversations, Notes and external apps
   9. Include unsanctioned apps to check the discovery features of CASB
6. It is often not possible to conduct full evaluation with real malware or virus infected files. Use the Simulation lab to do so.
7. Include realistic documents and data while conducting the Microsoft 365 Pilot
8. Make sure the Microsoft 365 E5 pilot involves business users at every step.
   • Security and Productivity must work together
   • The only way to ensure this is to make a regular user test every evaluation from a simplicity and efficiency point of view
   • If you do not include users in the Microsoft 365 E5 pilot, you will only get a one-sided, IT point of view.
9. For all features tested, quantify the business benefit, time saving and risk mitigation. This data can then be combined to create the ROI of the investment and reach an informed decision.
10. Even if your current setup is on-premise, conduct the Microsoft 365 E5 pilot in a hybrid (Azure AD connected to on-premise AD) manner.
    • In order to get M365 E5 benefits, it is not necessary to move completely to the cloud.
    • Lot of benefits are accrued by working in a hybrid mode
    • However, if you are purely on-premise, there is no benefit at all
    • Once you see the amazing improvement in your manageability, security and productivity by using the cloud integrated platform, you will be able to migrate in a confident and informed manner.
11. Use the most comprehensive plans while evaluating the platform. For example, use Azure AD P2 – not the free version. DO NOT worry about which feature is in which plan. It is irrelevant while learning the business benefits of the platform.
    Evaluation should be comprehensive.
    Procurement can be selective.
12. Throughout the Microsoft 365 E5 pilot, identity should be managed using Multi-factor authentication
    1. Do not use simple username / passwords. Use MFA
    2. For Windows 10 devices configure face sign-in if your hardware has Windows Hello compatible cameras
    3. If not, use Windows 10 pin for logging in
    4. Make sure you enable Windows Hello for business in Azure AD
13. Conditional Access is a very powerful tool. Understand and utilize the full flexibility of this toolkit during evaluation
    1. Use different type of devices, IP address ranges, locations and login types
    2. Configure automated policies for testing unsafe logins and vulnerable devices
    3. Evaluate Office 365 application level conditional access as well – especially for OneDrive, Teams and SharePoint
14. Enable External Sharing while conducting Microsoft 365 Pilot
    1. External sharing using Links is SAFER than sending attachments
    2. It also prevents users from indulging in shadow IT
    3. Use appropriate controls to ensure guest access is managed in a compliant manner
    4. Try Guest user Expiry and Activity Audit and Guest review features
    5. Use Cloud Access Security to get live visibility into user activities for cloud applications
15. Create Labels for sensitive data, DLP and retention. Now with Microsoft Information Protection, three types of labels can be unified. This is a significant advantage.
16. Detect cloud apps by running CASB discovery
    1. Microsoft 365 E5 pilot should also be conducted on real user machines to discover Shadow IT.
    2. This cannot be done on the test setup.
    3. Give the proxy server log as input for Microsoft CASB and let it discover shadow IT within your internal environment.
17. Monitor and learn from Security Score and Compliance Score
    1. Throughout the Microsoft 365 Pilot, look at the scores and perform recommended actions
    2. Monitor the improvement in score and appreciate the automatic actionable information which the platform provides out-of-the-box
18. For all activities being tested, compare the operational efficiency and functional benefits with existing products which you are using
    1. Microsoft 365 pilot has a dual purpose: One is to evaluate the Microsoft platform, but equally important is to compare and contrast it with solutions from other vendors which you are currently using.
    2. The significant reduction in the manual work involved in purchasing, managing, maintaining and operating multiple products itself is an important benefit of using the Microsoft platform.
19. Do not compare individual products. Compare the platform.
    1. This is because in Microsoft, productivity and security tools work together
    2. If you compare them with competing products in isolation, you are getting a lopsided and skewed view of the feature set.
    3. Integration is the key feature of Microsoft – which no other platform offers. This topic is missed in a typical feature by feature comparison. Avoid this pitfall.
20. Include an Office 365 and Power BI pilot as well

Give me your feedback about this  Microsoft 365 Pilot methodology

The process outlined above is based upon my experience of working with many customers. Of course, you may have your own opinion and methodology. Feel free to post your comments. I will update the content based upon reusable and globally applicable inputs based upon your comments and suggestions.