In the first part article I started describing the steps to select a monitor solution for a SBC/VDI infrastructure. In that first part I described the first two steps. The first step is determine the actual information needs, which leads to a wish list. This wishes are categorized on importance based on the MoSCoW principle. After all wishes are known step 2 can be done, checking if there is already a product and/or functionality is available within the organization that can fulfill the wishes. If a product is available you can stop the selection process and focus on implementing/improving the product or functionality already available. If the outcome is that additional tooling is required we can continue with step 3, which is actually the start of this second part.
Step 3: selecting products for a short list
In this phase we start really investigating the SBC/VDI monitoring space. You will be wondered how many products are available mentioning they offer SBC/VDI monitoring options. Nowadays you will start the research using the internet. Visit the websites of the supplier and read (online) brochures. Visit independent conferences, ask the community and re-use earlier experiences with monitor products.
Don’t think this is an easy task. Personally I was a bit disappointed about the information available. I expected that currently every supplier has a wonderful website, where besides the logical marketing fairy tales also technical details can be found including visualization of the tool. However I saw several sites that only have the marketing fairy tales available, which makes it really difficult to value the product. This can cause that a good product does not get on the shortlist, because the real power is not shown. Hopefully also lots of suppliers are ready this article and will fix that.
Luckily there are also suppliers that do get it. Mentioning the key features and showing how the product is visualizing those feature. A good example is available in below shown figure.
If suppliers still are onboard, here is another tip. Let the community talk about your product. Several bloggers do good and honest reviews. Those reviews are giving good insights in the possibilities of the product and shows the strong and weaker point of the product by independent people. Personally I found information provided by independent people really valuable and although some reviews are paid by a supplier most people are honest in those reviews about a product.
A good example who is doing a great job on the team behind ControlUp. Their product is mentioned/reviewed on lots of websites, as shown in the figure below.
Back on the actual topic again. Write down all products you have found that could fit in a survey format including their characteristics of the product. Think of unique features, but also advantages and challenges of the product. From this list you need to select a limited set of product that will continue to the next round. Logically you pick the products where you think based on the current gathered information will fit best and can fulfill the wishes on the list.
In our scenario we ended with a list of 12 products which we selected four products for the next phase. Those product were Goliath, EG Innovations, UberAgent and Aternity. Remember that the selection is based on the wishlist. Other wishes will lead to complete another shortlist with another product shortlist selection.
In the next round we will review the products on the short list in detail.
Step 4: Product Review
In this step we are really installing, configuring and using the product. It’s advisable to gather more information like implementation- and administration guides. That makes it easier to know which steps should be executed. Secondly you need to get the installation sources. Again this sounds easy, but that is not real life. On most websites there is link to evaluate the product where you need to provide your name and other details. What happens next is different per supplier. Most supplier are contacting you that would like to have an introduction conversation about the product and why you are requesting the evaluation. Logically they want to know if there is a potential business case for them. Personally I don’t think there is much added value of these conversations, but they are needed to get the software. There are some supplier that just send you an e-mail with the software download and contact you after some days if everything is going as expected. A few supplier directly provide you with a download link from the website, needless to say that I love this most. Last there are also some suppliers that actually don’t provide an evaluating version, mostly they offer access to their demo environment or only do proof of concept where they are involved themselves.
After getting the actual software the fun parts starts. Installing, configuring and working with the products on the shortlist. Write down your experiences of the general steps. Next check if and how the wishes are implemented/available in the product. Also write down these experiences. In my opinion after these activities it’s a good moment to contact the supplier. Share the wishes including your experiences with the supplier to validate your results. This is good time to discuss and get the vision of the supplier. After the validation you are ready for the final step.
As mentioned we selected four product for the shortlist. Aternity does not provide an evaluation version, so we gathered the information together with the Aternity team including working in their partner portal. Goliath, EG Innovations and UberAgent I got an evaluation version and installed and tested in a test environment. The experiences of these products will be written down in upcoming articles on this website later on.
Step 5: Rate the products
Now we have the insights if and how the wishes are available within the products we can start rating the implementation per wish per product. Logically it’s smart to do it with a few people together. In our case we both tried the products and showed the implementations to each other. When everyone in this group has the knowledge, we can rate the wishes together. We decided to use a rating between 0 and 5, where 0 meant wish is not available in the product and 5 the wish is completely available within the product. Between 1 and 4 it’s available, but not completely available.
Together with the MoSCoW point a final score per wish is available, which can be sum up together for a total score. Looking really black and with the product with the highest score is the winner, however I would not recommend that methodology. Also makes selections per MoSCoW category, there is a possibility that a product scores less on the Must Have (or even does not have a Must Have options in the product available), but get much points on Could Haves. I made selections on Must Have, Must Have and Should Have and Total Score.
In the perfect situation one product scores (much) higher than others, but you can imagine that a few products scores equivalently. If that’s the case you can make a decision on “feelings”, which product do you (and more important the organization/colleagues) like most. Another option is to do a full Proof of Concept of those products together with the supplier.
In our situation three of the four product scored almost the same, so there was not a real winner. I finalized the report and advised to customer to execute a full Proof of Concept of those product.
Summarization
In these two parts series I described the five steps to do a production selection for a monitoring product for SBC/VDI infrastructures. Besides the theoretical steps we set-up I also shared our experiences shortly. In the upcoming article I will go into more detail in the three product, which I evaluated completely in a test environment.