How to qualify an Enterprise-y software solution from a tech-guy’s perspective

At some point in your career you will inevitable come across one of those major projects which will replace what you’ve got with something better, more reliable, more suited for exactly your business needs, easier to maintain and a whole lot more… You know the drill. Let’s for the fun of it call the legacy system bound for replacement OldSystem and the new system for ShinySystem

If you’re really lucky someone at some point in time will ask you to qualify a number of vendor’s ShinySystem towards eachother. Maybe the decision has already been made and you find yourself stuck in an Middle-Eastern-style arranged marriage but at least you should still need to know the new ShinySystem on a higher level. What should you be looking out? I’ve had the pleasure of working with both legacy systems and shiny Sharepoint-solutions during the last three years so I’ve faced these type of questions before. I remember that I used to be terrified answering questions such as “What do you think of this ShinySystem” because frankly I really didn’t know what to look for from a technical point of view. During the last two or three years I’ve summoned up the gazillions of mistakes I’ve made along the way and here it is – the ultimate “What you should ask your vendor once the guys from Sales have left the room”.

Disclaimer: I’ve tried to keep the list technology-agnostic but keep in mind that I’m a .NET guy… Also bear in mind that this list is intended to qualify larger Enterprise-y systems. If you intend to use this list for qualifying a .NET control you intend to buy for 99 dollars you are way off target. I assume that you’re looking at something you will be hosting yourself on servers which are not cloudbased and you have full control over because cloudbased services is a whole other ballpark. Last, but not least – you’re a tech guy so I’ll be focusing on tech stuff and not issues such as licensing and stuff. Enough rambling – here goes:

Security model

I’ve come to the conclusion that in just about any system there is a concept of security. It can be disregarded - i.e. by placing an internal website on the local intranet and telling nobody it exists except the five developers who need access to it – but the concept of security still exists and have to be taken into regard when designing the system. The concept of excluding someone from doing something is what security is all about. The way security is handled by a system tells you if the system is designed with security in mind. If it is not – what else wasn’t taken into account you should ask yourself in your position… I would start bothering developers on ShinySystem with questions like the ones below:

• How are users and roles created and maintained in ShinySystem?
  • Users, roles, permissions and securables are completely different things. If you think “read” and “write” are roles, please stop reading until you’ve learned the distinction.
  • User authentication and user authorization are completely different things. If you don’t know what I’m talking about, please stop reading until you’ve learned the distinction.
• Is there a concept of a user group?
• How are permissions maintained?
• Is it possible to define your own set of permissions?
• Is it possible to break inheritance?
• Please explain how auditing works in ShinySystem

That should quickly lead you towards areas of the system which nobody else in your organization knows exist – but the have to live with the consequenses if ShinySystem doesn’t live up to business expectations regarding security. If you have a business case regarding security which OldSystem doesn’t solve according to business needs you should analyze why it doesn’t work in OldSystem, solve it on a whiteboard using ShinySystem and maybe make a Proof of Concept / prototype if you’re still uncertain whether or not it will work. I regard a feasible security architecture as a key factor when qualifying software products – simply because an inefficient architecture affects about 100% of the users who will be using the system and users expect things like security to “just work”. They will blame you and not the vendor if security issues comes to cause friction in their daily work so watch out on this one.

Custom code

You’re looking at any given ShinySystem and have a business case ready which should be possible to solve in ShinySystem. At some point even the sales guys are having a hard time keeping up their appearences because it isn’t possible to model your entire business using Drag&Drop in that shiny visual modelling tool in ShinySystems main window. This is where you probably will have to code something on your own. Most major vendors design their core applications with extensiability in mind - Firefox extensions have been around for eons by now – so you should expect that any Enterprise-y ShinySystem candidate provide some way of letting you write your own code in terms of “custom workflow”, “plugin”, “webpart”, “custom control” and so on. Given that you CAN write custom code and extend functionality – start wondering:

• Which programming languages can I use?
• What is the development cycle when writing custom code on your development machine
  • One answer could be “Write code, build, deploy to test website, reset webserver, test change”
• It is very likely that there is an API somewhere to interact with ShinySystem
• How does the ShinySystem API look like (there is likely to be one)
• Are there any undocumented features in the API?
• Which programming languages can you use?
• How does an API error message look like – can I tell what’s going on just by looking at it?
• How do I upload my changes to production?
• If my custom code fails how will it affect the stability of ShinySystem?
• How do I debug an error in a custom component?
  • During development?
  • Once an error shows up in production?

Storage

Applications come and go. Data lives forever. There are basicly two distinct types of replacing OldSystem with ShinySystem – those where you move the OldSystem data (or parts of it) to another, shinier location and those where you don’t. And then again: If you decide or is being asked to move parts of your data storage to another location there are two types of migration projects: Those where you are able to buy tooling and expertise and those where you’re on your own. If you are migrating from SQL 2005 to Oracle you have migration tooling avaliable to you. But what if you are migrating from i.e. Sitecore CMS written in .NET to Drupal which is written in PHP which is completely different using another programming language and another databasetechnology? You are on your own on this one. Maybe you’re lucky to find tools and guidelines which can help you but it’s up to you to find and use them. In either case you will end up in a migration project writing mapping code which extracts data from OldSystem and inserts them into ShinySystem. This discussion quickly descends into lowlevel technical details but remember that you want to know where your data is in ShinySystem and how it looks like – especially now when cloudbased services is all the rage and data flows around. Do you know on which servers your Google Emails are being stored? Are those servers located in Europe or United States? Do you care? If those emails are crucial to your business in regard to storage management policies you HAVE to care – that’s my point.

Side-note: Developers are usually in full control of an entire system but keep in mind that some Enterprise-y systems don’t allow you to mingle with the physical data storage. A dedicated Microsoft Support SWAT team will hunt you down and bleed you to death with a blunt knife if you as much as add a new column to a table in the database behind Microsoft xRM. In xRM you’re allowed access through webservices but the underlying data storage could be flat text files for all you know. You do of course but Microsoft early on decided to take full control of the database behind Sharepoint and xRM.

Keep in mind that vendors should have an answer ready on these topics:

• Where are my data once I click “Save” on a new user / a new item etc in ShinySystem?
  • The answer here is of great interest if ShinySystems is located in the cloud
• How do I perform a backup / restore of my new datastore?
• In regard to tooling
• How do I monitor, profile and optimize the current use of data?
• How do I upgrade the database scheme?
• How do I deploy database sceme changes to production?

Scalability

Scalability is a very loaded word – what is scalability after all? Does your system scale well if ShinySystem responds in a timely fashion? Wikipedia offers one feasible conclusion: “Scalability is the ability of a system, network, or process, to handle growing amounts of work in a graceful manner or its ability to be enlarged to accommodate that growth”.  The article also suggests possible scalability dimensions such as administrative, functional, geographic and load scalability. One might add that a system should be able to scale down as well – if you’re doomed to maintain 10 servers because that’s the way you installed it but you figure after 6 months that you only need 2 servers but your are unable to reconfigure your installation because “thats the way it is” – your system doesn’t scale very well in my opinion.

I’m from a webbased world and the bottleneck in just about any case I’ve experienced with users complaining about long response times in the end came down to problems extracting data from the database. The database might be flooded with requests – SQL statements were poorly written or there might be an infinite loop on a webpage with a database call in it which effectively exhausted the entire database from responding to other requests in a timely fashion. If ShinySystem is a webbased solution you want to dig into how ShinySystem communicates with the datastore that’s for sure.

• How would you suggest to scale up ShinySystem if the number of users exceed what we’re expecting?
• Can parts of ShinySystem be run on multiple servers?
  • Large websites might run on N webservers with one single database server but ShinySystem might consists of both the webserver AND the underlying database.
• How would you scale down ShinySystem?
• Is caching a baked-in feature of ShinySystem?
• How does it work?
• How do you administer and tweak caching features?
• How does ShinySystem communicate with it’s datastore – synchronous or asynchronous?
• How would you throttle or prioritize data requests from various parts of ShinySystem?

There is of course litterature available on the subject – I’m currently reading “Scalable Internet Architectures” by Theo Schlossnagle.

Upgrading

You’ve built your ShinySystem, the users are happy, you’re overall happy and have learned your way around the things and limitations within ShinySystem 4.0 which is the version you’re running. Then a new version 4.5 is ready for sale and management obviously wants to upgrade because reporting and statistics are better in version 4.5. Now you’ve got a headache because management of course expects everything to be running as usual with all the new stuff made available to them. Who can blame them on such an assumption?

How you and your vendor have prepared yourself for such a situation (and upgrades will be part of your life from time to time once you start playing with standard Enterprise-y software) is of great interest. Remember: Every time you decide to extend standard functionality you take sole responsibility for it to work after an upgrade of the underlying system. It can’t be emphasized enough. Often vendors will make a great effort to make it easy for developers to extend their standard software – by providing templates, base classes, fully documented APIs wrapped in a multitude of languages and so on. It looks nice but remember that whatever you do – if you extend standard functionality it’s you and not some ShinySystem key account manager who will be called upon once you perform an upgrade and things start to fall apart.

You should really be looking out for systems that are hard to model towards your business needs without writing custom code. If you’re facing problems where you’re trying to configure ShinySystem to fit business needs 100% and 9 out of 10 times it turns out you have to write custom plugins to make things happen you’re bound for trouble. The size of an upgrade project will grow exponentially to the number of plugins you need to test on a new version in my experience.

It is sometimes also a question of mind-fiddling with your managers and project leaders. If you give them a choice of delivering 80% of any given task within the next day using standard functionality ready on your testing environment – and delivering 100% of business requirements in about three weeks give or take a week because those last 20% requires you to write custom code, test it, test everything else and still risk breaking all the things you didn’t take into account because of obscure dependencies and you-know-the-drill… most (or some at least) sane project managers will go with the 80%. Sometimes the core business value is hidden in the last 20% and you have to take your time writing code and testing it but you will do it knowing that it is vital to the business that you do it and spend money on you giving a custom extension your very best shot.

Developer community

“Can I find solutions to my problem in ShinySystem on Google?” It really comes down to that. I make a Google search no less than 20 times a day every day at work when I’m doing programming chores. If you decide to go with any Enterprise-y system you really, really want to go with one that has a decent developer community – which means people who use the system you’re about to get married to and provide developer solutions to developer problems. It can’t be emphasized enough – solutions to your problems will take longer to develop, will lack quality and probably won’t follow best practises if you have to educate yourself in ShinySystem because noone outthere can help you. Developer communities embrace both opensource products and products like Microsoft Sharepoint – there are surprisingly many people out there who love Microsoft Sharepoint to a degree where they are building webparts every day providing them to you for free. Dedicated bloggers (also non-MVP's) are consistently adding new posts about problems and “how-to-avoid-strange-messages-like-XYZ-if-you-want-to-ZXY” experiences – things you are likely to experience for yourself once you make a final decision to go with ShinySystem. I don’t value the developer support you might get from ShinySystem as high as the one you get from real-world users with no relation to the company who built and sell ShinySystem solutions – simply because when it comes down to it ShinySystem’s own developers will never advise you to use another product which they may know of and which might solve your problems in a much better fashion than ShinySystem is able to.

• Have people before me used Shinysystem with success?
• If I write “Shinysystem blog” and “ShinySystem best practise” in Google what do I find?
• Are there any companies which offers developer and user education in ShinySystem?
• To what extend do people with wifes and families love ShinySystem so much that they can’t resist develop ShinySystem plugins in their spare time for free?

If you decide to go with any ShinySystem, even a major one with an active community surrounding it – spend a few hours writing blogposts about your own discoveries along the way. All those posts available to you have been written by someone for you to use for free, right? If you like what others have written and use their work to get things working at your current gig you should at least consider sharing your experiences with ShinySystem on a blog or similar – anything goes as long as Google indexes it once in a while.

Epilogue

There are a ton of questions regarding debugging, logging, development environments, licensing issues, testing abilities, how to automate builds and deploys, surveillance and monitoring of production systems and so on which I won’t cover simply because I’ve been writing for hours now and need to stop at some point and get some work done…

My final statement for now will be this one: There are always tradeoffs – it is impossible to have your cake and eat it too so you need to focus on what’s important for you as a developer.

• Is it an extensive range of tooling?
• Is it the programming language available to you?
• Will you be running screaming away if you’re told you can’t add indexes to the database because database optimization is being handled for you behind the scenes by ShinySystem?
• Do you need an extensive developer community you can turn to or are you best off by inspecting code and figuring things out for yourself?
• …

If you’ve come all the way to the bottom of this post (that would be around here, I guess) I guess you have a few ideas about other areas of concern or bulletpoints missing in the areas covered by this post. Do write them in the comments section and I’d be thrilled. Until next time…