Excel can’t count?

I’m currently working on a data-extraction gig – basicly we have 4 different data providers to persist information about attendees to a summit in July with some 25.000 participants – and I’m currently in the process of creating a small, shortlived console app which generates an Excel-file with information about each participants so we are able to print out identitcal badges used to identify participants at the gates during the summit.

I encountered one of the scariest things I’ve seen in months while working today on the application. First things first: One of the datastores for participants is an Excel spreadsheet – one line per guest that is. Easy enough, I found some code using the OleDB Jet-engine:

string connectionString = string.Format("Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties=\"Excel 8.0;HDR=YES;\"", _xlsFile);
DbProviderFactory factory = DbProviderFactories.GetFactory("System.Data.OleDb");
using (DbConnection connection = factory.CreateConnection())
{
    connection.ConnectionString = connectionString;

    using (DbCommand command = connection.CreateCommand())
    {                    
        command.CommandText = "SELECT * FROM [Ark1$]";
        connection.Open();

        using (DbDataReader dr = command.ExecuteReader())
        {
            while (dr.Read())
            {
                
                //do stuff

            }
        }
    }
}

During an ongoing testphase (yes, we actually DO this agile stuff here) I noticed that one of the columns in my output spreadsheed didn’t match the expected values. Quite an error actually because one of the things people at the summit do care a lot about is that the food they’d paid deerly for in advance ends up as a crossed checkbox in the right place on their ID-card. Otherwise they will not be allowed access to the dining area… You can imagine what people think of the IT-guy in charge if things like this doesn’t work like a charm.

So I started debugging. First I suspected that I simply counted wrong when extracting values – namely the output from these columns

 

As far as I’m concerned M is the 12. letter and N is the 13. However – debugging this stuff showed me the following output for the first and second row:

and row #2:

 

The birth date matches (as with names and other stuff) but notice the the 13. and 14. column which doesn’t match at all with the data in the spreadsheet… For one every cell has a value so how can row #2 suddenly be empty in the debugger???

I suspected I was working on another file than the one I thought – triplechecking that by inserting my own name and seeing it come up in the debugger proved that theory wrong.

I’ll personally buy the first person to explain this mystery a beer for an bulletproof explanation (other than my own which of course includes a major malfunction in Excel 2003 / .NET / OleDB / pick your own). What’s going on here? I’m not even close to being overworked (don’t tell my boss) and the code is simple… I tried converting the source spreadsheet to CSV to check the values in a plain textfile and they are what they should be according to the source spreadsheet so that’s the approach I’m taking now. Instead of parsing the Excel directly I’m converting it to CSV and using that as my datasource instead. Not very elegant but it makes me feel a lot safer.

Why developers love a big whiteboard

If you ever wondered why developer's love for whiteboards is so passionate you get the answer here:

http://www.agile-software-development.com/2009/03/power-of-whiteboard.html

I like the last comment best:

"Tools can certainly help to organise information more efficiently. But I would challenge any tool to do all of that! I'm not against tools. Not at all. But I think they should supplement the whiteboard, not replace it. Tools should be used for things they can do that a whiteboard can't. For instance, keeping track of longer lasting information, doing calculations, searching, etc."

Question, just out of curiouosity - it might even be a little stupid but here goes: Did you ever work somewhere (as a developer) where whiteboards were regarded as You Ain't Gonna Need It ??

Regards K.

Code reviews done "right"

This is the second post in a series of posts related to 10 things I don't believe in.

• Part 1

"I don't believe in code conventions" - hmm... Did I really write that? How can reviewing code be bad? It's not, actually. Code reviews can be beneficial if they are done right because to have someone review code you have written is one of the top 3 things you can do as a software developer to improve your skills. It's just the way things work - you can be a winner in life, in love and in any Xbox game by not repeating your mistakes more than once - and you will be a pitiful looser with no friends and no money if you have one success and spend the rest of your life doing whatever made you a success in the past over and over again.

When I think "code review" I think:

• Get a timeslot in everyone's calendar and book a meeting for i.e. half an hour or an hour
• Once at the meeting: Get everyone lined up in front of a projector
• Pick some code from the project you're working on
• Have the person who wrote it to walk you through
• Feedback

95% of all the formal code reviews I have ever attended followed that prescription - and it never really worked out the way it was intended for the following reasons:

• Get a timeslot in everyone's calendar and book a meeting for i.e. half an hour or an hour
• Developers hate meetings. It takes time from their coding, quite literally. Developers don't like to be interrupted unless they feel an urgency to attend the meeting - and code reviews aren't percieved as something you'll die of if you don't get your weekly dosis.
• Get everyony lined up in front of a projector
• The more formal the approach, the merrier. There might even be some sort of agenda: "Class X: 10.00 A.M, Class Y and Z, 10.20 A.M. T-SQL changes 10.40 to 11.00 A.M"... Let's see some code, right?
• Pick some code from the code you're working on
• Is everyone working on the same project? If yes that's fine - proceed. If no, you'll lose every single developer who hasn't been actively committing to the code you've picked. People just don't feel committed to concentrate on understanding Javascript if they regard themselves as being primarily Database guys. And the clientside guys and gals don't have much input to a discussion about classes not implementing IDisposable in a correct manner.
• Have the person who wrote it walk you through
• Being up on the stage in the spotlight takes some balls to everyone who isn't used to being looked at - you're in a very vulnerable position up there. Also having to communicate some crappy code you wrote in front of perhaps some senior developers who you know have at least a dozen better ways to implement a variation of the combined Visitor / Strategy pattern you've been working on can be frightening.
• Feedback
• If basic trust and respect between the parties attending the code review aren't there code reviews can actually do more damage to team morale than anything else because creative people (like developers) tend to block their minds when being critizied because they put so much of themselves into their work. Even if a developer has got the arguments to prove his way of doing things he might not be able to communicate well because of the spotlight and the crowd of people staring at him.
• If feedback to code reviews aren't backed by i.e. code conventions or architectual documentations in terms of "this is how we do things here when we deal with datetime formatting" they will mostly be based on personal bias towards a solution you like better - not necessarily the right solution for the problem at hand. That's the way it works in real life if there are no code quality measures beyond the team that the code should live up to.

Only very mature teams can handle a code review which includes a projector, I think. And why, why, why make things so complicated? Why call everyone in and disturb everyone because "code reviews are important"? When I want my code reviewed I do this:

• Write some code
• Ask anyone of my collegues to look it through
• He or she moves to my desk and we sit down together and go through things together
• We might refactor while pairprogramming
• Put my taskcard on the "To be verified" column on our sprint board

This informal approach has a number of advantages: It doesn't feel like a code review. You don't get to attend meetings which you feel eats away your coding-time. Nobody is put up on a stage and asked to speak loud and clear because there are 6 people sweating and a projector humming in the background. Feedback happens informally and instant - and you feel safer walking through the code because it is fresh in your mind (you just wrote it, remember?). Feedback is also more valuable because it happens 1-1 in an atmosphere of "how do I solve this problem the right way". You're in problem-solving mode while sitting at your desk - chances are that you're not if you're being interrupted on-stage while trying to remember why class X implements interface Foo instead of the more general interface Bar.

Martin Fowler once wrote that If it hurts do it more often. If code reviews don't feel right you're doing it wrong - and probably you should begin by loosen up the tie and try a less formal approach because informal approaches always works the best while dealing with developers.

Until next time...

Regards K.

How to write quality code without code conventions

The last blogpost I wrote caused quite a few hits on DZone. Due to the fact that a healthy conflict is the best way to learn new stuff I'll try to explain my thoughs on each of the items in my blogpost. I'll dedicate this post to #1: I don't believe in code conventions.

I'm a firm believer in agile values and principles. I think that processes and interactions are more valuable than processes and tools. Code conventions can be useful in certain environments - for instance: I live in Denmark and we speak Danish. Hence a lot of the code I see every day is a mix between English verbs and Danish nouns. If you are building an external API (a webservice for instance) for clients around the world you'll effectively cut off 99,9% of your potential customers if you write that API in a mix of Danish and English. Take for instance a method named in a typical Danish / English mix: "GetKonto" - how will you as a non-Scandinavian know that "GetKonto" translates very well to "GetAccount" which is the pure English name of that method? You won't - and that's why code conventions should be an issue whenever you are having conversations with unknown consumers of i.e. a webservice you're in charge of.

The two questions one must ask before adhering to a set of conventions are:

• Will they provide direct business value - will your customers care?
• What is an adequate penalty for not following the conventions?

You see - I don't believe in a set of rules that you can break without having to at least consider the risks involved. Code conventions work if there is an immediate fine or penalty for breaking them - i.e. a red build, a QA team who amongst other things are employed to review code and discard it if it doesn't meet certain standards such as a set of conventions. If there isn't any code police around to enforce the laws and standards there really isn't any reason for the developer to adhere to them - the code works, all tests are green and the customer is happy, right? If nobody have been authorized to vote down and discard code that doesn't meet written standards - these standards will not be followed consequently and are by default doomed to fail.

I don't subscribe to anarchy if that's what you think by now - not at all. How should code be written if there are no set of rules at all? Well - for starters:

• Your code should be readable to the human eye. If you're unsure if your code is get someone to read it and speak out loud what the code does. If that someone takes more than 10 seconds to interpret a line of code you've got a red Human Readability test.
  
• Document your code using unittests.
  
• Write meaningful comments - WHY does the code behave like this as opposed to WHAT is the code doing right now.
  
• As a rule of thumb: Don't use oneliners which spans several lines. There are exceptions of course - such as LINQ expressions but your eye gets "tired" mentally decomposing the true intent of a oneliner.
  
• For the sake and sanity of everybody: Give your parameters and attributes meaningful names. Prefix them or postfix them as you like but keep them meaningful. The only exceptions to this rule should be counters in i.e. a for-loop named i, but that's about it.

...with the Human Readability test being the far most important one.

As I stated before: I believe in humans and interactions and value them higher than processes and tools. If you are having doubts about your code: Speak up. If you see code which you are not able to read: Speak your case. Explain to your team that you're unable to maintain code that you can't read. If you're having trouble reading well written code because you i.e don't know the syntax - do not make the mistake to refactor it towards something you like better. I fall into that hole once in a while myself if I don't watch out but I try so hard to avoid it. Learn the full syntax of your language if that's your problem - you don't have to use it if you don't like it because there are no conventions, remember? Develop a coding style which makes it readable to humans - make your intentions clear in your code rather than of following a prescribed set of rules. That's the only code convention you'll ever need to follow.

This will be the first in a series of posts which digs a little deeper with the "10 things I don't believe in" posted earlier. Until next time...

Regards K.

10 things I don't believe in

 

1. I don't believe in code conventions. Code and the context where the code exists (projects, folder structure, file naming etc) should be selfexplaining and pay respect to i.e. Single Responsibility Principle
2. I don't believe in code reviews. Code reviews only works if you have code conventions - which I don't believe in. Nobody feels ownership towards the code being reviewed except the one who actually coded it.
3. I don't believe in automated frameworks testing your UI. Tests are a pain to write. They execute slowly. The reliability is mediocre at best. And they are fragile and tend to break a lot because changes in your UI which will break a test (i.e. renaming the ID of a DIV tag you ) are only caught at runtime.
4. I don't believe in MSTest. I miss the Explicit attribute. I miss RowTest. And I miss the fact that you can't qualify your tests as being either Unittests or Integration tests using attributes. It works and does the job - but it saddens me that MSTest is outperformed by miles by NUnit and/or MBUnit.
5. I don't believe in code coverage. Code coverage don't tell you anything about the quality of your code. You can have a near 100% coverage of a class or a namespace while both your tests and your code is unreadable to anything but a compiler - except the programmer who wrote it.
6. I don't believe in databases. Why use a fullblown database if your needs can be met by another storage mechanism - say, a flat CSV textfile or XML document?
7. I don't believe in Microsoft. I don't believe anybody can both read about, have a qualified opinion on, and use every framework Microsoft spits out on a daily basis and STILL get some work done. If there's a term called "framework fatigue" I'm having it right now.
8. I don't believe in fulltime architects. There's a reason that architects in the Roman Empire were 50% architects planning to build stuff and 50% craftsmen actually building stuff - if you don't get your hands dirty on a regular basis you loose touch and feel over the things you're making plans for in your architecture.
9. I don't believe in working through the night. You can't maintain a healthy mind in a healthy body if you work like a maniac because you're under pressure. You will pay deerly in terms of technical debt if you just code away even though your body tells you it is time to take a break and get some sleep.
10. I don't believe in Done. You're never Done - only on the time you stop breathing you're done but until then there's always more you can do.

What don't you believe in?

TDD and mocks

I just found a nice post on TDD and mocks / stubs written by Gabriel Schencker - if you want a thorough introduction to the subject of mocking his post is a good place to start. Check it out :o)

/ Regards K.

SubSonic - day #2

Did you ever see this errormessage: "Can't find the SubSonicService section of the application config file" when trying to get your autogenerated Subsonic DAL to work across multiple projects? Well, I did - and I took the liberty to blog about my bad experiences after trying SubSonic for the first time. Today I decided to tinker a little with the sourcecode of the SubSonic project to see if I could get my ideal configuration scheme set up. My idea was to have a separate configuration file (subsonic.config) placed in the project together with your autogenerated code. Configuration stuff related to database belongs to the DAL layer, I believe, so that's where I would like it to place it physically. I would then add the configuration file to the projects needing dataaccess using the "Add as link" feature in Visual Studio. This way I would have only one configurationfile for my dataaccess which could be shared across multiple projects regardless of their nature (console apps, webapps etc).

I solved it within an hour or two - this is how it was done:

I downloaded the code and quickly discovered that the code basicly traverses the application directory for either web.config or app.config. Afterwards the ConfigurationManager is used to read the various sections into wrapper objects inheriting from System.Configuration.ConfigurationSection - nothing new here.

I decided to attack the DataService class, specifically the static method LoadProviders(). It generates a SubSonicSection class which wraps the SubSonicService section of your app.config or web.config. This is how the code looks:

 

public static void LoadProviders()

{
// Avoid claiming lock if providers are already loaded
if(defaultProvider == null)
{
   lock(_lock)
   {
       // Do this again to make sure DefaultProvider is still null
       if(defaultProvider == null)
       {
           //we allow for passing in a configuration section
           //check to see if one's been passed in
           if(section == null)
           {
               section = ConfigSectionSettings ?? (SubSonicSection)ConfigurationManager.GetSection(ConfigurationSectionName.SUB_SONIC_SERVICE);
          

               //if it's still null, throw an exception
               if (section == null)
                   throw new ConfigurationErrorsException("Can't find the SubSonicService section of the application config file");
           }
      
           ...[snip]...
       }
   }
}
}

 

I modified it method to create a SubSonicSection based on the contents of subsonic.config if it had failed creating a section based on web/app.config:

 

public static void LoadProviders()

{
// Avoid claiming lock if providers are already loaded
if(defaultProvider == null)
{
   lock(_lock)
   {
       // Do this again to make sure DefaultProvider is still null
       if(defaultProvider == null)
       {
           //we allow for passing in a configuration section
           //check to see if one's been passed in
           if(section == null)
           {
               section = ConfigSectionSettings ?? (SubSonicSection)ConfigurationManager.GetSection(ConfigurationSectionName.SUB_SONIC_SERVICE);
          
               if (section == null)
               {
                   string configPath = Path.Combine(AppDomain.CurrentDomain.SetupInformation.PrivateBinPath, "subsonic.config");
              
                   if (!File.Exists(configPath))
                       throw new ConfigurationErrorsException(string.Format("Unable to read configuration file {0}", configPath));

                   var execfg = new ExeConfigurationFileMap();
                   execfg.ExeConfigFilename = configPath;
                   var cfg = ConfigurationManager.OpenMappedExeConfiguration(execfg, ConfigurationUserLevel.None);
                   section = (SubSonicSection)cfg.GetSection(ConfigurationSectionName.SUB_SONIC_SERVICE);
               }

               //if it's still null, throw an exception
               if (section == null)
                   throw new ConfigurationErrorsException("Can't find the SubSonicService section of the application config file");
           }
      
           ...[snip]...
       }
   }
}
}

I then created a subsonic.config resembling the following:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
<section name="SubSonicService" type="SubSonic.SubSonicSection, SubSonic" requirePermission="false"/>
</configSections>

<connectionStrings>
<add name="ExceptionLog" connectionString="Data Source=localhost\SQLExpress; Database=ExceptionLog; Integrated Security=true;"/>
</connectionStrings>

<SubSonicService defaultProvider="ExceptionLog">
<providers>
 <clear/>
 <add name="ExceptionLog" type="SubSonic.SqlDataProvider, SubSonic" connectionStringName="ExceptionLog" generatedNamespace="DataAccess"/>
</providers>
</SubSonicService>
</configuration>

...and placed it in my ErrorLoggerAPI project which contains all the autogenerated SubSonic DAL code in the folder DataAccess:

Next to follow was to add the subsonic.config using Add As Link in my webproject - like this:

Very important: Change the property "Copy to output directory" from "Do not copy" to "Copy when newer" on your linked copy of subsonic.config - otherwise it won't be copied into your buildfolder when you build your project and the following code will fail.

That's it! I now have complete separation between my web project and the corresponding storage working behind the scenes. The only dependency is the linked configuration file I named subsonic.config. When changes occur in my configuration (i.e. a database migration or relocation of the database server) I only have to alter the subsonic.config, rebuild the code in ErrorLoggerAPI and distribute the new subsonic.config to whatever application that's using it.

Last, but not less important: Since the configuration settings are now stored in a file called subsonic.config you will have to create a bat-file to execute the generation of your files. I haven't found a better way to do this so I created a file called deploy.bat and placed it in ErrorLoggerAPI:

The Subsonic documentation (referring the command "sonic.exe help" from the command line)dictates that if you haven't got a web.config or an app.config you will have to provide the database and databasename in the arguments when calling sonic.exe. You can still point to a configuration file where the remaining SubSonic configuration stuff can be placed.

The final deploy.bat looks like this:

"%programfiles%\SubSonic\SubSonic 2.1 Final\SubCommander\sonic.exe" generate /out DataAccess /db ExceptionLog /server "localhost\SQLExpress" /config subsonic.config

Until next time... :o)

Regards K.