Yesterday my client, a major metropolitan newspaper, told me they'd like to get an overall architecture designed for the ETL piece of the data warehouse I'm working on. (ETL stands for Extract, Transform, Load: it's how you get data into a warehouse.) So I wrote up a very general diagram and algorithm, and pointed out I really couldn't go any further: They've told me hardly anything about the source systems, and I've never worked in the newspaper business before. But I was told I need to finish up the document.
Well, now I'm writing up all possible ways of loading data. It's general enough to be a "HOWTO of data warehousing". Maybe I'll contact O'Reilly later to see if I can turn it into a book....
A hint for all the managers: you can't design something without knowing what it's supposed to do. It's usually called "requirements".
Clients, gotta love 'em.
I was in a roughly similar situation once where I was tasked with creating a program to summarize data in a system and dump the output for another process to grab, but I knew nothing about the systems involved. In this case, my ignorance paid off. I went to my boss and asked him what I should do. He had me make appointments with everyone involved and find out what their systems did, how they did them, what data formats they used, et cetera.
It was a long process, but eventually I wound up with a specs document that I then circulated and had everyone sign off on. By the time I was done, I knew exactly what I had to work with and what everyone was expecting of me. My piece of the project was done far earlier than everyone else's and whenever I received a change request, it turns out that either someone upstream was sending me bad data or someone downstream was handling my data incorrectly. Aside from a minor upgrade to replace an external app that I was using (it was removed due to Y2K compliancy issues), I didn't make a single change to the program.
Last I heard about the project, it was six months overdue, a million dollars over budget, the business section that it was automating was being phased out, and mine was the only part of it that worked -- all because I started out ignorant.
Re:Ignorance can be bliss
dws on 2002-09-12T18:33:33
Having worked with some of the big 5 4 3 consulting firms, I suspect that their unwillingness to appear ignorant by asking basic questions is a factor in some of the huge cost overruns that we've seen reported.
Humility pays off.
