I have been working pretty hard on a work project for about a month now (off & on and for the last week, mostly on) and I've come to a realization that perhaps most of you already have.
For the impatient, here's the lesson up-front:
When tasked with importing data from an external source, consider importing it into a separate db/table and then building/extending the necessary functionality on top of that (versus importing the data right into your existing data).
We run circulation data for two magazines, both on systems we built ourselves. It was decided to outsource circulation of one of these to another vendor. Several months later, I was asked to import their data for some functionality that the vendor doesn't provide. As it turns out, there are some similarities between the schemas of their data and ours, but mostly differences. A big one being when a user renews their subscription, I treat that as a separate subscription and they treat that as an extension of their existing subscription.
Anyway, like I said, I've been working hard and feel like I'm currently at 85 or 90% completion, but to nail the final non-conformities would require user-specific code, which makes my eyes bleed when I'm already facing some messed-up code (5 or 6 main IF branches and a few places that could be re-factored).
Perhaps this is when I should move to logic programming vs. functional programming, but I still don't have my head around that one. :(
Perhaps also, I'm just at a point that all programmers reach when they've invested too much time in a project and question
"WHY?" and I should just buckle down and knock out the remaining 10-15% cases.
Cheers,
Jason
Absolutely!
CromeDome on 2006-09-28T15:01:44
When tasked with importing data from an external source, consider importing it into a separate db/table and then building/extending the necessary functionality on top of that (versus importing the data right into your existing data).Data import is a large part of what we do. We collect property appraisal data, tax collection data from banks.... the list goes on. There are so many reasons this lesson is valid. If nothing else, if there is a bug in the import, you don't have to go through the hassle of actually parsing the file (or whatever) again - just tweak your program code and run with it.
In our case, since we deal with a lot of numbers, if people find a problem we immediately get bitched at. It's a trivial thing to write some SQL to prove the numbers we imported were those in the file. "Go beat on the people that generated this file! See, we balance to it. It's not our problem."
Good post, Purdy
:)
