Informal Survey on Databases

Ovid on 2006-02-17T16:54:30

I'll try to present this somewhat neutrally. There's a long-standing argument about business rules and databases and in my quest to get this written before breakfast, I'm going to oversimplify like mad. Don't shoot me.

Business logic in the application:

  • It's trivial to inspect and change. Stored procedures and triggers are typically hidden.
  • It's more like to be database agnostic. Life will be miserable if you try to port you Postgresql stored procedures written in Perl to Oracle.

Business logic in the database:

  • Business logic and data should not be separated. What if another app writes to the database?
  • There fewer worries that an inexperienced programmer will write code which circumvents the logic.

So my informal, useless survey is basically this:

  1. How often have you ported an app from one database engine to another (e.g., MySQL to Oracle)?
  2. How often have you had completely separate apps writing to the same data store (using completely different code bases)?
  3. Do you depend on non-portable database features?
  4. Have you experienced programmers damaging your data because the logic wasn't in the database layer?

Other thoughts, comments welcome.


survey response

thinc on 2006-02-17T18:21:10

1. How often have you ported an app from one database engine to another (e.g., MySQL to Oracle)?

Never.

2. How often have you had completely separate apps writing to the same data store (using completely different code bases)?

A couple of times. Putting the logic in the database generally also simplifies data retrieval and storage in the app layer.

3. Do you depend on non-portable database features?

Absolutely.

4. Have you experienced programmers damaging your data because the logic wasn't in the database layer?

Not to my recollection, because I place the logic in the database whenever possible.

Real world?

ChrisDolan on 2006-02-17T18:28:09

Heh, my largest database project so far was on a database with data that was so bad that stored procedures were not even realistic due to the amount of special-casing. We couldn't change the schema because legacy applications still did a lot of the work. We made heavy use of accessors and mutators in Perl to make the data look somewhat less hairy to the business logic.

So, I my responses:

1) porting: nearly never (almost 100% mysql)
2) multiple apps: on at least two occasions
3) db features: not often
4) damage: yes, but very rarely significant

Response

Enoch on 2006-02-17T18:41:15

How often have you ported an app from one database engine to another (e.g., MySQL to Oracle)?


Once.

How often have you had completely separate apps writing to the same data store (using completely different code bases)?


I would say 50% of the time (but I'm a Middleware Engineer and not a web programmer).

Do you depend on non-portable database features?


Quite frequently.

Have you experienced programmers damaging your data because the logic wasn't in the database layer?


Actually... not that I can think of.

Replies and nits

VSarkiss on 2006-02-17T19:53:48

You have to be careful whenever you use the word "all" or "always" when it comes to computing. :-)

Generally, the business logic that belongs in the database is usually "data rules", like "an order must have a customer", "a customer must have a name". Business logic that doesn't belong in the database are things like process flow logic. Generally.

Now on to your questions:

How often have you ported an app from one database engine to another (e.g., MySQL to Oracle)?
A few times: Informix to Sybase, proprietary to DB2, and SQL Server to Oracle.

How often have you had completely separate apps writing to the same data store (using completely different code bases)?
Writing, not often; reading, quite a lot. This is a typical pattern in data warehouses and reporting apps.

Do you depend on non-portable database features?
Yes. Unfortunately most databases don't fully comply with SQL standards yet.

Have you experienced programmers damaging your data because the logic wasn't in the database layer?
Depends on what you mean by "damage". I've had people generate garbage reports (and turn them over to management) because the database didn't enforce constraints properly.

Database Usage

ziggy on 2006-02-17T20:55:45

(Thinking about 15 years of projects, some web, some not):

1. How often have you ported an app from one database engine to another (e.g., MySQL to Oracle)?

One migration, from RDB to Oracle, but only because Oracle bought RDB, and that was the only path forward. (Come to think of it, that may have just been a re-branded version of RDB, so there was no real migration to speak of.)

Lots of work with custom data storage solutions (as in, write your own database engine), typically because there was an expectation of managing our data on a customer's server, and out of our control. (Haven't worked on a project like that in ~5 years; may be an artifact of mid 90s economics, big data and prevailing strategy.)

2. How often have you had completely separate apps writing to the same data store (using completely different code bases)?

Pretty frequently. Most common scenario is one codebase for front-end, customer facing apps, and a separate suite for managing the database. (Sometimes it starts as a unified codebase, but diverges, with database mutation going into the backend only.)

In one case, front end and back end were written in multiple languages. (This is where referential integrity really saves your bacon.)

3. Do you depend on non-portable database features?

All the time. Designing database-neutral code is a non-starter for app development. The typical mission is to deliver features by any means necessary, and retarget if/when the current stack runs out of steam.

4. Have you experienced programmers damaging your data because the logic wasn't in the database layer?

Probably. Don't remember anything serious, or at least anything that made it out to a customer.

Worked on a couple of projects where the logic was in the database layer. One specific project seemed like an exercise in job security for the DBA (rules all in the database, impenetrably coded); then again, if you're a DBA for something other than Oracle, that's sufficient job security right there. ;-)

Databases and Data Logic

n1vux on 2006-02-17T21:20:30

Short answer -- If you have DBAs who earn their big bucks, do it their way. If not, ask why not; and do it so that your next project will be easier. Data logc belongs in the DB. Logic specific to a specific use of the data belongs in the specific application.

Business logic in the application:
Stored procedures and triggers are typically hidden. DB Neutral
Logic that is data centric belongs with the Data, so that it's programming language neutral. I'm more likely to have programs in two different languages accessing the same data than to change vendors. Just because I'm using Java for the GUI I should have to use Java DAO's for the batch processing ?? I don't think so, says the Perl guy.

Stored Procs may hide the SQL from the programmers, but embedding the SQL in the 3GL hides the SQL from the DBAs who may be able to optimize the SQL. Same problem as embedding Perl in HTML or vice versa -- a content management system that lets you keep HTML templates so web designers can design 'em without breaking their template-ness and without breaking the embedded Perl (or Python with it's magic whitespace *shudder*) is easier for big projects. Same with big DB projects, you want to have the SQL where the experts can get at it without breaking the Perl or Java.

Business logic in the database:
If the DB has a rich enough Data Dictionary and Relational Integrity, use it. From Day 1 ... it's hard to turn RI on after you've got any data at all. As much of the data semantics as will fit in the DB is better off there, because it's done once, and in a tool designed for it: a DataBase.

How often have you ported an app from one database engine to another (e.g., MySQL to Oracle)?
We've moved a number of apps from MS SQL Serer to Sybase, and from Sybase and MS SQL Server to DB2. I expect no difficulty moving a small DB from MS Access to MySQL when I get the Tuits. Moving to or from ORACLE can be hard; if you need Oracle's richness, use it and love it. If you don't need Oracle's richness, be very very careful about using features that will keep you captive. Moving between the less-extended SQL DBs is easier.

(ex-Oracle users keep asking IBM when they'll add HINTS to DB2. The answer always is that the DB2 optimizer doesn't need your hints, would only be thrown off by them; and if you want a feeble optimizer that needs to be micro-managed by hints so you feel important, you can go back to using Oracle. DB2 admins have better things to do, like optimizing the IOs to increase throughput, or adding an index where even the optimizer can't do better than a tablescan.)

Coding those non-portable Triggers, Stored Procs by hand will require adjustment on a later port. If that's what it takes, it's worth it. But using tools is better.

If using professional DBA tools, those are all auto-generated from the design drawing, porting is a snap. I haven't fully checked the capabilities of the FL/OSS DBA design tools, I don't know how much of the Trigger/StoredProc/RI feature set they support yet. I may need to check it out soon, since I need to port some data from MySQL to MySQL. It will be a hard port because the PHP WebSite schema changed between versions ... different problem

How often have you had completely separate apps writing to the same data store (using completely different code bases)?
All the time! That's life in the big city. Big organizations have legacy systems whose databases need modern GUIs. There are the GUI/On-Line apps for the end-users and the Batch / Real-Time apps that communicate with other applications that provide bulk-updates.

Do you depend on non-portable database features?
Since I learned SEQUEL back when it had vowels in the name, before ANSI/ISO, I usually code in a very portable subset of SQL :-). The first rule of optimization is DON'T. The second rule is for experts only: DON'T OPTIMIZE YET. If I need to use a non-portable feature to do something, I will. Sadly, Stored Procs are still not quite portable ... but the SQL inside them can be. I use Stored Procs on projects where we have real DBAs. Tools that generate good queries on the fly are good too.

Have you experienced programmers damaging your data because the logic wasn't in the database layer?
Sure. If you haven't, you haven't worked on a big project. Even if you have a DB Access Object Layer in the application, someone will code direct to the DB. (Those wiley perl hackers, avoiding calling the JavaDAO!) This is why Stored Procs and Relational Integrity and Triggers will save you on a BIG project. On a small project, one that Catalyst can handle on it's own, all that is of course overkill. But Relational Integrity rules.

ASIDE#1 -- The best database I ever used was WANG PACE. It claimed to be relational, but had some pretty OO features before there was a name for that other than "Post-Relational": It was really class/method oriented without the OO Jargon. The rich Data Dictionary cataloged Types for columns by type names, so natural joins could match the FirstName, LastName fields by type even if the column names were different. Types, columns, tables could all have stored procs (exits, methods) registered for and triggered on events. These "exit procs" could be coded in your choice of SQL, C/C++, or COBOL -- interoperably. Relational integrity was a given; a checkbox at on table-def says to reject add w/o parent or auto-create parent; another says to drop parent if last child deleted or not. With RI having Foreign Key declarations, it was trivial to generate default CRUD screens (Create, Retrieve, Update, Delete) with drill-detail, multiple lines of detail, etc. All on TERMCAP-style green-screens in the late 80's. It's taken 20 years for the industry to catch up, *sigh*

ASIDE#2 -- My Second favorite DB was IBM QBE, Query By Example, which iirc also had RI/FK. You built your query by sketching examples of records you wanted on T-diagrams of the table schemata on a 3270 green-screen, using constants in a column, variables to link joins, and a condition box to constrain relations. Elegant -- but really for on-line ad-hoc query, not for programming.

Survey Response

nmcfarl on 2006-02-18T05:43:55

How often have you ported an app from one database engine to another (e.g., MySQL to Oracle)?

3 times from MySQL to PostgreSQL, but they were all small DBs. A couple times from Access to MS SQL but I'm not sure how much that counts.

I've never done this when there was more than app using the DB.

How often have you had completely separate apps writing to the same data store (using completely different code bases)?

Many, many times.

Do you depend on non-portable database features?

Always - but as little as is possible.

Have you experienced programmers damaging your data because the logic wasn't in the database layer?

Nope.

survey response

plural on 2006-02-18T17:26:29

1. How often have you ported an app from one database engine to another (e.g., MySQL to Oracle)?

only small applications.

2. How often have you had completely separate apps writing to the same data store (using completely different code bases)?

every day. it is maddening.

3. Do you depend on non-portable database features?

yes, but they are mainly isolated to a a library that generates the sql we need, so it doesn't frighten me (well, not completely, anyway) to need to change it.

4. Have you experienced programmers damaging your data because the logic wasn't in the database layer?

i would attribute the damage done to inconsistent application logic that is distributed and duplicated more than to it not being in the database. They can be viewed as one in the same problem, however.

Response.

jarich on 2006-02-19T00:52:15

1. How often have you ported an app from one database engine to another (e.g., MySQL to Oracle)?

Once in 8 years.

2. How often have you had completely separate apps writing to the same data store (using completely different code bases)?

About a dozen (still maintained) projects.

3. Do you depend on non-portable database features?

It depends. Beyond default values, not nulls, primary keys, foreign keys, autoincrement and constraints (cascade delete...) I generally don't encode business logic in the database. Nevertheless even this isn't very portable.

I also use database functions such as "NOW()" and date formats which all seem to vary across databases, although I try to use standards where they exist.

4. Have you experienced programmers damaging your data because the logic wasn't in the database layer?

Never.

"It's trivial to inspect and change." is a biggie

markjugg on 2006-02-19T03:47:50

I've tried before to move some business logic into the database because I thought it should be there.

I got stuck figuring how to write the procedural language code. After fussing with it a bit, I pragmatically backed out, realizing the maintenance programmers were also strong in Perl, not procedual SQL programming.

I rationalized that this decision made the application cheaper and easier to maintain, because a smaller skill set was involved.

In this case, we still able to enforce some rules through the application, and things generally worked out fine.

Re: Informal Survey on Databases

Adrian on 2006-02-19T14:29:09

1. How often have you ported an app from one database engine to another (e.g., MySQL to Oracle)?

A fair number of times. Must be a dozen or so in the last 10 years. Once already this year and it's only February - and this was the second time that this particular application had it's database changed from underneath it.

2. How often have you had completely separate apps writing to the same data store (using completely different code bases)?

A significant minority of applications.

3. Do you depend on non-portable database features?

Often.

4. Have you experienced programmers damaging your data because the logic wasn't in the database layer?

Occasionally.

About as many times as I've had people damaging data by going around the logic in the database layer. Some dumb ass will always find a way to damage data if that's what they're trying to do :-)

Personally I think the location of the business logic depends more on the application, the development team, how it's going to be supported, the size of the application, amount of development time, responsiveness of the DBA, etc. than on more abstract issues of technical merit.

If I've got an application that's talking to a large corporate database used by three other 24x7 applications written in a combination of four different development languages then I'm going to stick the business logic down in the database layer.

If I've got an intranet tagging tool that's being developed for a single department, a technology development path that means we're going to be switching databases in 18 months, rapidly changing requirements and a database department with a 15 working day turn around on change requests then I'm going to put the business logic in the application layer.

Answers ...

drhyde on 2006-02-20T11:38:40

1. I often use SQLite for prototyping and then throw the prototype away and re-write in MySQL or Oracle. I've only once ported a working application from one db to another though, and that was Access to MS SQL Server.

2. Whenever I have seperate applications writing to the database, they're pretty much always just seperate wrappers around a library - eg, a GUI front-end and a command-line front-end.

3. I try not to use DB-specific features, at least in low-end databases like SQLite and MySQL, on the (almost certainly spurious) grounds that I might upgrade later without re-writing the app. Writing overly-portable code is a bad habit of mine. I don't particularly mind using DB-specific features for high-end databases like Oracle though, as the probability of ever moving is so low. And anyway, you usually choose Oracle precisely for its proprietary features.

4. Yes, but the real reason was poor documentation. What they did was correct by the docs.