Cool it looks like I could build a funky vector space search engine using the mysql cosine function.
...that would be quite cool.
Is it just me that thinks that search engine internals are fun?!?
Not quite. MySQL has a scalar cosine function, like Perl and damn near every other programming language you can name. Because this is a vector based search engine, you need to take the cosine of the two vectors, as the article clearly states:Cool it looks like I could build a funky vector space search engine using the mysql cosine function.
(PDL code for this relation is also provided in the article.)The formula for calculating the cosine is this:Where V2 and V2 are our vectors, the vertical bars indicate the 2-norm, and the * indicates the inner product. You can take the math on faith, or look it up in any book on linear algebra.cos = ( V1 * V2 ) / ||V1|| x ||V2||
Re:Scalar vs. Vector
TeeJay on 2003-02-23T11:47:57
I guessed it wouldn't be that easy;) That confirms my initial concerns that it wouldn't scale well, as you would have to do a lot of mathematical grunt work in Perl (or more likely the C PDL library), as opposed to using a specialised tool like a relational database to the bulk of the processing.
I don't think MySQL would manage the cosine math involved here. Oracle probably would using horrid PL/SQL perversions, but that would subtract rather from the initial elegance.
It would be nice to be able to do simple / fast similar document matches and scoring, but reverse index still offers advantages such as boolean matches and search term weighting and phrase matching that could be hit and miss with vector based searching.
