I like this idea. Why not make an ISAM and more filesystem. Layer files and directories on as tables, and then think about full text indexing, the death of CSV, and more.
--Nat
Huh. Sounds suspiciously like how Be OS was originally designed.
Re:Be
Theory on 2002-03-14T06:11:43
Actually, that should have been "BeOS", or, more specifically, BFS.
We really need an alternative to complement the directory structure. Something like a light database that indexes files on their type, date and other keywords (I am not even talking about indexing the content here), so you can just select the files you want to list. I am really tired of looking all over the system for files when I know what I want but not where it is.
Of course the problem is to generate as many of the keywords automagically, with the user being able to add more in a simple way... quite a tall order I guess! Especially as such a system would need to be aware of all file I/O to be complete. But maybe an improved slocate would be a good start, any volonteer?
Re:Alternative to directory structure
Matts on 2002-03-14T10:22:06
The problem is speed. Ever worked up to or past midnight (or whenever your slocate cron job runs)? The system slows to a crawl as it thrashes all over your filesystem. Now imagine that also doing full text indexing? Yikes.
Plus I think there are a lot more questions with complex filesystems than there are answers, for instance the problem with sending a file from such a filesystem via email.
Personally I'd rather keep filesystems simple, and work out funkier ways to actually work over the top of those. Maybe DBD::SQLite holds the key;-) Re:Alternative to directory structure
mirod on 2002-03-14T12:15:03
The problem is speedI realize this, plus I am not so sure full text indexing is necessary. Not if you can narrow down the list of files to grep by date, type, and maybe header fields for emails.
I guess it's buzzword time
;--) What we need first is probably ontology to tell us how to describe files better than just through a list of names (directories and file name) and a 3 letter extension (or a Mime type, or neither). Then we can start having fun implementing all sorts of funky file systems, or layers on top of the file system (DBD::SQLite works just fine BTW, I love not having to worry about rights management when I write the prototype of a product). Re:Alternative to directory structure
ziggy on 2002-03-14T14:26:51
I read an article on O'Reillynet some time back about a user who was migrating to OS X (it didn't jump out at me looking at the index). One of the really nasty things about dumb full-text indexing is full-text indexing every damn file -- including MP3s.I realize this, plus I am not so sure full text indexing is necessary.The problem is speed:-) Taking the BFS approach, full-text indexing is within reach, once you intelligently focus on text files. After all, what's the point of indexing executables, PDFs or PNGs? The heuristics to reduce the number of files to be indexed are quite simple. After all how many files do you touch on a daily basis that are text files? And how many of them actually change in any given day?
Also, the speed cost at 2am for running slocate or mklocatedb is partially because it is dumb, and partially because it does everything. Amortizing the cost over the course of the day such that only active files are checked shouldn't impose such a huge penalty.
And, of course, DBD::SQLite is likely part of the solution to the problem. The BFS idea of making the filesystem into a database sounds nice, but I don't think the *NIX world is quite ready for that just yet.
:-) Re:Alternative to directory structure
pudge on 2002-03-15T17:07:50
MP3s, JPGs, and PNGs have comments in them. Might you want those indexed?Re:Alternative to directory structure
ziggy on 2002-03-15T18:29:43
The comments indexed? Sure.Full-text indexing the rest of the binary data structures in those file formats is not likely to turn up any other hidden messages though.
:-) Re:Alternative to directory structure
darobin on 2002-03-14T15:03:58
I guess it's buzzword time
Man, we have it, we have it!!! I mean throw away those old clunky ideas of a Semantic Web, what we want is a Semantic File System, YAY!!! Lets get a startup rolling, lets become billionaires
:-) Seriously though, I'm not sure what but this strikes me as a good idea. Maybe it's those mushrooms from last night...
Re:Alternative to directory structure
belg4mit on 2002-03-14T15:11:32
Funny how nmh has pick for that:-D
alias pscan "scan `pick !:*`"
pscan --from Bob
pscan --subject Perl
...Re:Alternative to directory structure
belg4mit on 2002-03-14T15:12:28
Of course mgrep might make a more
intuitive name...?Re:Alternative to directory structure
djberg96 on 2002-03-14T19:42:41
As someone else has already mentioned, BeOS tried this originally and tossed it out due to speed problems. But then, hey, speed has never been an issue that's stopped Sun's Java. Why should it stop Microsoft? Maybe advances in hardware will hide any problems.Being able to add my own filesystem attributes was one of my favorite things about BeOS, though. With OpenBeos making progress, I might make it a goal to get back into Perl & Ruby on the BeOS platform. Perl 5.6 never was ported properly, though, so I guess I'm in for some serious work.
Re:Alternative to directory structure
Matts on 2002-03-14T20:22:52
NTFS already supports alternate streams in a single file - you can use this for attributes if you want.Re:Alternative to directory structure
ziggy on 2002-03-15T15:14:05
Not in the demo's I've seen. Back when Be was releasing BeOS 3 or BeOS 4 (the first release for x86), they demoed BFS and Tracker together. First, they created a live query on the filesystem to find some kind of file. Next, they opened it up in Tracker. Next, they added some files to the filesystem and the new files magically appeared (and the old files magically disappeared).As someone else has already mentioned, BeOS tried this originally and tossed it out due to speed problems.Next, they demonstrated processing 3 or 4 simultaneous Quicktime movies in realtime. Then they drop the box into single-CPU mode, and a few frames per second were being dropped (you had to have a really good eye to notice).
At the end of these demos they claimed that BFS was offering 96% of the maximum theoretical throughput on these 9GB IDE drives.
Perhaps the numbers are a lie. Perhaps they dropped BFS after 1998 or so. But what they did demonstrate was fast and impressive. From the few weeks I was running BeOS, I can say that it wasn't a faked demo, either. That OS and filesystem were damn fast.
Re:Alternative to directory structure
djberg96 on 2002-03-15T21:37:53
Actually, I was referring to the *original* attempt by BeOS at an RDBMS turned filesystem. I'll have to look up the info in my BeOS Bible.By all accounts, BeOS is *still* one of the fastest and best filesystems to ever hit the market. I'll take it over Linux/Solaris any day. I'm actually planning on re-installing it, in lieu of the progress being made with OpenBeos.
From what I've read, the folks working on OpenBeos have already made improvements by as much as 600% in some areas over the original BeOS.
:)
