eann writes "I've been given the opportunity to put on my Advocacy hat and write a short article about how useful Perl is for things other than web-related programs (CGI, mod_perl, etc.) and automating system administration tasks. I doubt it'll actually convince any of the PHBs around here, but if they keep seeing the word 'Perl' enough we might be able to make them believe it's a buzzword. My trouble is, other than some data conversions, I've never really used Perl for anything else. So, I'm asking for examples of Perl success stories that don't involve web or sysadmin stuff."
My non-Web usage sample
tereschenko on 2000-07-05T19:26:03
I use Perl to control Windows TCP/IP stack (some changes in INIs and Registry + file management). And this is only a small sample...
Don't forget data munging
ziggy on 2000-07-05T20:13:23
> I've been given the opportunity to put on my Advocacy hat and write a
> short article about how useful Perl is for things other than web-related
> programs (CGI, mod_perl, etc.) and automating system administration tasks.
Well, don't forget data munging.
:-)
A lot of stuff that looks cute and simplistic with Perl is much harder
to do with C/C++/Java. I've written data mungers for big data in both
Perl and C, and I don't recommend C for this kind of work except in
the most extreme circumstances. I've worked for three online publishers
where data munging was a core part of their value added service.
> I doubt it'll actually *convince* any of the PHBs around here, but
> if they keep seeing the word 'Perl' enough we might be able to make them
> believe it's a buzzword. My trouble is, other than some data conversions,
> I've never really used Perl for anything else. So, I'm asking for
> examples of Perl success stories that don't involve web or sysadmin stuff.
Well, much of the work being done with shops like yahoo, amazon
and valueclick may seem like "just CGI" or "just mod_perl", but that's
ignoring the fact that development has migrated to that model and
away from those early nineties buzzwordy "Client/Server Development Tools".
In that vein, Siemans did a project for the Land Resource System of
Scotland where they used Perl as their n-tier client/server
development environment. Client-side using Perl/TK, intermediate layers
in Perl, and I believe the servers were either legacy systems or
Perl-based.
Perl is a complete development environment that scales well beyond
CGI and sysadminning.
Automation...
bwallace on 2000-07-05T20:57:06
I've mostly used Perl to automate tedious tasks.
I'm an astronomer and, in my last job, I had to process numerous large datacubes as part of of a survey. This required that the processing of the cubes be as similar as possible, and there needed to be a log of the processing in case we needed to trace back problems. The (UNIX-based) software was prompt-driven, and the data had to pass through several progams before reaching its final state. I used perl to drive the individual programs and tie several programs together. In the end, a process that would have used 5 or more programs, needed literally hundereds of keystrokes, and taken anywhere between 3 and 12 hours to complete (depending on processing path) with the user waiting for one process to end before starting the next, now took one perl script with a maximum of 7 arguments. Saved huge amounts of my time, allowing me to go on to other tasks (such as creating perl scripts to automate the analysis of the resulting data for science).
At my current place of employment, we use perl to drive our radio telescope. Low level tasks (such as moving the telescope to a new position, tracking the object, etc) are written in C, but perl scripts are used to drive each run, log the results, and then parse and analyse the data (I've been using PDL for the analysis). At this point in the telescope testing process, many of the more tedious tasks could not be done effectively without perl.
Re:Don't forget data munging
tnight on 2000-07-05T21:03:19
I've been programming in Perl for about five years, and I've used it for quick hacks, writing distributed publishing tools, generating reports, controlling jobs on remote hosts, scraping information from web pages, and sending emails, among other things like complex CGI and FastCGI applications.
I think Perl is sufficiently general-purpose to satisfy almost any application need. The real question is whether it's the best tool for your specific need.
Sidebar from article in IEEE Software Magazine
Dominus on 2000-07-05T23:43:57
A couple of years ago IEEE Software magazine carried an article on Perl. There was a sidebar with a list of commercial perl uses:
Perl was invented in 1986, when Larry Wall, then at the Jet Propulsion
Labs, needed to generate reports from their configuration management
system. Perl continues to be used in CM settings; US West uses a
software configuration management system written entirely in Perl.
The Human Genome Project, whose goal is to sequence the entire human
genome uses Perl programs to manage and collate the huge amounts of
genetic data they have. In extracting genetic material from a human
cell, the information is broken into short segments. Perl programs
correlate and re-assemble thousands of short segments back into the
original sequence.
In issue #7 of The Perl Journal, Will Morse describes how
geoscientists at BHP Petroleum use Perl as for data management and
preparation of data for sub-salt petroleum prospecting.
The University of Pennsylvania's computer science department uses an
expert system to help filter users' requests for more disk space.
Before the expert system, a staff member had to respond individually
to every request, often having to exchange several messages with each
user to find out whether the request should be granted. The expert
system knows when to grant a request automatically, and how to
assemble and mail a form letter with all the necessary information, so
that the computer staff can decide right away how to proceed.
Majordomo, probably the most widely-used mailing-list management
software in the world, is written in Perl. It automatically
subscribes and unsubscribes users from thousands of electronic mailing
lists world-wide.
In 1996, the Asociation for Computational Linguistics decided to
accept submissions to their annual conference via E-mail. A Perl
program automatically received paper submissions; it formatted,
printed, and filed them, and returned confirmations (or error reports,
if the paper was not in the correct format) to the authors by mail.
Bank of America uses a Perl program, devloped by Kirby W. Hughes, to
stress-test BA's home-banking system. The Perl program contacts the
banking system over the network, sends many requests to it at once,
and reports trouble if the replies are not what they should be, or if
there are no replies at all.
Re:Automation...
cs on 2000-07-06T00:35:11
The previous post already mentioned PDL in passing. PDL (the Perl Data Language) is a full fledged array processing language module (a la IDL, matlab) that opens up the world of scientific data analysis to the perl hacker. Find more details at
http://pdl.perl.org.
Client / Server programming
frosted on 2000-07-06T02:03:05
Perl makes it easy to write graphical clients with Perl/Tk or gtk... It is also aweful handy for the server than sits behind it. A recent example from my own experience is a system that receives data from a large number of users (non-web based) and queues up jobs for various sorts of processing. It computes summary statistics and reports those to questioning Perl/Tk network monitoring applications.
Perl used for testing
supersalo on 2000-07-06T07:53:43
We only use Perl for testing our software.
We have built huge perl libraries for connecting to various devices (routers, unix machines, etc).
Our tests can be developed quickly (since we look for "string" output!) and adapted to any new "devices" that come along.
Uses of perl
oops on 2000-07-06T09:53:20
I've adopted the policy that if I do something twice, then it's a candidate for scripting. Consequently I end up building a huge armoury of tools using Perl.
These include simple scripts for performing basic tasks consisting of a few commands, plus some elementary error checking. At the other end of the scale, at my last position I built the complete development/test/release mechanism in Perl, tying together all the development tools and providing generation of web logs, auditing etc. Once the system was released, a set of install/verification tools took over to provide configuration and sanity checking. Finally the production support tools were all Perl based, and a number used the Tk interface so even the PHBs could work out what was going on.
Needless to say I provided a lot of the CGI-type work as well. On the development side I wrote prototypes in Perl plus a bunch of useful tools to aid in development. (eg. check out classfind and jdbtool - actually these were developed at home but you get the idea).
The bottom line is that I was employed as a Java/CORBA programmer, but a lot of my work was in Perl, making my working day less tedious and error-prone through the automation of tasks and provision of tools. Once I followed the 'if you do it twice, script it' paradigm, then you'll see a lot more uses for Perl.
Everything
jns on 2000-07-06T11:08:16
I work for an ISP and most of my work revolves around the billing system which itself is largely written in Informix 4GL - however all the bits where it needs to interact with other systems and where it needs to do fancy output are written in Perl, all reports (since I have been here) are written in Perl and of course all of the stuff where the system has to interact with the web.
We also use Radiator which is an authentication server written in Perl.
/J\
Using Perl for Educational Admin
Patrick on 2000-07-06T12:28:35
We use a set of perl scripts to 'skim-read' students' 'English 101' essays and point up possible areas for teacher attention. They don't actually
replace the teachers reading and commenting on the texts (honest!), but they give them some useful information such as sentence length, use of key grammatical structures and type-token reference (a measure of language variety).
Perl's ability to slurp whole files through a sequence of regexes and spit out useful comments has proved very useful for harried teachers with a heap of 'what i did in my holidays' essays!
Following its success, we have discussed how much of 'Strunk and White' could be rendered into regex form... some bits would be easier than others!
We use it for automated testing
bhilton on 2000-07-06T13:21:28
Our product is sort of an ATM router with various
interfaces, including T1, E1, T3, and OC3. Our
entire automated testing system is written in Perl. With this system, we can control the product via SNMP, as well as controlling various physical devices... test sets, data switches, power systems, even an oven!
Of course, the interface to the system is a Perl
CGI program, which is integrated with our Perl CGI test case tracking system and our Perl CGI requirements system.
Development
Two major uses we have, are
- Maintaining cross-references in a pile of requirements documents, specification etc. We check cross-references to other documents and
automatically create back-references. And since the whole lot is guaranteed to be accurate, designers tend to keep the documents up to date.
- Code generation from an xml repository. Perl is used to translate program templates into perl scripts which use the repository data to write programs of arbitrary complexity. This means that system layers such as service wrapping and persistence can evolve very rapidly
and still stay coherent.
Currently under construction, an automated system build, deploy, test and report facility.
In fact Perl has become an invaluable developer's tool box.
Perl as a "Virus Scanner"
Scorch on 2000-07-06T15:39:55
The company I work for can't afford to pay for a commercial MS Exchange virus scan package (plus we don't want to lose MS's support for Exchange). So I wrote a Perl script that de-taches attachments and waits for a virus scanning program to scan them. If they have been cleaned, I reattach them. It was pretty easy to do using the CDO library.
Re:Perl as a "Virus Scanner"
thisbox71 on 2004-08-19T18:00:52
Wow.
Would it be possible to get that code?
Thanks,
Rob Hill
rhill@thisbox.com
Re:Using Perl for Educational Admin
frosted on 2000-07-06T16:45:04
This is interesting. Can you make the code available? Also, are students at that early educational level submitting essays digitally now?
Used very heavily on wall street...
TheOtherGuy on 2000-07-06T19:45:09
Perl is used very extensively in throughout Wall Street; particularily on the trading desks. Its used for all kinds of process (grabbing and organizing market data (prices), product data (financial instruments: stocks, bonds), company data); and then doing something with that data (like coming back with a list of stock trades.)
Not a lot of CGI stuff is used; generally its easier just to dump it to the screen, into a database, a mail message, or a file. A lot of the stuff is on Unix (Solaris).
Computer Performance Analysis
moonbeam on 2000-07-07T00:36:07
The field of computer performance analysis is data rich. but the tools for analysis are rare. Perl's flexable data structures makes the task of data analysis more structured and easier than using C or it's must faster than awk.
Thrashing reports
Silver on 2000-07-07T02:52:07
The bank got a lot of data from their ATM/debit card provider (Visa), but it was all in pre-chewed report form. The reports changed semi-randomly, had variable numbers of lines per item, all sorts of mean, nasty, ugly things. After the senior programmer spent a couple of days trying to get RPG to account for every possible exception and so on, I tossed out a Perl script in about an hour that thrashed the nastiest of the reports into a nice flat data file.
Now they use Perl to pre-process data for RPG programs regularly. The heavy stuff still happens on the AS/400 in RPG, but anything that needs to be a "write programs quicker" instead of a "write quicker programs" happens in Perl.
(If I was still there, I'd have Perl on the AS/400 and start taking over the "quicker programs" side of things too.)
clothing
mstevens on 2000-07-07T12:56:26
Going off at a tangent, should we have an _actual_ advocacy hat? what perl clothing is available?
data warehousing; individualization, etc.
stolkin on 2000-07-07T18:37:14
I've used Perl for >7 years and never written a single web related program (no CGI, no mod_perl etc.)
The largest program in production is about 3000 lines. It does automated dimension table management in a star schema data warehouse. (If you don't know what that means read Ralph Kimball's book, The Data Warehouse Toolkit, and look up slowly changing dimensions.)
Another fairly big projects did text summarization of a collection of documents.
Another fairly big program computed the "connected components" of an undirected graph of data, to do "customer individualization". (Another industry buzzword; you can look in http://www.google.com if you're curious.)
I have a total of 261 programs with the extension *.pl, and there are several others without any extension because I wrote them to replace shell scripts. (Plus another 48 programs that I copied from someone else.) There are programs to:
* generate XML from a database
* generate SQL DDL to create table and indexe, etc.
* generate SQL queries
* clean and preprocess various data files, for loading, performance analysis, etc.
* do simple statistical analysis
* do a variety of text reformatting
* check that the files in the RCS directory are in synch with those in
../RCS
* drive a makefile, or diff, or copy, or zip or other programs
* do joins that would be too awkward to express or too slow to run in a relational database (e.g. outer joins mixed with calculations)
* try to teach bubblesort to first graders (the $ makes the variables easier to see)
etc., etc.
Hopefully helpfully yours,
Steve
--
Steven Tolkin steve.tolkin@fmr.com 617-563-0516
Fidelity Investments 82 Devonshire St. R24D Boston MA 02109
There is nothing so practical as a good theory. Comments are by me, not Fidelity Investments, its subsidiaries or affiliates.
Content Management
Illiad on 2000-07-07T19:55:44
I use perl to provide a system of content management, which delivers MPEG/2 files one of 8 Plan 9 Based VideoServers based on a database schedule. It handles load ballancing, content management and disk-volume management and comprehensive error reporting (so we know well ahead of time when a channel's going to drop out). The entire set of 6 applications communicate via SNMP and mail for the more remote systems.
In theory, it could all be written in a shell script or two, but I'd loose alot of the error handling I do, and it would rely on about 8 separate languages instead of one.
--A
My best perl experience
gej on 2000-07-09T23:26:09
I worked doing support in the manufacturing division of a once great work station maker whose initials are sgi. One piece of the manufacturing puzzle we had to contend with was a beast known as an ASRS, an automated storage and retrieval system. Think of two tall and long bookshelves facing each other with a crane on a track between them. The crane would pick up systems from an inbound conveyor, put them in a place on the shelves, inform the database (through an application known as LAWS (Litton Automated Warehouse System)) which would inform the watchdog software which would start automatically testing the system. Eight hours later (or less, depending on when during the quarter it was) the ASRS would be told to pick up the system and deliver it to the outbound conveyor to be sent to either reject analysis (if it failed) or final config (if it worked).
A nutty chain of things to happen, with one real weak link: LAWS. It got confused often about what and where it would be putting things. Recovering from physical problems, like if something blocked the crane and it had to stop, or if it couldn't stick the system into it's bay because the bay was full, or if the ASRS was full, was a non trivial task, and database corruption became a major problem. Physical reconciliations were a daily occurance. My first real perl script was to print out a snap shot of what our product database (not the LAWS database) thought was in the ASRS, in a nice map to make the reconciliations go faster.
It became painfully obvious to us after only a short time that the LAWS application should be replaced. And we knew perl could do it, and do it well. The problem is the suits trusted a piece of crap software system that they paid upwards of $20k in support per year more than they trusted a couple of young guys with a mysterious language known as perl. It didn't stop us from saying to them that if they gave us the $20k and a sixpack, we'd give them a better piece of software. And we said it often.
Then one day it happened. One of the younger and more technically savvy managers was on the shop floor when LAWS went south in a big way. I think it took us the better part of a day to get it back up and running. I had rewritten (and vastly improved) part of the shopfloor conveyor system in C earlier in my career, so this manager decided to take a chance and see what I could do with the ASRS.
Another co-worker and I began work. He took a copy of the ASRS technical specs and worked out the serial communication necessary to the hardware. I spent my time devising the data structures (a couple of hashes tied to dbm files) and throwing test cases at them to work out all the problems we saw with LAWS. It took us about two months to crank it out, largely because we were not trusted to work alone on the project and had to answer (and spend many hours meeting with) the testing department. Once done, we put the two pieces together, spent a couple of late nights working out the kinks, and then flicked the switch. It worked, and it worked beautifully. Support needs dropped to nothing. Where once support personnel were called at least three times daily to the floor for ASRS issues (nothing was coming out, nothing was going in, etc.) now they got called once a week. Most of the recovery tasks I either automated or made so foolproof anyone could follow them. Bye bye data corruption, hello smooth operations. It was very much a night and day difference.
Unfortunately, this all happened late in the ASRS life cycle, and it only lived on for another 8 or so months. But those eight months went very smoothly, and freed up a lot of support people for other projects.
To be fair, LAWS was designed for a much more static environment that didn't require as much outside communication as we needed. It had a lot of additional functionality that just got in the way of what we wanted. With perl we were able to cut out everything we didn't need. Plus we had the source code available to us, so changes could (and were) made quickly and easily.
We never did see the $20k, though.
Perl in the middle tier
dws on 2000-07-10T20:41:55
An Enterprise product that I'm involved with has has most of its middle-tier, a set of single-threaded application servers, coded in Perl. The servers are big. The Perl codebase is about 100KLOC.
The folks who built the system agree that it couldn't have been done on time (before funding dried up) if they'd had to use C++ or Java, and performance is good enough. We use a half dozen or so CPAN packages, which saved a few months of developement.
Uses of perl
Bruce on 2000-07-19T19:15:40
I am a physicist working in the field of x-ray absorption spectroscopy. I use Perl for a wide variety of professional chores, ranging from throw-away data processing to geometrical calculations for programming goniometer motion, to large complex object-oriented crystallogrphy calculations.
One of my professional projects is a data analysis tool which deals with crystallographic data. Once upon a time my little program was written in Fortran, but about two years ago I rewrote the whole thing in Perl. Now it is does more stuff, has graphical and web interfaces, and is easier to maintain and extend.
I would find science much harder to do without Perl.
Re:Don't forget data munging
gcrookham on 2001-03-29T13:50:20
>In that vein, Siemans did a project for the Land Resource System of
>Scotland where they used Perl as their n-tier client/server
>development environment. Client-side using Perl/TK, intermediate layers
>in Perl, and I believe the servers were either legacy systems or
>Perl-based.
It's a system we (Siemens) implemented for Registers Of Scotland and is 100% Perl on both Client and Server side
