Cowboy Neal with Hat
(concentrate on your petition)
Perl Monks 

Tricks with DBI

by btrott on Apr 14, 2000 at 01:36 ( print, xml )
log zgrebe out | zgrebe | The Monastery Gates | Super Search | Snippets | Cool Uses For Perl | Poetry | Code | Obfuscation | Q&A | Library | Seekers of Perl Wisdom | Craft | Meditations | Perl Monks Discussion | Perl News | Reviews | Tutorials | Newest Nodes | Offering Plate
Need Help??
++ --

Tricks with DBI

1. Check for database errors.

You're going to run into errors with databases for similar reasons that you do when using system calls and the like. And just as you should always check the return code of your system calls, so should you always check the return status of your database calls.

The easiest way to do this is by setting DBI's RaiseError attribute to 1; first connect to the database (and check the return), then set the RaiseError attribute:


    my $dbh = DBI->connect('foo', 'bar', 'baz', 'mysql')
        or die "Can't connect: ", $DBI::errstr;
    $dbh->{RaiseError} = 1;
By doing this, you ensure that any database error will cause a die. Why is this good? Because generally, if you're writing a database application and you have a database error, you don't want to continue as if nothing happened. :) (Besides, you can always catch the die in an eval; just make sure that you handle the errors rather than ignoring them.)

The other way to check for errors, of course, is to check the return of each method call, eg.:


    my $sth = $dbh->prepare("select id from foo")
        or die "Can't prepare: ", $dbh->errstr;

    $sth->execute
        or die "Can't execute: ", $dbh->errstr;
So as not to make your code one big C-like mess of error checking, though, you might as well just use RaiseError.

2. Use placeholders instead of literal values.

This is covered pretty thoroughly in What are placeholders in DBI. Suffice it to say here, then, that you should use placeholders instead of literal values. Always (or pretty much, at least).

And, for the same reasons, you should use prepare_cached instead of prepare.

3. The fastest way to fetch.

When you execute a SELECT statement, you want to get the data back as quickly as possible. The fastest way to do this is to use the bind_columns and fetch methods, because they don't copy a bunch of memory around.

bind_columns binds Perl variables to columns returned from your SELECT statement. For example, if you had the following SQL statement:


    select id, name, phone from people
You'd want to bind 3 variables to the associated columns. So you set up the variables, then use bind_columns to bind them:

    my($id, $name, $phone);
    $sth->bind_columns(undef, \$id, \$name, \$phone);
(The first argument to bind_columns is actually a hash reference specifying DBI attributes to associate with this particular method; we don't want to associate any attributes with this particular method, so we'll just pass it undef.)

After you've called execute on your statement handle, you'll want to fetch the data. To do this, use the fetch method, which fetches the next row and returns an array reference holding the field values. But you don't need to use that array reference, because you've got the column values bound to Perl variables, and you can just use those directly:


    while ($sth->fetch) {
        print join("\t", $id, $name, $phone), "\n";
    }
Each time you call fetch on your statement handle, the values of the bound variables get updated.

A Brief Interlude

Putting Tricks 1-3 together, we get something like the following:

    # Connect to the database and set the RaiseError
    # attribute so that any database error will
    # cause a die
    my $dbh = DBI->connect('foo', 'bar', 'baz', 'mysql')
        or die "Can't connect: ", $DBI::errstr;
    $dbh->{RaiseError} = 1;

    # Setup our statement handle using a placeholder
    my $sth = $dbh->prepare_cached(<<SQL);
select id, name, phone from people
where birth_month = ?
SQL

    my @months = qw/January February March April May
                    June July August September October
                    November December/;

    # Bind Perl variables to columns returned
    # from SELECT
    my($id, $name, $phone);
    $sth->bind_columns(undef, \$id, \$name, \$phone);

    for my $month (@months) {
        print "People born in $month:\n";

        # Execute the statement for this $month
        $sth->execute($month);

        # Fetch each row and print out the values
        while ($sth->fetch) {
            print "\t", join("\t", $id, $name, $phone), "\n";
        }
    }

    $sth->finish;
    $dbh->disconnect;

4. Using DBI with mod_perl.

Part of the purpose of mod_perl is to make Perl programs run much faster on the Apache web server. mod_perl accomplishes this by building a Perl interpreter into the web server and compiling your programs in memory; thus, when it gets a new request, it maps that request onto a program compiled into memory--this is very fast.

If you're going to use DBI with mod_perl, you'll want to make sure that your database transactions are fast, as well (because you don't want the database being the bottleneck). The easiest way to accomplish this is to use the Apache::DBI module, which gives your programs persistent database connections. It does so by overriding the DBI connect method and keeping a cache of open database handles. One of the nicest things about Apache::DBI is that you won't have to modify your existing DBI code to use it. Just add


    use Apache::DBI;
BEFORE the

    use DBI;
in your code, and you're set. You don't even have to take out the calls to disconnect, because Apache::DBI overloads those calls and makes sure that the handle doesn't actually get disconnected.

See Also

The new Programming the Perl DBI; Mark-Jason Dominus's A Short Guide to DBI; the DBI manpage (also (probably) available as perldoc DBI).

comment on Tricks with DBI
d/l code
RE: Tricks with DBI
by nutate on May 30, 2000 at 20:53
++ --

    Very tasty intro. I have been doing some ugly PostgreSQL/ Perl CGI stuff at work, but I am working on prettying it up and modularizing it. Hopefully, I will be putting some of it up on perlmonks.

 [reply]
RE: Tricks with DBI
by Michalis on May 31, 2000 at 08:40
++ --
    Well done. Some corrections if I may.
    (a) bind_columns SHOULD be after execute has been called. As DBI's manual says, that's for maximum compatibility.
    (b) You call bind_columns with four parameters (the first one been undef) while your execute only returns three (id, name and phone). While I have never used bind_columns before execute, I beleive that should fail.

    My way is similar but not the same (I include just the different part):

    
    my $sth = $dbh->prepare_cached(<<SQL);
    select id, name, phone from people
    where birth_month = ?
    SQL
    
    for my $month (@months) {
      print "People born in $month:\n";
    
      my($id, $name, $phone);
     
      # Execute the statement for this $month
      $sth->execute($month);
      $sth->bind_columns(\$id, \$name, \$phone);
      # Also valid:  $sth->bind_columns(\($id, $name, $phone));
      # Fetch each row and print out the values
      while ($sth->fetch) {
         print "\t", join("\t", $id, $name, $phone), "\n";
      }
    }
    
 [reply]
++ --
      Thanks for the suggestions. You're quite right about bind_columns being called after execute; I've been bitten by that in the past.

      As for the 4 arguments to bind_columns, and the first argument being undef... I believe this is an anachronism of DBI. I just looked at the manpage, and you're right that it seems to only expect 3 arguments, now.

      In the past, though, I *think* the first argument to bind_columns was supposed to be a hash ref. So I got into the habit of using undef as the first arg, because I never cared about passing anything in the hash ref. I can't find that in the DBI manpage anymore, but I did find this:

      
          bind_columns
              ....
              For compatibility with old scripts, if the first
              parameter is undef or a hash reference it will be
              ignored.
      
      I don't remember what the hash ref was supposed to hold--nor would I, really, since I never actually used it. :) I always just used undef. Thanks for pointing this out.
 [reply]
Re: Tricks with DBI
by coolmichael on Mar 11, 2001 at 00:58
++ --
    Besides, you can always catch the die in an eval; just make sure that you handle the errors rather than ignoring them.
    I haven't seen anything like this before. Could you post an example? Thanks.

    Michael
    who just became a monk

 [reply]
++ --
      This is documented in eval. Here is how it works:
      
      eval {call_function_that_may_die()};
      if ($@) {
        # This is your error case
        print "Caught a die saying '$@'\n";
      }
      
 [reply]
++ --
        Instead of using die, you can unset $dbh->{RaiseError} locally and use the other C-like error check.

        Ths is usefull when you have a SQL-statement that can fail, but where you don't want to die because of the failure. (I have used it when dropping temporary tables, that perhaps aren't there and other suchlike tings).

        The code looks like this:

        
        my $dbh=DBI->connect(....{RaiseError=>1}) or die...
        my $sth=$dbh->prepare(...);
        {
           local $dbh->{RaiseError} = 0;
           $sth->execute;
           if ($sth->Errstr) {
               # handle the error
           }
        }
        # $dbh->{RaiseError} is back to normal here
        
        The neat thing about setting $dbh->{RaiseError} with local is that it is automagically set back to whatever it was when you leave the block, however that is done - even if it is by way of a die, that is caught in an eval somewhere else.
 [reply]
Re: Tricks with DBI (esp. Connect)
by htoug on Aug 15, 2001 at 10:33
++ --
    In newer versions of DBI the connect call has changed (the old one still works, but is depreceated). The call should be:
    my $dbh = DBI->connect("dbi:mysql:foo", "bar", "baz",
          { AutoCommit => 1, RaiseError => 1})
            or die die "Can't connect: ", $DBI::errstr;
    
    Where the "dbi:mysql:foo" bit is "dbi:'DBD-driver-name':'database-name-and-options'".
    You will have to see in the documentation for your chosen DBD to see what is possible and legal for the 'database-name-and-options' part.

    RaiseError is set in an anonymous hash - along with other database attributes.

    Note that I set AutoCommit state in the connect call. This is highly recommended, and will become mandatory in a coming DBI release.
    The reason is that the ODBC spec (which DBI follows as closely as possible) has AutoCommit=>1 as default, while many (most?) major database systems have AutoCommit=>0 as default. This has lead to much confusion and tearing of hairs in the past, as things were commited unintentionally.

 [reply]
Re: Tricks with DBI
by Arguile on Sep 17, 2001 at 23:34
++ --
    "One of the nicest things about Apache::DBI is that you won't have to modify your existing DBI code to use it. Just add use Apache::DBI; BEFORE the use DBI; in your code, and you're set."

    A great article, but this line could potentially be misleading.

    The Apache::DBI module is completely transparent; so you don't need to touch your scripts for any reason. Where Apache::DBI actually is defined, is in Apache's configuration file (httpd.conf by default).

    Sample mod_perl section of httpd.conf

    
    ...
    # preload these modules
    PerlModule Apache::Registry;
    PerlModule CGI;
    PerlModule Apache::DBI;
    PerlModule DBI;
    
    # startup script
    PerlRequire /path/to/startup.pl
    ...
    

    If you have a lot of modules to preload, or to use Perl syntax as illustrated above by bttrot, you can define them in the startup script instead. The rule about Apache::DBI going before any other DBI modules still applies.

    Just thought I'd clear that up for anyone who read it to mean, "Add use Apache::DBI; to your CGI scripts."

    See Also

 [reply]
Re: Tricks with DBI
by runrig on Oct 19, 2001 at 00:10
++ --
    my($id, $name, $phone);
    $sth->bind_columns(undef, \$id, \$name, \$phone);
    
    A shorter way is:
    
    $sth->bind_columns(\my ($id, $name, $phone));
    
    And regarding this statement:

    And, for the same reasons, you should use prepare_cached instead of prepare.

    Choosing prepare or prepare_cached really depends. The caching is a slight overhead, so if I can reasonably arrange my SQL so that something is only prepared once (or if its used just once anyway), then I'll just use 'prepare'. Or if you're dynamically creating a SQL statement (lets say a million times) and the number of possible unique combinations is large, then you don't want to use prepare_cached, e.g. you have a couple of 'IN' clauses, and each one might have 1-100 elements, so you end up with something like:

    
    select stuff
    from table
    where field1 in (?,?,?,?)
    and   field2 in (?,?,?,?,?)
    
    (Note: placeholders are still a good idea in this case)
 [reply]
++ --
      And, for the same reasons, you should use prepare_cached instead of prepare.
      Never prepare_cached() ping calls, because you will get true each time instead of actual ping results.
 [reply]
Re: Tricks with DBI
by mpeppler on Nov 13, 2001 at 18:34
++ --
    Great article - however: And, for the same reasons, you should use prepare_cached instead of prepare.

    Don't use prepare_cached() with DBD::Sybase - this would open multiple connections to your server (one for each cached statement handle). If you have a need to have a few requests that you are going to call a *lot* you may want to create stored procedures for them instead.

    Michael

 [reply]
Another Good Reason to use RaiseError
by grantm on Jun 19, 2002 at 10:08
++ --

    btrott mentioned that RaiseError is the recommended way to check for errors but that you could alternatively check the error return value for each DBI method call. Here's subtle case where not using RaiseError can bite you:

    
        while ($sth->fetch) {
            print join("\t", $id, $name, $phone), "\n";
        }
    

    On the face of it, this will repeatedly call fetch until there is nothing more to fetch. In fact, fetch will return undef either when all rows have been fetched or when there was an error. Here's a couple of cases where fetch might successfully return a number of rows then return undef for an error:

    • you selected a calculated value (eg: total/count) which caused a divide by zero error
    • one of your columns is a long text field that exceeds the default buffer size (LongReadLen) and because you left the LongTruncOK option at default, it throws an error

    In each case, you might see some output and mistakenly believe you'd seen all matching rows. The moral - always set RaiseError and wrap DBI calls using eval

 [reply]
help
by comexos on Dec 17, 2002 at 13:38
++ --
    hello, i just tried to use such a persistent db-connection, but it won't work... when i load the script, it calls the Database-module which opens the db-connection , this works fine... but when i reload the page, or call it with different params, the module first closes the db-connection, then some server-info is printed and after that it reopens the db-conection and prints the rest...
    here's my code:

    file Database.pm:
    =============
    package Database;
    
    use strict;
    use warnings;
    # with or without use Apache::DBI; the result is the same
    # i also load Apache::DBI in the startup.pl file...
    use DBI;
    use DBD::mysql;
    
    our (@EXPORT, $dbh);
    
    use base qw/Exporter/;
    @EXPORT = qw($dbh);
    
    #BEGIN {
    print "Content-Type: text/html\n\n";
    print "\nopening db connection!\n";
    $dbh = DBI->connect('DBI:mysql:dbname', 'user', 'pwd');
    #}
    
    END {
       $dbh->disconnect;
       print "\nshutting down db-conn!\n"; #should not happen...
    }
    
    1;
    file index.cgi:
    ==========
    #!/usr/bin/perl -w
    
    use strict;
    use warnings;
    use CGI;
    
    use Database;
    
    print "Content-type: text/html\n\n";
    
    my $q = new CGI;
    my %params = $q->Vars;
    
    my $userid = delete $params{USER};
    
    my $user = $dbh->selectrow_array("select NAME from USER where ID=?", u
    +ndef, $userid);
    
    print "name of the user: $user!";
    -----
    and here you can see what's happening:
    http://www.detonation.org/cgi-perl/comexos/index.cgi?USER=6

    hm, has anyone an idea?
    thx for help ...
 [reply]

Back to Tutorials


XP Nodelet
You have 8 votes left today.
Node Status
Node Type: perltutorial
help
Chatterbox
<PodMaster> the key name is whatever "pNum%5B%5D" translates to
<z28> that works great! thanks so much!

How do I use this?
Other Users
Others lurking around the monastery: (17)
davorg
rob_au
ar0n
atcroft
theorbtwo
PodMaster
davis
valdez
snafu
AcidHawk
artist
Callum
zgrebe
aging acolyte
chimni
Coruscate
gopi
Sections
Seekers of Perl Wisdom
Categorized Q&A
Cool Uses for Perl
Obfuscated Code
Snippets Section
Code Catacombs
Meditations
Perl Poetry
Craft
Perl News
Information
Guide to the Monastery
Perl Monks Site FAQ
Site How To
Voting/Experience System
Outside Links
Tutorials
Library
Perl FAQ
Your Input
Perl Monks Discussion
Make your petition
Editor Requests
Leftovers
log zgrebe out
Super Search
The St. Larry Wall Shrine
Saints in our Book
Perl Monks User Search
Newest Nodes
Quests
Awards
Random Node
Perl Monks Merchandise
Buy PerlMonks Gear and Books
Voting Booth
Top goal for 2003
Lose weight
Get In Shape
Money Related
Get a Date
Become an Uber-Coder
Win the Lottery
Get a life
Get a (new)? job
Other
[results]
  [304 votes][past polls]