Site-wide Search

• Documentation on working with interMedia: http://www.arsdigita.com/asj/intermedia
• User-accessible directory: /search/
• Site administrator directory: none
• data model : /doc/sql/site-wide-search.sql

The Big Idea

Under the Hood

Deeper Under the Hood

create table site_wide_index (
        table_name              varchar(30) not null,
        the_key                 varchar(700) not null,
        one_line_description    varchar(4000) not null,
        datastore               char(1) not null, -- place holder for datastore column
        primary key (table_name, the_key)
);

We must have the primary key constraint on this index because Oracle Intermedia can't deal with any table that doesn't have a primary key.

We need one_line_description so that we can display a nice list of query results to users who do a site-wide search. If we didn't keep this, we'd have to outer join with every possible table that contributes to the index or laboriously look up every row with a PL/SQL procedure. And even then we'd need to keep some record of, for each table, what was the best way to get a one-line description.

For consistent site-wide user interface, we keep a table of what the table names in the index should be called as far as sections go. For example, the table name for the /bboard module is "bboard" but a publisher might wish this presented in a search form as "Discussion Forums". This is done by entering rows in the table_acs_properties table, described in the user profiling documentation. We reproduce the table definition here:

create table table_acs_properties (
             table_name      varchar(30) primary key,
             section_name    varchar(100) not null,
             user_url_stub   varchar(200) not null,
             admin_url_stub  varchar(200) not null
);

Triggers that fill the table

Note that the datastore column is just a placeholder; its value is unimportant (we just happen to use the letter 'a'). Updating this column causes the index to be rebuilt for a row by calling the user datastore procedure (see next section).

create or replace trigger static_pages_sws_insert_tr
  after insert on static_pages for each row
BEGIN
  IF :NEW.index_p = 't' THEN
     -- we have to create a new row in the index table for this row.
     insert into site_wide_index (table_name, the_key, one_line_description, datastore)
       values ('static_pages', :new.page_id, :new.page_title, 'a');
  END IF;
END;
/
show errors
  

CREATE OR replace trigger static_pages_sws_update_tr
  after UPDATE ON static_pages FOR each row
BEGIN
  IF :old.index_p = 'f' AND :NEW.index_p = 't' THEN
     insert into site_wide_index (table_name, the_key, one_line_description, datastore)
       values ('static_pages', :new.page_id, :new.page_title, 'a');
  ELSIF :old.index_p = 't' AND :NEW.index_p = 'f' THEN
     DELETE FROM site_wide_index
       WHERE table_name = 'static_pages'
       AND the_key = :old.page_id;
  ELSIF :NEW.index_p = 't' THEN
     update site_wide_index 
       set the_key = :new.page_id, one_line_description = nvl(:new.page_title, '(no title)'), datastore = 'a'
       where the_key = :old.page_id
       and table_name = 'static_pages';
  END IF;
end;
/
show errors


CREATE OR replace trigger static_pages_sws_delete_tr
  after DELETE ON static_pages FOR each row
BEGIN
  IF :old.index_p = 't' THEN
     DELETE FROM site_wide_index
       WHERE the_key = :old.page_id
       AND table_name = 'static_pages';
  END IF;
END;
/
show errors

User Datastore Procedure

This procedure is run to gather the text that is to be indexed for any row of the site_wide_index table. Its arguments are the rowid of the row that it is to retrieve the content for, and a temporary clob in which the results are to be stored.

Different sections can be indexed in different ways. Note that the bboard section indexes entire threads into a single entry in the site wide index, instead of indexing each message as its own entry.

The user datastore procedure must be loaded as the ctxsys user, and must be granted select access on the tables you want to index from. You must grant execute permissions on the user datastore procedure to the user who will be using it to create the index.

CREATE OR replace procedure sws_user_datastore_proc ( rid IN ROWID, tlob IN OUT nocopy clob )
IS
   v_table_name  VARCHAR(30);
   v_primary_key VARCHAR(700);
   v_one_line VARCHAR(700);
   v_static_pages_row photonet.static_pages%ROWTYPE;
   cursor bboard_cursor(v_msg_id CHAR) IS
     SELECT one_line, message 
       FROM photonet.bboard
       WHERE sort_key LIKE v_msg_id || '%';
BEGIN
     -- get various info on table and columns to index
   SELECT table_name, the_key, one_line_description
     INTO v_table_name, v_primary_key, v_one_line
     FROM photonet.site_wide_index
     WHERE rid = site_wide_index.ROWID;
   
   -- clean out the clob we're going to stuff
   dbms_lob.trim(tlob, 0);
   
   -- handle different sections
   IF v_table_name = 'bboard' THEN

      -- Get data from every message in the thread.
      FOR bboard_record IN bboard_cursor(v_primary_key) LOOP
         IF bboard_record.one_line IS NOT NULL THEN
            dbms_lob.writeappend(tlob, length(bboard_record.one_line) + 1, bboard_record.one_line || ' ');
         END IF;
         dbms_lob.append(tlob, bboard_record.message);
      END LOOP;
   ELSIF v_table_name = 'static_pages' THEN
      SELECT * INTO v_static_pages_row
        FROM photonet.static_pages
        WHERE page_id = v_primary_key;
      
      IF v_static_pages_row.page_title IS NOT NULL THEN
         dbms_lob.writeappend(tlob, length(v_static_pages_row.page_title) + 1, v_static_pages_row.page_title || ' ');
      END IF;
      dbms_lob.append(tlob, v_static_pages_row.PAGE_BODY);
   END IF;
END;
/
show errors

grant execute on sws_user_datastore_proc to photonet;

Querying

select 
  score(10) as the_score,
  the_key,
  one_line_description,
  map.user_url_stub 
from site_wide_index swi, table_acs_properties map
where swi.table_name = map.table_name
and contains(indexed_stuff,'about($user_entered_query)',10) > 0 
order by score(10) desc

select 
  score(10) as the_score,
  msg_id,
  one-line,
  posting_time,
  topic
from site_wide_index swi, bboard
where swi.the_key = bboard.msg_id 
and swi.table_name = 'bboard'
and contains(indexed_stuff,'about($user_entered_query)',10) > 0
order by score(10) desc

The about() in the query specifies a theme query, which takes the words of the query to be "themes" to search for. Oracle has had geniuses locked up for the last several years dividing up the English language into a taxonomy of semantic topics. interMedia will decide which of these themes your documents are most relevant to, and match those up to the words in your query. Leaving out the about() gives you a simple word query, which is more like what you'd expect.

In practice, feeding a user entered query directly to interMedia turns out to be very bad, because it is expecting queries to be specified using their little query language. Any syntactical errors in defining a query causes interMedia to cough up an error. This is very easy to do, since various punctuation characters and words like "and" are special in this query language. We use the PL/SQL function im_convert to massage the user input into a form which is safe to pass to interMedia, and which performs a combination of text search and theme search to try to bring the most relevant documents.

im_convert() cannot be called directly from within contains(), so we must use a separate database query to convert the user search string. This may be replaced at a later date into a Tcl procedure, to make modifications easier.

Tcl processing of the results

set counter 0 
while {[ns_db getrow $db $selection]} {
    set_variables_after_query
    incr counter
    if { ![info exists max_score] } {
        # first iteration, this is the highest score
        set max_score $the_score
    }
    if { ($counter > 25) && ($the_score < [expr 0.3 * $max_score] ) } {
        # we've gotten more than 25 rows AND our relevance score
        # is down to 30% of what the maximally relevant row was
        ns_db flush $db
        break
    }
    if { ($counter > 50) && ($the_score < [expr 0.5 * $max_score] ) } {
        # take a tougher look
        ns_db flush $db
        break
    }
    if { ($counter > 100) && ($the_score < [expr 0.8 * $max_score] ) } {
        # take a tougher look yet
        ns_db flush $db
        break
    }
...
}

The heuristic cutoff in the above procedure has been packaged into the Tcl procedure ad_search_results_cutoff.

For the user who has a PhD in computer science

"More Like This"

In practice, the themes returned by interMedia for a document are fairly general, and occasionally irrelevant (including things like HTML tags). So we throw in the one_line_description as part of the query, which seems to do a better job of bringing up relevant results. See /search/query-by-example for an example of query by example.

Using a Separate Search Server

Keeping the index in sync

• run /bin/ad-context-server.pl as a cron job every hour.
• saddle your Unix system administrator with the task of making sure that the ctxsrv process is kept running at all times (this is in addition to the six Unix process that constitute a normal Oracle installation)
• create a DBMS_JOB to alter the indexes.

If you want to manually update the index sws_ctx_index, you can execute:

alter index sws_ctx_index rebuild online parameters('sync');

Optimizing interMedia indexes

alter index sws_ctx_index rebuild online 
parameters('optimize full maxtime 60');

If you care about performance