Though it may be an imposition, I think we should be dealing with the C code and would very much appreciate seeing the C code result. I should not delay the MMS, but I could soon take your C code and compile it on Linux and see how it and the sequence of going from Basic on MS to C on Linux works. If it does work then that may be an interim solution.

Regards,

I quickly unzipped the Patternizer_C.zip but have not tried to run it since I am working full-out to get the last MMS work done. What we may now need is to have a way of either providing the C++ subroutines I have to Jure that will connect to the Patternizer or some method for Jure to be able to assemble the code on my computer.  That is, if I understand Jure's procedure correctly, he is coding in VB and then translating to C, but then he does not likely have the ability to integrate C code modules and compile them.  What is required is a small subsystem utilizing some of my routines in conjucntion with Jure's routine that identifies possible search sites and then identifies the item extraction pattern. I could easily provide Jure with the basic routines that would help him assemble that subsystem.

Jure, if you are interested, can you work in C on your computer or will we need to set up some remote communication for my computer?

Regards,

The primary objective then is to take the HTML return from an arbitrary search engine and then secondarily from an arbitrary source that uses a list format with links and get three items:

(1) The link or anchor that allows the user to click on the name and be sent to the link,

(2) The name of the link that appears after the beginning anchor tag 'a ...' (and enclosed in angle brackets but those will not appear here because of the automatic HTML translation) and before the ending anchor tag '/a', and

(3) Any description that commonly appears after the link name (2).

I have just sent you by email some basic routines for parsing and writing the tag structure from an HTML file.  The objectives would be to:

(1) Use the Google.HTML file as input and get the two files Google.tree and Google.ankr as output.

(2) Then you would want to run the procedure against an HTML you get from using one of the search engines.

(3) Get back to me so that I can send you my current item extraction routine using the already sent code.

(4) Integrate your Patternizer code with my item extraction routine and test it against search engine HTML pages that you can save or copy.

You will likely have a number of questions and issues, and I should remark that I just pasted the jureparse.cc routine together from pieces of the MMS and have not attempted to compile it.

I am now to the point of doing miscellaneous cleanup on the new version of the MMS.

Regards,

You do not need to open a URL and read an HTML file at this time since you only need to work on examples from particular engines or a small group of engines.  I have always tested against examples I have copied to disc because it is easier, faster, and the HTML remains exactly the same each time.

You appear to be saying that you need to have each tag represented by a different character which then requires you to build a tag name vs. character translator/lookup.  Once you get the get_html_tags routine running it will extract tags of whatever name and you can then accumulate together several files and sort on tag and in that way identify what tags need to be in the table.  You will likely need to modify the write_html_tree routine to not print the leading periods that indicate how deep the tag is nested in order to sort and extract the tag names properly.

But for the moment you only need to study the code in the jureparse.cc file starting with the routine called main or at the line: int main().  This is where the program starts.

// This line declares a string variable file_name and assigns `Google' to that variable.
  string file_name="Google";
// This line takes the result of get_res_test above in the file and assigns it to res which
// is declared global.  The purpose of get_res_test is to make a memory file or character
// string from the disk file before processing it.  Normally the MMS uses a memory
// character string as the return from the search engine and there is no disk file.
// It may be easier to just read the disk file directly and that is what the last routine
// in jureparse.cc does. And then you could take code out of get_res_test to make that
// happen. Once you get into how to open and read disk files, it should be easy to follow.
  res = get_res_test(file_name); // Load file into res

// This declares a integer variable and assigns 0 or NULL to it.  This null character is
// then appended to the file_name for use by the C++ file opening routine
// `ihtml.open(ifile_name.c_str());' which is above in jureparse.cc.
  int end_null=0;

// The following two string variables and assignments assemble the output file names.
// You will want to change the directory location to one you will be using.  It should be
// possible to remove the initial quoted string and just start with `file_name" to have
// the output go to the directory you run the program in.// *** put in your own output location directories here.
  string tree_file_name = "/home/n_nelson/web_search/nnget/prime_result_trees/"
                        + file_name + ".tree" + char(end_null);

  string anchor_file_name = "/home/n_nelson/web_search/nnget/prime_result_trees/"
                        + file_name + ".ankr" + char(end_null);

// This declares a variable that is defined in htmlparse.h. The purpose of this variable is
// to allow easy handling of both the tag tree and anchor linked list obtained from
// get_html_tags.
    Html_Tag_Classes html_tag_classes;

// This is how comments are displayed to the screen when the program is running.
// The flush line is used for the Apache web server so that when a program aborts I can
// get as much output as possible. Otherwise messages in the buffer are lost.
  cout << "before get_html_tags
\n";
  cout.flush();

// This line runs the get_html_tags outine that extracts the tags and anchor tags.
  html_tag_classes = get_html_tags(html_tag_classes);

// Runtime display as previously.
  cout << "write_anchor_tags html_tag_classes.top_anchor_tag= "
       << html_tag_classes.top_anchor_tag << "
\n";
  cout.flush();

// This line calls the routine that writes the Google.ankr file containing the anchor
// tags and related detail including link name according to their sequence in the file.
  write_anchor_tags(anchor_file_name, html_tag_classes.top_anchor_tag);

  cout << "write_html_tree html_tag_classes.top_tree_tag= "
       << html_tag_classes.top_tree_tag << "
\n";
  cout.flush();

// This line calls the routine that writes the Google.tree file.
   write_html_tree(tree_file_name, html_tag_classes.top_tree_tag);

// The following lines clear allocated memory.
  delete_html_tag_tree(html_tag_classes.top_tree_tag);
  delete_anchor_tags(html_tag_classes.top_anchor_tag);
  delete [] res;
  return 0;
}

The get_res_test routine above the main routine just opens the input disk file Google.html and copies it into a string variable and then copies that string variable to a char type variable which is the kind returned by the search engine and makes on-line and off-line testing easier.

But you only need to do items 3 through 6 above at this time.  I am using a different detection method based on anchor tags at the moment without any pattern matching but can see that a combination of the new method and the pattern matching method will be better.  Ths point being that when you get to the point of applying the Patternizer, you may be able to use my current anchor method to improve the overall results.

Regards,

All the lines from the top and down to and not including the line beginning the get_res_test subroutine.

Append to the previous copy the lines just referenced for the main routine.  A main routine is required for all C programs as the first routine called from the OS when the program is executed.

And then comment out all the subroutine calls using two forward slashes `//'.  Those would be

get_res_test
get_html_tags
write_anchor_tags
write_html_tree
the three following lines beginning with `delete'.

This should then allow that first main routine to compile and execute, showing the display lines. And then we can add get_res_test routine to that new program and remove the comment for that call and see if it compiles.  Then we can run the program to see if both routines work together.

Then we need to see if the htmlparse.cc program will compile.  The routines from this file will be linked later, and the compile method will likely (as done on Linux) need to reflect that.

And that gets us quite far along.

Ragards,