WebTracer2
WebTracer is an project based on the intention to visualise the structure of the web. There are many applications that analyse websites for structural integrity and diagnostic purposes, but few reveal the visual structure that web hypertext creates. Webtracer represents this structure as a three dimensional molecular diagram, with pages as nodes(atoms) and links as the strings(atomic forces) that connect those nodes together.

Download
webtracer2.zip (PC) (approx 780k)
UPDATE: I have changed the download file to include some .dll files that are needed to run the spider on some PCs. This should fix the problems some users have had (you should be able to just unzip and run now)

Usage
WebTracer is spilt into two programs the 'Spider' and the 'Visualizer'. The Spider is used to visit websites and produce map files containing information about their internal hypertext links. The Visualizer allows these maps to be viewed in 3D and represents the pages and links as an abstract molecular structure. Thier operation is as follows.

The Spider
You launch the spider with 'spider.exe' and after entering a URL of a website in the top field you can hit return or press the start button below. The spider will then start scanning the chosen website for pages and links, a status report is displayed in the center window. You can toggle the program from 'running' to 'paused' by clicking the button next to the Trace Progress header. The spider will continue to map the site until it reaches 1000 pages or it has found all the pages it can. At this point the map will be automatically saved into the 'Maps' folder. Using the buttons at the base of the dialog you can save a trace while it is in progress and resume mapping on an incomplete website map. You must be connected to the internet to use the spider and scanning a full 1000 pages may take some time (The Spider program icon will animate to show you that the program is still running). Because of this you can leave the spider running and continue to use your machine for other tasks.

The Visualizer
You launch the Visualizer with 'visualizer.exe' and after loading you will be shown an empty 3D space and some helpful text hints. The program will automatically detect the number of maps available for viewing and allow you to select and load one (use the arrow keys and then hit return). You can then interact with the Map using the following key commands.

H - Toggle help info
L - Load a new map
ESC - Quit
F1 - Toggle fullscreen/windowed
Arrow keys - Move in/out and rotate view
V - View selected page in web browser
You can also use the mouse
Click and drag right mouse button to rotate view
Click left button to select a node/page

You do not need to be online to use the Visualizer and you can view any map file that is placed in the 'Maps' folder. You can alos move map (.wtm) files from one machine to the next or even email them. The download comes with a few demonstration maps provided so you can try out the Visualizer.


(Screen Shots)

Requirements
The Spider application needs a connection to the internet to run, dial-up modems will work but adsl or higher speeds are preferable. The Visualizer uses OPENGL for rendering and requires a graphics card capable of displaying 3D (most modern cards will be fine). The number of pages being viewed will greatly effect the speed of the rendering (the maximum of 1000 pages may be very slow on older machines). You can try different sized maps to find a compromise between complexity and speed (the nullpointer.co.uk map is very small!, the microsoft.com map is quite large). You can try and alter the screensize and color-depth settings by editing the text file 'settings.txt' in the 'Data' folder, but be careful if you try this(maybe make a backup of the file). You can also edit the file 'blocks.txt' in the same folder to stop the spider from following certain links (links to images or audio files e.t.c.), again be careful and read the file first.

Issues/Bugs
There are some known issues with the current version: Spider doesn't follow pages with framesets properly, You cannot adjust the maximum number of pages mapped (1000) or the maximum number of links followed per page (50). Loading many maps consecutively sometimes causes the Visualizer to freak out or crash (If you just restart the program it usually works ok). Some sites just won't map and some sites just won't display but most will so keep trying. If either of the programs crash or error ther are two debug text files 'visual-debug.txt' and 'spider-debug.txt' that might contain useful information on why the problem occurs. You can send any bug reports along with these files to bugs@nullpointer.co.uk. I might not deal with them immediately but they will help inform the next version of the software.


Notes
A MAC OS port is possible, just don't hold your breath (its the networking library thats the problem!) Also a copy of this html page is included with the zip (but you wont see the images if you are offline while reading it). As usual I can accept no responsibility for any damage to you or your computer caused by the use of this software.


< back