Previous | Next
WireHose Developers Guide

Importing in a separate thread

Since crawling the feeds should happen repeatedly while the application is running -- and we don't want to delay application startup while the feeds are crawled -- the feed crawler should run in its own thread.

  1. Add this method to Importer.java:
    public static void crawlFeedsInThread() {
        Thread crawler = new Thread() {
            public void run() {
                try {
                    sleep(1000 * 15); // wait 15 secs before first crawl
                    while (true) {
                        crawlFeeds();
                        sleep(1000 * 60 * 5); // crawl every 5 minutes
                    }
                } catch (InterruptedException e) {
                    System.out.println("crawler: "+e);
                }
            }
        };
        crawler.start();
    }
  2. And change this line in Application.java:
    Importer.crawlFeeds();

    to this:

    Importer.crawlFeedsInThread();

Now, when you launch Hello World, application startup isn't delayed, and the application will check every five minutes for feeds which haven't been crawled in the last hour to fetch.

Note: Since EOF is not by default multithreaded, this technique will still lock the EOF stack for each fetch or commit by the importer to the database. For maximum multithreadedness, you can create a new EOF stack for the importer. To do this, you create a new object store coordinator for the importer, and use it as the root object store for the importer's editing contexts.

  1. Add this line to Importer.java:
    static EOObjectStoreCoordinator objStoreCoord = new EOObjectStoreCoordinator();
  2. And change all EOEditingContext constructors in Importer.java from this:
    new EOEditingContext()

    to this:

    new EOEditingContext(objStoreCoord)

WireHose posts a ShouldInvalidateCache notification when new items are tagged, so objects in other EOF stacks can keep their caches up to date. See the reference documentation for WHFetcher, WHCachingDataSource, WHTagFetcher and WHTagDataSource for details. If you are deploying multiple instances of your application, you can listen for this notification and propagate it to the other instances to ensure that all the instances stay up to date.


Previous | Next