Allen Institute for AI Releases Paper Finder: A New Approach to Academic Literature Search

The Allen Institute for AI (AI2) has released Paper Finder, a new literature search system that uses large language models (LLMs) to help researchers find academic papers.

A New Approach to Literature Search

Paper Finder represents a departure from traditional academic search engines by attempting to replicate the natural research process that human scholars use. Rather than relying solely on keyword matching or citation metrics, the system employs a multi-step approach that includes analyzing queries, following citations, and evaluating relevance through the lens of how researchers typically conduct literature reviews.

For instance, when searching for papers about "unscripted dialogue datasets with speaker property annotations," Paper Finder breaks down the query into components, searches through multiple channels, and provides explanations for why each result is relevant. This approach can be seen in action through this example search.

Technical Implementation

The system operates through several key components:

1. Query Analyzer: Breaks down search requests into intents and components
2. Query Planner: Develops execution strategies based on the analyzed query
3. Multiple Search Sub-flows: Including specific paper search and semantic search capabilities
4. Relevance Judgment: Uses LLMs to evaluate how well papers match search criteria

To optimize performance, Paper Finder employs a "fast mode" by default, which provides quick results for common queries. Users can request more comprehensive searches when needed by specifically asking for "extensive" results.

Conclusion

AI2 positions Paper Finder as part of a broader initiative to develop comprehensive research assistance tools. The institute plans to expand capabilities in:

• Literature organization
• Experiment design
• Statistical analysis
• Experiment execution

The system is currently available for public use at paperfinder.allen.ai.