dtSearch excludes noise words

Thu, 20 May 2010 17:52:00 GMT

A document and search results in the dtSearch boolean search sample.

Version 7.64 adds dtsListIndexSkipNoiseWords to list words in an index without including any noise words.

The dtSearch product line can instantly search terabytes of text across a desktop, network, Internet or Intranet site. dtSearch products also serve as tools for publishing, with instant text searching, large document collections to Web sites or portable media. Developers can embed dtSearch's instant searching and file format support into their own applications.

The following editions are available:

dtSearch Desktop with Spider
Instantly search your desktop, as well as selected Spidered Web sites.
dtSearch Network with Spider
Instantly search the many forms of data that exist across a large enterprise network; Spider adds remote Web sites to a searchable database.
dtSearch Publish
Quickly publish instatnly searchable document collections or web site content to portable media (CDs, DVDs, external hard drives, etc.)
dtSearch Text Retrieval Engine for Win and .NET (32-bit / 64-bit)
Add dtSearch search features and built-in format support to your application; API supports .NET, C++, Java, SQL, etc. .NET Spider API.
dtSearch Web with Spider (32-bit / 64-bit)
Quickly publish a wide variety of file types to a Web site; Spider adds local or remote web sites (including dynamically-generated content) to a sites's searchable database.

Updates in V7.64

Enhancements (dtSearch Engine)

Added dtsSearchLanguageAnalyzerSynonyms flag to enable using a language analyzer to generate morphological variations on a search term at search time. When this flag is set, the language analyzer is called for each word or phrase in the search request. The flag dtsLaInputIsSearchTerm is passed to the language analyzer in dtsLaJob.flags, so the language analyzer knows why it is being called.
Added dtssGetWordBreaker API function to provide direct access to the dtSearch Engine's internal word breaker using the language analyzer API. For sample code demonstrating how to use this API, see the WordBreak example in examples\vc8\WordBreak.
Added more structural information to the output generated by conversion to the it_ContentAsXml file format.
Added to COM interface: WordListBuilder.ListFieldValues, WordListBuilder.SetFilter, and IndexJob.EnumerableFields.
Added dtsListIndexSkipNoiseWords flag for ListIndexJob to list words in an index without including any noise words.
Added dtsoFfSkipDataSourceFields flag for Options.FieldFlags to prevent DocFields values from appearing in FileConverter output

Fixes and minor enhancements

Fixed incorrect display of CreationDate and ModDate properties in PDF files
Fixed incorrect hit highlighting when Unicode Filtering options at search time different from options used to index a file. To ensure consistent options, Unicode Filtering options are stored in the index when the index is created, in the index_a.ix file.
Fixed error updating index when directory specified for temporary files is inaccessible.
Fixed index merge bug causing "Inconsistent doc ids from target index" error during merge.
Fixed two search report bugs causing incorrect hit highlighting.
Improved formatting of documents converted from Ami Pro and Quattro Pro to HTML
Added automatic detection of gb2312 and JIS encoding.
Added automatic detection of XyWrite, XBase, WordStar 3.x, and WordPerfect 4.2 and TAR files.
Improved reporting of file types by FileConverter.DetectedTypeId, providing much more specific information about Microsoft Word versions and adding type detection for additional file formats
Added support for text extraction from Adobe Framemaker MIF, XFA form templates in PDF files, and Visio XML files
Fixed "Excessive nesting" error indexing OpenOffice document due to bug parsing table structure
Fixed RTF file parser bug affecting handling of the \upr tag
Other file parser bug fixes affecting Multimate, Lotus 1-2-3, PDF, Word, PowerPoint

About dtSearch Corp.

A leading supplier of text retrieval software, dtSearch Corp. develops, manufactures and sells the dtSearch text retrieval product line. dtSearch products have been the smart choice for Text Retrieval since 1991. The dtSearch product line is known for its "industrial-strength" (PC Magazine) ability to instantly search terabytes of text. dtSearch product line includes end-user, enterprise and developer text retrieval products. dtSearch product line also includes publishing capabilities, for publishing large document collections to Web sites or CD/DVD and Spidering capabilities, for remote site and distributed searching access. dtSearch products have received multiple awards and hundreds of excellent press reviews. Fortune 500 companies and others with some of the most demanding document search needs in the world rely on dtSearch. 4 out of 5 of Fortune Magazine's most profitable companies have dtSearch developer or multi-user licenses. Typical corporate uses of dtSearch products include general information retrieval, Internet/Intranet site searching and access to technical documentation.

dtSearch updates Language Packs

Wed, 21 Oct 2009 00:00:00 GMT

Highlighting search results in dtSearch.

Now includes Bulgarian and Slovenian, plus bilingual French/English and German/English stemming.

dtSearch Text Retrieval Engine and dtSearch Web with Spider are supplied with stemming rules and a noise-word file for English(US). Stemming is the only search expansion option which is 'on' by default in the dtSearch end-user products; the reason for this is that stemming is almost always useful when making a search, and adds little to the time required to make a search. Unlike some other search engines, dtSearch applies stemming at search time, there is no need to build indexes specifically to apply stemming and no need to build separate indices for each language in use.

With the stemming option selected dtSearch will find plurals and many other variations; for example a search on print will find printers, printing, printed automatically. However, if you are searching documents written in other languages, the English stemming rules will cause you to miss many word variations which do not occur in English (e.g. verb and noun changes with gender), and you may find that words which are unrelated are found in error. Furthermore, the English noise word list, which is designed to remove unwanted English words from your index to keep the index size small, is not suitable for other languages; your indexes may contain many words which will not be useful in searches and which will add to the size of your indexes.

The solution is to use language specific files in place of the default US English files. These are supplied in the form of Language Extension Packs which contain files for many languages. All files are in Unicode format.

Updates

Bulgarian language available in Eastern European Group extension pack
Slovenian language available in Eastern European Group extension pack
French/English stemming available in Western European Group extension pack
German/English stemming available in Western European Group extension pack

ComponentSource Topic | dtSearch Corp.

dtSearch excludes noise words

Updates in V7.64

About dtSearch Corp.

dtSearch updates Language Packs

Updates

About dtSearch Corp.