Indexing

From FindAndLearn::Cookbook
Jump to: navigation, search

Disambiguation[edit source]

Entities keyworded Indexing[edit source]


Indexing Terms Mentioned In

Indexing Domains:


Indexing involves 2 steps:

  1. Feeding resources from a ResourceSilo into a Datastore
  2. Applying an Indexer to resources in a Datastore

Feeding resources from a ResourceSilo into a Datastore[edit | edit source]

See SystemComponent "datastores"

From E-Mail account to Datastore by

Data fed?
Configuration
Subsequent pipeline
Tools

From File system to Datastore by https://github.com/dataspects/dsdocument-cli

Data fed?
Configuration

Configure the datastore:

<Datastore ID>        automatic (e.g. 12)
<Datastore Label>     Shown in information sources list on https://ui.dataspects.com/search
<Datastore API Key>   automatic (e.g. c8b89bc3-0139-11wa-8ef3-8c164563716b)
<Datastore Doc Regex> Only file names matching this regex will be fed to the datastore (Regex Tester - Golang)

Configure and run the feeder to index matching files in and below the current folder:

user@workstation:/yourfolder$ ./dsdocument-cli \
                              --id  <Datastore ID>      # From https://ui.dataspects.com/datastores/files \
                              --url https://datastore.dataspects.com \
                              --key <Datastore API Key> # From https://ui.dataspects.com/datastores/files
Subsequent pipeline
Tools

From Repository to Datastore by https://github.com/dataspects/dsrepository-cli

Data fed?
Configuration

Configure the datastore:

<ID>      automatic
<Label>   Shown in information sources list on https://ui.dataspects.com/search
<API Key> automatic
<Regex>   Only file names matching this regex will be fed to the datastore (Regex Tester - Golang)

Configure the feeder:

user@workstation:/yourrepo$ ./dsrepository-cli \
                              --id  <ID>      # From https://ui.dataspects.com/datastores/repositories/code\
                              --url https://datastore.dataspects.com/repositories/code \
                              --key <API Key> # From https://ui.dataspects.com/datastores/repositories/code
Subsequent pipeline

Items fed to the datastore will be fed on to an Indexer.

Currently the CodeIndexer doesn't index the actual file but scans it for (?i)error *(\d{4}:\d{2})(.*), creates an ErrorCode entity named \d{4}:\d{2} with an annotation "OccursInFile".

Tools


Applying an Indexer to resources in a Datastore[edit | edit source]