Missing bins in Elastic Search

On a chilly November day we have been presented with an interesting technical problem: IPWeb and Adobe Premiere Pro IPLink plugin working off a large IPDirector system have been missing a root-level bin, where a majority of customer’s media resided.

It is worth noting that IPWeb and newer versions of IPLink plugin have one thing in common: they rely on the Indexing Service (based on Elastic Search) to get the asset, log and bin information from IPDirector.

My first port of call was to actually check the index contents. Elastic Search _head plugin has helped me view the exact contents of the index:

http://elastic-search-ip:9200/_plugin/head/

There was no trace of the bin within the plugin, so I could conclude that the Bin information has never made it to the Index itself.

The Bin has been definitely present within the IPDirector DB, and the respective audits have been created by Bootstrapping service:

The question was – does the Crawler service correctly process this entry?

I’ve turned to Crawler logs and noticed the following:

[67] DEBUG SqlBinCrawler – BinDocument >> 918 operations Bin.BootStrap prepared in 100 ms
[67] FATAL CrawlerService – System.InvalidOperationException: Sequence contains no matching element

at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)

at EVS.IPDirector.Logsheet.Repository.LogsheetRepository.<>c__DisplayClass8_0.<GetLogsheetWithPaths>b__1(NormalizedLogsheet normalizedLogsheet)

at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()

at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)

at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

at EVS.IPDirector.Indexing.Crawler.SqlCrawler.Processor.Index.IndexProcessorBase`3.Prepare(IEnumerable`1 indexingServiceAudits)

at EVS.IPDirector.Indexing.Crawler.SqlCrawler.SqlCrawler`2.PreparedServiceAuditContainers(List`1 auditPack, String documentClassName)

at EVS.IPDirector.Indexing.Crawler.SqlCrawler.SqlCrawler`2.ProcessAudits(Int32 lastIndexedDocumentAuditId)

at EVS.IPDirector.Indexing.Crawler.SqlCrawler.SqlCrawler`2.Crawl()

 

We already know that Bootstrapping Bin elements for IS involves Logsheet Directories (from the relevant stored procedure):

So a corruption in the Logsheet Normalised and Materialised Path tables can cause the Crawler to fail half-way through treating a batch of bins, without ever proceeding.

 

Let’s try to fix the corruption by stopping services and re-populating the relevant tables:

 

exec BootStrapLogsheetNormalizedAndMaterializedPath

(1 row(s) affected)

(77276 row(s) affected)

(77277 row(s) affected)

 

Now our Crawler had healthy data to work with. Let’s delete the Elastic Search index, re-build it from a clean slate and start IS services.

After this we no longer get errors in the Crawler logs – it is happily working away, handing assets to Pusher.

Once the Elastic Search index has been re-built, the missing bin has re-appeared in IPWeb and IPLink plugin.