ELK: Architectural points of extension and scalability for the ELK stack

elasticsearch-logoThe ELK stack (ElasticSearch-Logstash-Kibana), is a horizontally scalable solution with multiple tiers and points of extension and scalability.

Because so many companies have adopted the platform and tuned it for their specific use cases, it would be impossible to enumerate all the novel ways in which scalability and availability had been enhanced by load balancers, message queues, indexes on distinct physical drives, etc… So in this article I want to explore the obvious extension points, and encourage the reader to treat this as a starting point in their own design and deployment.

Continue reading “ELK: Architectural points of extension and scalability for the ELK stack”

ELK: Federated Search with a Tribe node

elasticsearch-logoAlthough the ELK stack has rich support for clustering, clustering is not supported over WAN connections due to Elasticsearch being sensitive to latency.  There are also practical concerns of network throughput given how much data some installations index on an hourly basis.

So as nice as it would be to have a unified, eventually consistent cluster span across your North America and European datacenters, that is not currently a possibility.  Across availability zones in the same AWS datacenter will work, but not across different regions.

Federated Search

But first let’s consider why we want a distributed Elasticsearch cluster in the first place.  It is not typically for geo failover or disaster recovery (because we can implement that separately in each datacenter), but more often because we want end users to have a federated search experience.

We want end users to go to a single Kibana instance, regardless of which cluster they want to search, and be able to execute a search query against the data.  A Tribe node can bridge two distinct clusters for this purpose.

Continue reading “ELK: Federated Search with a Tribe node”

EMC OnDemand: Federated Identity Management and Silent SSO

Identity Management for On-Premise Applications

Our industry today has some very proven technologies for providing a single set of login credentials to applications installed on-premise.  Most commonly, companies use a central Identity Management system (e.g. Microsoft Active Directory/Oracle Internet Directory/IBM Tivoli), and these systems implement an LDAP interface that 3rd party applications can call to validate user credentials.

This allows end users to login to their internal HR portal, SharePoint site, or local Documentum Webtop with the same credentials they used to gain entrance into their Windows Desktop, and is termed SSO (Single Sign-On).  This has dramatically improved the end user experience, as well as improved the ability of IT to mange the risk and policies surrounding identity management.

Continue reading “EMC OnDemand: Federated Identity Management and Silent SSO”