LogStash Filter Plugin to Store in ElasticSearch

JSONRubyRedisElasticsearchEC2Other

Key Information

Register
Submit
The challenge is finished.

Challenge Overview

TopCoder paltform uses Logstash service which allows to pull data from single or multiple sources and store to different data stores. The first task here is to read data from a log file and store inside elasticsearch (which is setup locally).  This task is already done and the logstash configuration file is attached.

Next time, for deltas, it tries to load data into ElasticSearch, we should even handle deltas (updation and deletion) without fully replacing the elasticsearch record.

Elasticsearch download: https://www.elastic.co/downloads/elasticsearch

Logstash download:  https://www.elastic.co/downloads/logstash
Logstash input, filter and output plugin documentation is available here:

1. input plugins - https://www.elastic.co/guide/en/logstash/current/input-plugins.html
2. filter plugins - https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
3. Output plugins - https://www.elastic.co/guide/en/logstash/current/output-plugins.html

There are 1 file formats in this challenge - users. The logs should be stored into elasticsearch store.



Final Submission Guidelines

Task 1) This is already implemented. Please see the attached configuration file:

For users, please use the following records:

ChallengeId  Roles                       userId
30049004     Approver                 151743
30049004     Copilot                     151743
30049004     Iterative Reviewer   151743
30049004     Manager                  262467

The Json structure to be stored inside elasticsearch is shown below:

{
  "challengeId": 30049004,
  "users": [{
    "id": 151743,
    "roles": ["Approver", "Copilot", "Iterative Reviewer"]
  },
  {
    "id" : 262467
    "roles": ["Manager"]
  }  
]
}
This step is done and i have provided the logstash configuration file.
 
Task 2)
Next time, for delta update, please use the following records:
ChallengeId  Roles          userId
30049004     Manager    151743
30049004     Manager    151742

The Json structure which needs to be stored inside elasticsearch is shown below:

{
  "challengeId": 30049004,
  "users": [{
    "id": 151743,
    "roles": ["Approver", "Copilot", "Iterative Review", "Manager"]
  },
  {
    "id" : 262467
    "roles": ["Manager"]
  },
  {
    "id" : 151742
    "roles": ["Manager"]
  }
]
}
As you can see, the roles are updated and entire elasticsearch document is not replaced.
 
Task 3)
For deletion, Please use the following records which can be in a new file named 'delete_users'
ChallengeId  Roles          userId
30049004     Manager    262467
30049004     Copilot    151743
 

The Json structure which needs to be stored inside elasticsearch is shown below:

{
  "challengeId": 30049004,
  "users": [{
    "id": 151743,
    "roles": ["Approver", "Iterative Reviewer", "Manager"]
  },
  {
    "id" : 151742
    "roles": ["Manager"]
  }
]
}
 
As you can see below, for 151743, Copilot role is deleted and the entire records for userId 262467 is deleted 
because there is only one role for this user
 
The attached 'logstash_users.txt' has task 1 implemented where the source is coming from Informix DB. Please feel
free to replace this with json file.

ELIGIBLE EVENTS:

2016 TopCoder(R) Open

REVIEW STYLE:

Final Review:

Community Review Board

Approval:

User Sign-Off

SHARE:

ID: 30052889