Topcoder Project ES Processor - Improve local setup

Challenge Objectives

Update a docker-compose file to run a Kafka instance for easy local development. 

This challenge requires testing submission on macOS and Linux systems.

Project Background

Project ES Processor is the helper microservice for the Project Service V5. At the moment, when we work on the Project ES Processor locally, we have to manually set up a local version of Kafka as per README. Instead, we would like Kafka to be set up locally automatically using docker-compose. We already did this for Project Service V5 in a recent challenge and we would like to copy/implement it a similar way for Project ES Processor setup now.

Technology Stack

• Docker Compose

Code access

- Project ES Processor repo https://github.com/topcoder-platform/project-processor-es branch develop, commit b0ec1d759671fe18dda3ac4314a4e7bb70d385cf or later.

- Config to deploy Project ES Processor locally is provided on the forum.

Individual requirements

Improve setup

• Rename folder “docker-es” to “local”.

• Update “local/docker-compose.yml” to run all the services which are required to run Kafka locally. We already did this Project Service V5, so you may copy and adjust it from there https://github.com/topcoder-platform/tc-project-service/blob/v5-upgrade/local/full/docker-compose.yml#L31-L48.
  - when Kafka is started we create all the required Kafka topics https://github.com/topcoder-platform/tc-project-service/blob/v5-upgrade/local/full/kafka-client/topics.txt. For Project ES Processor we don’t have to create so many topics and we have to create only 3 of them: https://github.com/topcoder-platform/tc-project-service/blob/v5-upgrade/local/full/kafka-client/topics.txt#L53-L55.

• There is one important moment that has to be taken care of. In Project Service V5 docker-compose file we run Kafka and Project ES Processor both inside the docker so Project ES Processor can easily connect to Kafka inside the docker. But now we want to run Kafka inside the docker, while we run Project ES Processor on the host machine (outside the docker). When we try to do so on macOS there are errors “Error: getaddrinfo ENOTFOUND d882585f6df1 d882585f6df1:9092” https://monosnap.com/file/mPapQ2QxsTwS4h6CPs5wNGXpPuDkjB. Even though other services inside the docker can be accessed from the host machine, somehow Project ES Processor running on the host machine cannot access Kafka inside docker.
  Please, make sure that Project ES Processor running outside the docker can access Kafka inside docker on macOS and Linux (Windows would be nice but is not required).
  Hint. Probably the solution may be found here.

Update README

• Remove existent section about how to setup Kafka, and instead, add to README section with Kafka commands https://github.com/topcoder-platform/tc-project-service/tree/v5-upgrade#kafka-commands
  - additionally, add a command example on how to send messages to the topics from a file and where all the files with message examples are placed https://monosnap.com/file/mjBRATvuXPt8fguqXtUF184fjzAzaW.

• As now we run Kafka inside docker, update the README file to reflect this fact. In particular, update all the command examples for Kafka to command inside docker. As now all the Kafka commands are referring the local Kafka folder.

Verification

[Required] Please, provide 2 demo videos showing that setup works well on macOS and Linux (Ubuntu is preferred, but other Linux systems are also acceptable). In particular, show that Project ES Processor running on the host machine can successfully connect to the Kafka inside Docker and handle events. The submission would not be reviewed without demo videos.

General requirements

• DON’T include the values for environment variables AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_URL, AUTH0_AUDIENCE, AUTH0_PROXY_SERVER_URL inside the repository or git patch. These variables should be always configurable manually by setting environment variables.

If you have any questions or concerns, please, feel free to ask on the forum.

Final Submission Guidelines

Submit a zip file which would include:

• Git patch with changes you’ve made to the code in our repository.

• Text file with links to the demo videos or demo video files.

Additionally, the winner would be required to raise a pull request to the repository after the challenge is completed.