Challenge Overview
Update a docker-compose file to make Kafka service accessible inside a docker container and outside the docker container (from the host machine) at the same time.
Project Background
Topcode Connect is a client-facing application of Topcoder. Topcoder Connect uses several microservices like Topcoder Project Service and Topcoder Notifications Service. And these microservices by themselves depend on other services. As the quantity of microservices grows it became hard and time-consuming to set up all the services locally during development.
Part 1.
In the previous challenge we created docker-compose which runs all the of the microservices altogether, so our local setup process became as easy as:
1. start "full/docker-compose" with all dependant microservices
2. start Topcoder Project Service - the main backend of Topcoder Connect App
3. start Topcoder Connect App and point it to use a local version of "Topcoder Project Service" and "Topcoder Notifications Service" (which is run in the docker container at step 1.)
Usually, when we run locally we want to make some changes in the code of these apps, and such setup works well as long as we only want to change the code of "Topcoder Project Service" or "Topcoder Connect App". But it's quite often that we want to also change the code of other apps, which we run inside a docker container. The obvious idea if we want to change some of the apps which we run inside the docker container we have to stop it inside the container, which we can do by running command "docker-compose stop <app-name>", and start that app on the host machine manually. Though there appears to be one difficulty: almost all of the apps inside docker-compose depend on Kafka service, they either produce Kafka events or consume them. And Kafka service can be only accessed by apps inside docker-compose, but cannot be accessed by apps run on the host machine manually. So when we start one of the apps which depend on Kafka manually, it cannot communicate with other services inside docker-compose, as it cannot access Kafka inside docker-compose.
Part 2.
In another challenge, we've managed to run Kafka service inside docker-compose so it is accessible from the host machine, see the resulting docker-compose. Though, now such Kafka service cannot be accessed inside docker-compose. So if we use this config of Kafka in our previous "full/docker-compose", such Kafka services can be accessed by the apps we run manually on the host machine, but cannot be accessed by other services inside docker-compose. As a result, if we want to work on some apps locally, we have to run each app manually using this guide which is again time-consuming and prone to errors.
Technology Stack
-
Docker Compose
Code access
- Topcoder Project Service repo https://github.com/topcoder-platform/tc-project-service branch `develop`, commit `0af475e753eac020bade85b4252812543709d9a2` or later.
- Config and references to local setup guides are provided on the forum.
Individual requirements
Now we need your help to manage to configure Kafka service inside a "full/docker-compose" file so it can be accessed by the services which are run inside docker-compose and which are run outside on the host machine at the same time.
Our main aim that we can work locally on these apps like this:
Imagine that we want to change the code of Topcoder Project Service.
1. start "full/docker-compose" which runs all the dependant services and apps inside docker
2. start Topcoder Project Service manually on the host machine
At this moment everything should already work, and all the services can access Kafka inside docker.
Now imagine that we found out that we have to additionally change the code of Project ES Processor which is run inside docker-compose see config. So we want the next process to work:
3. stop "Project ES Processor" inside docker by running "docker-compose stop project-processor-es"
4. start "Project ES Processor" manually on the host machine so now we can dynamically change it
At this moment we expect that "Project ES Processor" can successfully connect to the Kafka which is running inside the docker. Other apps that are still run inside docker-compose should be still able to access the same Kafka instance too.
This is how it looks when "Project ES Processor" cannot access Kafka: “Error: getaddrinfo ENOTFOUND d882585f6df1 d882585f6df1:9092” https://monosnap.com/file/mPapQ2QxsTwS4h6CPs5wNGXpPuDkjB.
References
Wurstmeister Kafka Docker Connectivity Guide
Connect to Kafka running in Docker from local machine (StackOverflow)
Topcoder Project Service - Improve local setup challenge
Topcoder Project ES Processor - Improve local setup challenge
General requirements
-
The process above should work for any app which we run inside docker-compose: "tc-bus-api", "project-processor-es", "tc-notifications-processor", "tc-notifications-api", so they should work "inside" and "outside".
-
The process should be is as easy as shown in the requirements above, without changing configuration, rebuilding Docker Images etc.
-
The setup process must work on macOS and Linux operating systems. Supporting Windows would be nice but is not required.
-
DON’T include the values for environment variables AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_URL, AUTH0_AUDIENCE, AUTH0_PROXY_SERVER_URL inside the repository or git patch. These variables should be always configurable manually by setting environment variables.
Validation
[Required] Please, provide demo videos clearly showing that setup works well on macOS and Linux (Ubuntu is preferred, but other Linux systems are also acceptable).
If you have any questions or concerns, please, feel free to ask on the forum.
Final Submission Guidelines
Submit a zip file which would include:
-
Git patch with changes you’ve made to the code in our repository.
-
Text file with links to the demo videos or demo video files.
Additionally, the winner would be required to raise a pull request to the repository after the challenge is completed.