Challenge Overview
To speed docker-compose start process update bash script for Dockerfile so existent Kafka topics are not created again.
Project Background
Topcode Connect is a client-facing application of Topcoder. Topcoder Connect uses several microservices like Topcoder Project Service and other related microservices. Many of these services communicate using Kafka messages. Recently, we create a docker-compose file that runs all the services automatically and it also automatically creates all required Kafka topics which is done using "kafka-client" container via bash script. It works well, but creating topics in Kafka is slow and takes several minutes. Many containers inside "docker-compose" depend on the Kafka topics, so all of them are waiting for "kafka-client" to create all the topics and exit before other containers can start.
When we first time start "docker-compose" files we cannot avoid it. But if we stop docker-compose and after run it again "kafka-client" would try to create topics again even though they have been already created and they are still there. And we can see in the output of "kafka-client" all the attempts to create topics again, see screenshot. So even it fails all other containers still have to wait several minutes until the "kafka-client" finish.
Technology Stack
-
Docker Compose
-
Bash
Code access
- Topcoder Project Service repo https://github.com/topcoder-platform/tc-project-service branch `develop`, commit `52f9cb2fa733fce3011f575e912357da4dd07a0d` or later.
Individual requirements
Update "kafka-client" Dockerfile and/or its bash script so it only runs a command to create a Kafka topic if such a topic doesn't yet exist.
-
Get the list of already created Kafka topics with a single command. And load the topics.txt file with a single command too, as it's done now.
-
Run a command to create a topic only if some topic from topics.txt is not yet created.
For testing you may run commands inside "local/full" directory:
-
docker-compose start -d // to start containers
-
docker-compose down // to stop containers
-
docker-compose logs -f kafka-client // to see log of "kafka-client" container
General requirements
-
Make sure that permissions for files are not changed in your Git patch.
-
Git patch should be without errors or warnings.
If you have any questions or concerns, please, feel free to ask on the forum.
Final Submission Guidelines
Submit a zip file which would include:
-
Git patch with changes you’ve made to the code in our repository.
Additionally, the winner would be required to raise a pull request to the repository after the challenge is completed.